Has anyone seen this problem before? QHYCCD QHY600PH M · John Hayes · ... · 52 · 3170 · 10

jhayes_tucson 22.76
...
· 
·  1 like
A couple of months ago my ATIK Horizon 2 guide camera started acting "wonky."  The background level suddenly went way up.  I mitigated it by subtracting a dark frame but it has screwed up the SNR of my guide signal.  Fortunately it still works well enough to guide.  Fortunately, I already have a replacement camera sitting at the observatory ready to be installed.  This is the second failure of a Horizon 2 in less than a year.

Just last night my main imaging camera, a QHY600PH M, went "wonky" in a different way.  It almost looks as if the 16 bit image data has been bit-shifted left by ~ 4-5 bits--but that's not what is happening. We fixed it last night by pulling and reconnecting cables but I'm out of luck tonight.  I can't get it to work correctly.  Here is a comparison of two images of the same object through a LUM filter at the same cooling temperature with the same exposure before and after the "glitch".

Screen Shot 2022-11-02 at 10.10.11 PM.png

Here are histogram plots of both images.  First, before:

Screen Shot 2022-11-02 at 9.05.45 PM.png

And second, after the failure:

Screen Shot 2022-11-02 at 9.48.08 PM.png

There is no dew and the sky is clear.  This sure looks like a camera problem.  As I said, we fixed it last night by pulling cables and reconnecting them.  I've also tried a brand new USB3 cable with no improvement.  We haven't yet measured the voltage at the camera so that might be a problem.  This particular data was taken using MDL but I get the exact same results using SGP (my normal imaging program).

My question:  Has anyone else seen this problem and is there a simple fix?  I want to exhaust all possibilities before jumping on a plane to go to Chile with a replacement camera.  With two dead cameras, I'm already mentally packing my bags.


John
Like
barnold84 10.79
...
· 
·  1 like
John,

I don't have a straight answer to your question. But maybe a test to make for checking the electronics:
Can you capture a dark frame on your scope and do you have dark frames from previous sessions, when the camera worked ok? If you'd capture a new dark and it's characteristics match the "old" dark, it would point to the direction that the electronics is fine and the issue must be infront of the sensor.

EDIT: Ok... After giving it another thought, if plugging cables helped once, it could be electronics and doing this dark test won't probably help too much.

Björn
Edited ...
Like
qqbin 0.00
...
· 
John

I can see that the first image set the binning at 2 while the second set at 1?

Regards,
Robin
Like
PABresler 0.00
...
· 
I have two QHY600s. I just starte using them a couple of months ago. One has been working fine. A few days ago the cooling failed on the other. I had a Teamviewer session with Cha. The problem persisted with different power supplies and computers. Images from that camera also were produced a wide band like artifact. I was not sure if was the filters, but it appears it was the camera. The camera now is in Lompoc.

Peter
Like
MoTab 0.00
...
· 
Hi John,

It sounds like a power supply issue if plugging/unplugging fixed it and it would be good to check the voltage across the input to the camera. Before changing to a high-quality power-supply and higher gauge cables, I would frequently have the camera cooling go berserk. That said, I think it's also worthwhile to repeat the experiment using e.g., NINA or EZCAP that talk to the camera through the native interface.

Cheers,
Mo
Like
rockstarbill 11.02
...
· 
·  2 likes
John Hayes:
A couple of months ago my ATIK Horizon 2 guide camera started acting "wonky."  The background level suddenly went way up.  I mitigated it by subtracting a dark frame but it has screwed up the SNR of my guide signal.  Fortunately it still works well enough to guide.  Fortunately, I already have a replacement camera sitting at the observatory ready to be installed.  This is the second failure of a Horizon 2 in less than a year.

Just last night my main imaging camera, a QHY600PH M, went "wonky" in a different way.  It almost looks as if the 16 bit image data has been bit-shifted left by ~ 4-5 bits--but that's not what is happening. We fixed it last night by pulling and reconnecting cables but I'm out of luck tonight.  I can't get it to work correctly.  Here is a comparison of two images of the same object through a LUM filter at the same cooling temperature with the same exposure before and after the "glitch".

Screen Shot 2022-11-02 at 10.10.11 PM.png

Here are histogram plots of both images.  First, before:

Screen Shot 2022-11-02 at 9.05.45 PM.png

And second, after the failure:

Screen Shot 2022-11-02 at 9.48.08 PM.png

There is no dew and the sky is clear.  This sure looks like a camera problem.  As I said, we fixed it last night by pulling cables and reconnecting them.  I've also tried a brand new USB3 cable with no improvement.  We haven't yet measured the voltage at the camera so that might be a problem.  This particular data was taken using MDL but I get the exact same results using SGP (my normal imaging program).

My question:  Has anyone else seen this problem and is there a simple fix?  I want to exhaust all possibilities before jumping on a plane to go to Chile with a replacement camera.  With two dead cameras, I'm already mentally packing my bags.


John


John pull the USB 3 cables and replace them with USB 2. Should solve the problem. It's not the cameras fault, USB 3 is highly prone to faults.
Like
gregm 0.00
...
· 
·  1 like
For the QHY600M, I would suggest a test using the supplied QHY power supply and a known good USB cable directly to your computer 
Use the latest QHY all in one driver making sure qhy5III driver is checked as well as your imaging software so it can install the native drivers into the app

https://www.qhyccd.com/download/

Note SGP doesn’t use native camera drivers, just ASCOM. (Just one of the reasons I went to NINA)
Then use the QHY QT app to connect to the camera 
Besides seeing if if shows an image, it will update the camera with the latest firmware 

connecting with QT app, using supplied power supply and directly connected USB to computer, are usually the basic QHY test requirements before they accept a fault (from experience)

good luck
Like
MJNorman 1.81
...
· 
Hi John.

We may have seen something similar a few nights ago. I see you're using Maxim DL and have tried SGP.

We are using Voyager for image capture (and most everything). For the most part, it has been working very reliably. The other night, though, it started recording "dark" frames that may have been very similar to what you're seeing. We haven't looked too closely at the bad frames. They weren't all zeros, but I'm not sure if they were recording any meaningful signal. We'll look at some of them to gain some more insight. The strange thing is that it seemed to happen on a filter change. i.e. a long sequence of 180s L frames was fine, but shifting to 300S R/G/B frames resulted in the errant behaviour.

Voyager wasn't re-started before running the night's sequence when this happened.

Re-starting Voyager (and re-booting the mini-pc) seemed to cure it.

I haven't reported it to Voyager as it hasn't happened since and it seems better practice to re-start the imaging software each night. I suspect some sort of "memory leak" or driver initialization problem as opposed to a power / USB / other hardware issue. We do use a Pegasus Ultimate Powerbox, so we can remotely power cycle and "reset" USB connections.

I know this is a lot of detail for a problem that might have nothing to do with what you're experiencing, but my point is that it could as easily be a software "glitch" as hardware.

QHY suggests using their software to test their cameras ... I'd try using EZCap to test it. If you still have problems, it's likely time to get QHY support. They've been really good at diagnosing problems remotely for us.


​​​​Hope this helps.

Michael and Jon Norman
Like
jhayes_tucson 22.76
...
· 
·  1 like
Thank you all for your feedback and thoughts.  As usual, when I did this post, it’s easy to forget things so I didn’t supply quite enough background on what I’ve tried and how I operate.  So, in no particular order, here is some additional background that will form a response to many of the good comments and suggestions here.

1). I’ve been running this scope in it’s current configuration every clear night for almost exactly one year.  In Chile, that translates to around 300 nights.  I always run the scope with SGP.

2). The power supply is one originally supplied by FLI for my FLI-ML16803.  Before the scope went to Chile, I carefully checked the voltage against the QHY supplied power brick and they were within a few 10s of milli-volts of each other.  The FLI power brick has more current capacity than the original QHY supply so it runs really cool.  As I said in the original post, I haven’t yet asked the guys at the observatory to put a DVM on the power cable, but I will.  It could be a supply problem but I personally put that as “low” likelihood.

3). One thing that I forgot to mention is that when this problem occurred, I homed the filter wheel and then defocused the scope to confirm that nothing was blocking the pupil.

4).  I tested the system using both 1x1 and 2x2 binning.  When I posted this data, I goofed and pulled up one of the 2x2 images to compare with a 1x1 image.  Binning makes no difference in the results and I apologize for the mistake.

5)  I normally power up my system each night and completely power it down in the morning.  That means that the power is normally off every day so that the systems does a “clean boot” every evening.

6) I tried EZCap to look at the camera output but for some reason, it won’t boot.  It complains about a missing DDL.  I don’t have patience to debug that kind of nonsense so I used Maxim DL to confirm what I was seeing with SGP.

7) I looked at the ASCOM driver setting to make sure that nothing had changed and the driver is set just as it always has been.

8) Two nights ago pulling the cables fixed the problem but last night nothing we did could bring the camera back to life.  I switched power to the camera on and off and completely rebooted the system.

9)  Because I use ONAG for autocovariance focusing and covariance guiding, neither NINA nor Voyager will work on my system.  I am wed to SGP and this does not appear to be a problem with the imaging software.  Remember that I have ~ 300 nights x ~8 hours under my belt operating with SGP.

10)  We have tried both the old USB3 cable that has been on the system and a brand new USB3 cable and the problem persists with both cables.  To be clear, these are active USB3 cables since the required run length exceeds the USB3 3 m standard.  I don’t see any USB bandwidth transmission errors that normally look like a wrap-around or synch error in the image data.

I’ll hit up QHY about this to see what they have to say.  The problem with this kind of problem at a remote observatory is that it can take literally months to get the camera pulled, sent back for service, get it returned, and get it remounted and aligned on the scope.  It is much faster to simply order a new camera and hand carry it to Chile.  That’s why I already have a spare guide camera at the observatory and why I may even bring a 3rd guide camera down there when I go.  I am extremely unimpressed with that ATIK camera so I’ll swap it with a ZWO camera. My old FLI cameras ran for years without a glitch so I’m not very impressed that this QHY camera only made it one year before going flaky.  We’ll see how QHY will deal with it.


John
Like
MJNorman 1.81
...
· 
The missing DLL message and EZCAP not starting sounds like a telltale sign of a software configuration problem to me that might be at the root of your problems. Did EZCAP work previously?

I know it's a pain to mess with, but it's a lot less of a pain than having to make the trip to Chile.

Totally understand that swapping in a working camera is better / quicker than going through the warranty cycle. We've taken the same approach. 

M & J
Like
hbastro
...
· 
Hi John,
I had a number of similar strange issues associated with low voltage until I put DC-DC power supplies on critical components like cameras and computers. My setups are completely solar powered, as you know, so I am more susceptible to those issues when tied directly to batteries, even with separate local batteries for computer-camera, mount, dome.

Dave
Like
Rouzbeh 8.40
...
· 
That is very odd indeed! Must be frustrating.

I had an old QHY camera that some frames would download with the data all jumbled up. Ended up being the USB output board, they sent one out and it was fixed. Not sure if this is the case here. Looks closer to the sensor electronics it would seem (guessing).

The unplugging exercise may be a clue. I know being remote is challenging, for me, I like to eliminate all external sources - swap PCs if possible, I saw you tried USBs. I try direct connection to the PC and no hubs. My QHY600 runs off 13.6v coming out of the Powerwerx. 
Did you try changing modes, gain, and offsets?

If you eliminate all external sources then it may be the camera.

Good luck,

Rouz
Like
jhayes_tucson 22.76
...
· 
Michael & Jon Norman:
The missing DLL message and EZCAP not starting sounds like a telltale sign of a software configuration problem to me that might be at the root of your problems. Did EZCAP work previously?

I know it's a pain to mess with, but it's a lot less of a pain than having to make the trip to Chile.

Totally understand that swapping in a working camera is better / quicker than going through the warranty cycle. We've taken the same approach. 

M & J

The "missing DLL" message is only for EZCAP, which I don't use.  It has never worked and I have never needed it so it has nothing to do with the camera problem. Yeah, doing a factory repair is a very long cycle for a lot of reasons.  Having a scope sit useless through some of the best weather isn't worth simply replacing the camera ASAP and then having the broken one repaired on a parallel, non time critical path.

John
Like
jhayes_tucson 22.76
...
· 
·  1 like
Hi John,
I had a number of similar strange issues associated with low voltage until I put DC-DC power supplies on critical components like cameras and computers. My setups are completely solar powered, as you know, so I am more susceptible to those issues when tied directly to batteries, even with separate local batteries for computer-camera, mount, dome.

Dave

Thanks Dave.  I am aware that the QHY cameras are sensitive to input voltage so we'll get that measured next.  At least we have some techs available at Obstech to help out. Your remote systems are even harder to deal with when you aren't around.

John
Like
jhayes_tucson 22.76
...
· 
·  1 like
Rouz Astro:
Did you try changing modes, gain, and offsets?

Oh yeah, I tried everything without success.

John
Like
skybob727 6.08
...
· 
Hi John,

I know this dosen't help, but why did you stop using the FLI ML-16803. I use the PL-16803 and it's unlikely I will go QHY or ZWO. Just because it's a 9um chip
dosen't mean it can't get really good detail, and the chip is much bigger then the 455 chip, and it's very unlikely the FLI camera will ever have issues.
Like
Overcast_Observatory 20.43
...
· 
·  1 like
So you didnt try a USB 2 cable?  Pretty painless to try that.  USB 3 is a terrible interface for astrocameras.  It can work for months or years to suddenly have intermittent failures.  Bill mentioned it earlier and you only replied that you are not seeing any transmission errors.   I'd still try it as it just might be the easiest solution to your problem.  I cant tell you how many times I have had issues with USB3, and the issues range from not being able to download, to corrupted frames, to just about any form of weirdness.  I've sent countless tech support messages to ZWO and QHY over the years trying to get USB3 to be reliable. 

I can tell you that I have had zero file issues with any of my cameras with respect to USB2.0.  Worth a try.
Like
MJNorman 1.81
...
· 
So you didnt try a USB 2 cable?  Pretty painless to try that.  USB 3 is a terrible interface for astrocameras.  It can work for months or years to suddenly have intermittent failures.  Bill mentioned it earlier and you only replied that you are not seeing any transmission errors.   I'd still try it as it just might be the easiest solution to your problem.  I cant tell you how many times I have had issues with USB3, and the issues range from not being able to download, to corrupted frames, to just about any form of weirdness.  I've sent countless tech support messages to ZWO and QHY over the years trying to get USB3 to be reliable. 

I can tell you that I have had zero file issues with any of my cameras with respect to USB2.0.  Worth a try.

The QHY 600M is a USB 3.0 device and requires a USB 3 cable ...

Like
rockstarbill 11.02
...
· 
·  1 like
Michael & Jon Norman:
So you didnt try a USB 2 cable?  Pretty painless to try that.  USB 3 is a terrible interface for astrocameras.  It can work for months or years to suddenly have intermittent failures.  Bill mentioned it earlier and you only replied that you are not seeing any transmission errors.   I'd still try it as it just might be the easiest solution to your problem.  I cant tell you how many times I have had issues with USB3, and the issues range from not being able to download, to corrupted frames, to just about any form of weirdness.  I've sent countless tech support messages to ZWO and QHY over the years trying to get USB3 to be reliable. 

I can tell you that I have had zero file issues with any of my cameras with respect to USB2.0.  Worth a try.

The QHY 600M is a USB 3.0 device and requires a USB 3 cable ...

It can use USB2 cables just fine. That is how I run mine.
Like
Overcast_Observatory 20.43
...
· 
·  1 like
Michael & Jon Norman:
So you didnt try a USB 2 cable?  Pretty painless to try that.  USB 3 is a terrible interface for astrocameras.  It can work for months or years to suddenly have intermittent failures.  Bill mentioned it earlier and you only replied that you are not seeing any transmission errors.   I'd still try it as it just might be the easiest solution to your problem.  I cant tell you how many times I have had issues with USB3, and the issues range from not being able to download, to corrupted frames, to just about any form of weirdness.  I've sent countless tech support messages to ZWO and QHY over the years trying to get USB3 to be reliable. 

I can tell you that I have had zero file issues with any of my cameras with respect to USB2.0.  Worth a try.

The QHY 600M is a USB 3.0 device and requires a USB 3 cable ...



No it doesn't.
Like
wittinobi 0.90
...
· 
...or try to limit your usb3 speed to a lower range.

edit:
oh sorry, that of course only works if your camera is still available on your system.
Edited ...
Like
Rouzbeh 8.40
...
· 
If possible to install NINA on a separate laptop (fresh drivers) with an independent power source and  cable.
If that fails then you know its the camera.
Like
jdowning 0.00
...
· 
·  1 like
I have some experience with the QHY600M (early bird edition).    Two of them were put into service several years ago.    The QHY camera USB3 connection first negotiates the connection with USB2 signaling.   This presents random problems with some USB3 hubs and amplified extension cables.   By the way, the Tripp Lite USB3 active extension cable seems to be the "least worst."    I'd reconnect the USB3 camera cable to a USB2 port on the PC.   Keeping the signaling USB2 may do the trick.   You will still be subject to the 5 m length limitation without an active repeater or powered hub in the middle.   The QHY thermal controllers have had issues which can cause a failure in the image capture circuitry.   I returned one camera to QHY for exactly that reason.     It does sound to me like the fault is likely in the camera which will require a factory return.   Last May I returned a QHY128 to the factory for repair.    I'm still waiting for a diagnosis and repair / replacement.   I did get a note a week or two ago indicating that they were starting to work on it.....    The QHY600 is an excellent camera but for a remote site where everything has to work all the time I'd choose something else.   I have three OTAs at Sierra Remote.   All  three use Moravian C3-61000EC PRO cameras which have performed perfectly every night.   And they do image every night.    

Just my .02,

John
Edited ...
Like
jdowning 0.00
...
· 
One more thing - It does sound like you are on track to replace the QHY600 with something else.    Probably with another IMX455 sensor camera.   There are two versions of the IMX455 - the commercial version and the industrial version.    QHY, to their credit, uses the industrial version.  Less expensive cameras use the commercial version.   The cost difference in the sensors is substantial.    The difference is in the mounting (ceramic vs aluminum) and other details which are primarily duty cycle and thermal cycle related.    The commercial version, according to Moravian, is suitable for imaging less than 300 hours / year.   The industrial version for far more.    Sensors demise (and this may have been what happened to your camera) is driven by thermal cycles and how fast the camera cools and warms up.   For your application in Chile (I'm jealous!) the industrial version is the better fit.

John
Edited ...
Like
gregm 0.00
...
· 
I would have thought EZCAP should have worked. Perhaps install all in one driver, tick 5III and EZCAP as administrator, reboot and run as administrator 
(use the latest beta version as QHY rarely seem to update from beta even though it’s been out there for many months)

EZCAP also updates any needed firmware on the camera 

also, active cables are just a cable with a two port hub in the middle of the cable. Sometimes they also need a bit of extra power if your computer USB port can’t supply it.
Like
 
Register or login to create to post a reply.