Has anyone seen this problem before? QHYCCD QHY600PH M · John Hayes · ... · 52 · 3175 · 10

jhayes_tucson 22.75
...
· 
Bob Lockwood:
Hi John,

I know this dosen't help, but why did you stop using the FLI ML-16803. I use the PL-16803 and it's unlikely I will go QHY or ZWO. Just because it's a 9um chip
dosen't mean it can't get really good detail, and the chip is much bigger then the 455 chip, and it's very unlikely the FLI camera will ever have issues.

Bob,
I agree that the FLI cameras are superb and I happen to have two FLI-ML16803s on the shelf in my shop.  I even started with that camera on this scope.  However, there were some solid reasons to go with the IMX455 sensor with this scope in Chile.  I've given a number of talks on this subject.  One is here:  https://www.optics.arizona.edu/news-events/events/opti617-public-talk-john-hayes-optics-adventures-during-pandemic-engineering-remo.   You can get a brief intro to the subject in this write up here:  https://www.optics.arizona.edu/news-events/events/opti617-public-talk-john-hayes-optics-adventures-during-pandemic-engineering-remo.  

John
Like
jhayes_tucson 22.75
...
· 
I have some experience with the QHY600M (early bird edition).    Two of them were put into service several years ago.    The QHY camera USB3 connection first negotiates the connection with USB2 signaling.   This presents random problems with some USB3 hubs and amplified extension cables.   By the way, the Tripp Lite USB3 active extension cable seems to be the "least worst."    I'd reconnect the USB3 camera cable to a USB2 port on the PC.   Keeping the signaling USB2 may do the trick.   You will still be subject to the 5 m length limitation without an active repeater or powered hub in the middle.   The QHY thermal controllers have had issues which can cause a failure in the image capture circuitry.   I returned one camera to QHY for exactly that reason.     It does sound to me like the fault is likely in the camera which will require a factory return.   Last May I returned a QHY128 to the factory for repair.    I'm still waiting for a diagnosis and repair / replacement.   I did get a note a week or two ago indicating that they were starting to work on it.....    The QHY600 is an excellent camera but for a remote site where everything has to work all the time I'd choose something else.   I have three OTAs at Sierra Remote.   All  three use Moravian C3-61000EC PRO cameras which have performed perfectly every night.   And they do image every night.    

Just my .02,

John

John,
I use a Tripp Lite USB3 active 5m cable to connect this camera directly to the PC.  I've swapped cables and I'm convinced that this isn't a cable problem.  I'm now on the 3rd night since this problem first appeared and I booted the system tonight to see how it would behave.  Bam... it looks like it's working again!  This is very strange.  For a year, it worked without a glitch and now it's a crap shoot.  Intermittent is extremely frustrating and I'm losing trust in this camera!  I was hoping for a simple fix.  Maybe QHY will have something but I'm still waiting on an answer.

John
Like
ItalianJobs 0.00
...
· 
·  1 like
Michael & Jon Norman:
Hi John.

We may have seen something similar a few nights ago. I see you're using Maxim DL and have tried SGP.

We are using Voyager for image capture (and most everything). For the most part, it has been working very reliably. The other night, though, it started recording "dark" frames that may have been very similar to what you're seeing. We haven't looked too closely at the bad frames. They weren't all zeros, but I'm not sure if they were recording any meaningful signal. We'll look at some of them to gain some more insight. The strange thing is that it seemed to happen on a filter change. i.e. a long sequence of 180s L frames was fine, but shifting to 300S R/G/B frames resulted in the errant behaviour.

Voyager wasn't re-started before running the night's sequence when this happened.

Re-starting Voyager (and re-booting the mini-pc) seemed to cure it.

I haven't reported it to Voyager as it hasn't happened since and it seems better practice to re-start the imaging software each night. I suspect some sort of "memory leak" or driver initialization problem as opposed to a power / USB / other hardware issue. We do use a Pegasus Ultimate Powerbox, so we can remotely power cycle and "reset" USB connections.

I know this is a lot of detail for a problem that might have nothing to do with what you're experiencing, but my point is that it could as easily be a software "glitch" as hardware.

QHY suggests using their software to test their cameras ... I'd try using EZCap to test it. If you still have problems, it's likely time to get QHY support. They've been really good at diagnosing problems remotely for us.


​​​​Hope this helps.

Michael and Jon Norman

Is not a Voyager problem but a QHY driver problem .. be sure to use the latest beta from their website. Usually they fix driver so many times during the year.
Restarting the PC daily base help if you have a 24H pc observatory running with this camera.

All the best
Leonardo
Edited ...
Like
Wjdrijfhout 4.89
...
· 
Hi John,

Looking at your images, it looks like the 'wonky' one is just receiving only a tiny fraction of the light that it should get. And you've checked already all settings on gain, offset, exposure, etc and verified any potential filter blockage. So yes, a camera and/or software fault seems like a likely explanation.

Just to drop in a completely different angle though. If I'm not mistaken, you're using on-axis guiding, isn't it? So you will have a dichroic beam splitter in your light pathway. How critical is such a splitter for a slight mis-alignment? Or is there any chance that the filter got damaged?
Have you checked the ONAG filter? If there is a little mechanical play in it, could that explain that sometimes the light beam is just not reflected (enough) into the camera? If it is mechanical, it would explain why sometimes it works and sometimes it doesn't, or start working again when you physically touch the system by pulling out cables.
If it is an issue with the ONAG, it might also explain why you already went through 2 Horizon cameras for guiding in one year. Regardless make/build quality/grade of cameras/sensors, three broken cameras in one year sounds quite exceptional.

Perhaps it makes no sense what I'm saying here, I have no experience with ONAG at all. Just a thought.

Good luck troubleshooting the issue.

Willem Jan.
Like
skybob727 6.08
...
· 
John Hayes:
Bob Lockwood:
Hi John,

I know this dosen't help, but why did you stop using the FLI ML-16803. I use the PL-16803 and it's unlikely I will go QHY or ZWO. Just because it's a 9um chip
dosen't mean it can't get really good detail, and the chip is much bigger then the 455 chip, and it's very unlikely the FLI camera will ever have issues.

Bob,
I agree that the FLI cameras are superb and I happen to have two FLI-ML16803s on the shelf in my shop.  I even started with that camera on this scope.  However, there were some solid reasons to go with the IMX455 sensor with this scope in Chile.  I've given a number of talks on this subject.  One is here:  https://www.optics.arizona.edu/news-events/events/opti617-public-talk-john-hayes-optics-adventures-during-pandemic-engineering-remo.   You can get a brief intro to the subject in this write up here:  https://www.optics.arizona.edu/news-events/events/opti617-public-talk-john-hayes-optics-adventures-during-pandemic-engineering-remo.  

John

Hi John, Thanks for the links.

I had planned to ask you what you thought about the QHY600, but you just answered my question. I once asked someone you may know, he uses to make filters and holds a Ph.D. in near-infrared spectroscopy, about what he thought about QHY cameras, granted this was a long time ago and I'm sure there much better now. He said when they fail to just throw them away rather than trying to fix then. I would very much like to try the IMX455, but not yet. Not sure if you have heard, but FLI is going introduce their new BioLine cameras sometime next year and some of the chips will be the IMX571, IMX492 and the IMX455. That will be the day I may switch to CMOS, will see, not sure yet if I want to go with a smaller chip then the 16803 even if it does have better SNR. 

Anyway, hope you get the issue fixed soon, after all, you're imaging from one of, if not the best places there is.  I know someone that helped Astro-Physic setup their equipment at the Las Campanas Observatory in Chile.

Bob
Like
jhayes_tucson 22.75
...
· 
Bob Lockwood:
John Hayes:
Bob Lockwood:
Hi John,

I know this dosen't help, but why did you stop using the FLI ML-16803. I use the PL-16803 and it's unlikely I will go QHY or ZWO. Just because it's a 9um chip
dosen't mean it can't get really good detail, and the chip is much bigger then the 455 chip, and it's very unlikely the FLI camera will ever have issues.

Bob,
I agree that the FLI cameras are superb and I happen to have two FLI-ML16803s on the shelf in my shop.  I even started with that camera on this scope.  However, there were some solid reasons to go with the IMX455 sensor with this scope in Chile.  I've given a number of talks on this subject.  One is here:  https://www.optics.arizona.edu/news-events/events/opti617-public-talk-john-hayes-optics-adventures-during-pandemic-engineering-remo.   You can get a brief intro to the subject in this write up here:  https://www.optics.arizona.edu/news-events/events/opti617-public-talk-john-hayes-optics-adventures-during-pandemic-engineering-remo.  

John

Hi John, Thanks for the links.

I had planned to ask you what you thought about the QHY600, but you just answered my question. I once asked someone you may know, he uses to make filters and holds a Ph.D. in near-infrared spectroscopy, about what he thought about QHY cameras, granted this was a long time ago and I'm sure there much better now. He said when they fail to just throw them away rather than trying to fix then. I would very much like to try the IMX455, but not yet. Not sure if you have heard, but FLI is going introduce their new BioLine cameras sometime next year and some of the chips will be the IMX571, IMX492 and the IMX455. That will be the day I may switch to CMOS, will see, not sure yet if I want to go with a smaller chip then the 16803 even if it does have better SNR. 

Anyway, hope you get the issue fixed soon, after all, you're imaging from one of, if not the best places there is.  I know someone that helped Astro-Physic setup their equipment at the Las Campanas Observatory in Chile.

Bob

Bob,
I actually think that in spite of this problem, the QHY600M-PH is a very good camera.  It is a very worthy replacement to the FLI-ML16803 and it costs a little less than half of what I paid for my last FLI camera.  Up until now, it has been very reliable so I hope that I can identify what is going on with it.  I still haven't ruled out a fault with the observatory line voltage regulator or something else in my system that might be messing up the supply voltage to the camera. This is a problem with a remote observatory.  It's a LOT harder to trouble shoot things when things go wrong.  That's why I posed this problem as a question.

John
Like
jhayes_tucson 22.75
...
· 
Now I'm really puzzled.  The camera has worked just fine for the last two nights.  If it weren't for the fact that the problem went away on the first night after we swapped cables, I would conclude that the camera window had frosted over.  This camera might be old enough to require a new desiccant pellet and if the humidity were sufficiently high, frost could have formed.  I don't know.   I didn't mention that when I was first having the problem, I slewed over to the moon and took a few images, which were sharp enough to rule out frost.  If it keeps working, I'm ok with that but this problem will be in the back of my mind going forward.  Weird.


John
Like
rockstarbill 11.02
...
· 
John Hayes:
Now I'm really puzzled.  The camera has worked just fine for the last two nights.  If it weren't for the fact that the problem went away on the first night after we swapped cables, I would conclude that the camera window had frosted over.  This camera might be old enough to require a new desiccant pellet and if the humidity were sufficiently high, frost could have formed.  I don't know.   I didn't mention that when I was first having the problem, I slewed over to the moon and took a few images, which were sharp enough to rule out frost.  If it keeps working, I'm ok with that but this problem will be in the back of my mind going forward.  Weird.


John



The camera has no desiccant pellets inside of the chamber, just the desiccant tube you thread into the side of the camera body. If that isn't installed, I would get it installed. If it is instated, I would have the local team dump and fill the tube.
Like
Rouzbeh 8.40
...
· 
John HayesNow I'm really puzzled.  The camera has worked just fine for the last two nights.  If it weren't for the fact that the problem went away on the first night after we swapped cables, I would conclude that the camera window had frosted over.  This camera might be old enough to require a new desiccant pellet and if the humidity were sufficiently high, frost could have formed.  I don't know.   I didn't mention that when I was first having the problem, I slewed over to the moon and took a few images, which were sharp enough to rule out frost.  If it keeps working, I'm ok with that but this problem will be in the back of my mind going forward.  Weird.


John

Plug.jpg

You'll need this plug and some desiccant pellets.

https://www.amazon.com/Wisedry-Rechargeable-Desiccant-Indicating-Reusable/dp/B01M7V19JE/ref=sr_1_11?crid=TANO8OCBM4MK&keywords=dessicant&qid=1667665432&sprefix=dessican%2Caps%2C148&sr=8-11

I use these and now the plug is on permanently. 
There is nothing inside the camera. My QHYs usually lasted 6 months before a recharge.




--Rouz

Like
Rouzbeh 8.40
...
· 
john,


Also, I'm don't know about the imaging software you are using but with NINA, the chamber -Relative Humidity- is constantly displayed. I try to keep mine low, under 50%. The desiccant has recently been charged and its at 24% despite it being 80% outside.


--Rouz
Like
jhayes_tucson 22.75
...
· 
·  1 like
The scope has been in Chile for a year and that camera is roughly 1.5 years old.  The chamber has never been "dried out" so that's a good suggestion.  Still, there's not enough evidence to suggests that frost was the problem.

John
Edited ...
Like
scott1 5.29
...
· 
Hi John
 Clicking on the two links you show in your response to Bob, I get "page not found"
Is there a new link?

Scott
Like
dkoslicki 1.51
...
· 
Scott Lockwood:
Hi John
 Clicking on the two links you show in your response to Bob, I get "page not found"
Is there a new link?

Scott

@Scott Lockwood I found a talk with the same title via YouTube: https://www.youtube.com/watch?v=te4UVYi6n44&ab_channel=UAOSC
Like
Rouzbeh 8.40
...
· 
John Hayes:
The scope has been in Chile for a year and that camera is roughly 1.5 years old.  The chamber has never been "dried out" so that's a good suggestion.  Still, there's not enough evidence to suggests that frost was the problem.

John

*It does look too even to be frost which usually starts at the center of the window being furthest away from the heated edges of the glass.

Rouz
Like
jhayes_tucson 22.75
...
· 
Rouz Astro:
*It does look too even to be frost which usually starts at the center of the window being furthest away from the heated edges of the glass.

Agreed.
Like
jhayes_tucson 22.75
...
· 
Scott Lockwood:
Hi John
 Clicking on the two links you show in your response to Bob, I get "page not found"
Is there a new link?

Scott

DOH!  Yeah, those link were screwed up and I should have tried them first.  I see that he UA talk got moved to the address that David found.   The other link was suppose to take you to this post:  https://www.astrobin.com/txk760/F/.  I'm really sorry for the error!  I'll try to be more careful the next time...

John
Like
rockstarbill 11.02
...
· 
Michael & Jon Norman:
The missing DLL message and EZCAP not starting sounds like a telltale sign of a software configuration problem to me that might be at the root of your problems. Did EZCAP work previously?

I know it's a pain to mess with, but it's a lot less of a pain than having to make the trip to Chile.

Totally understand that swapping in a working camera is better / quicker than going through the warranty cycle. We've taken the same approach. 

M & J



I get this same issue with the 64 bit EZCap client. The 32 bit one works just fine for me.
Like
distantnova 0.90
...
· 
·  1 like
I had a different problem with my QHY600PH-M (random horizontal banding noise, like this guy: https://www.cloudynights.com/topic/700917-qhy168c-randomized-horizontal-banding/) and it too had to be sent back for repairs (from Chile as well...). I have a suspicion that the data during the USB transfer has neither error detection nor error correction, making me question the quality of the data and the applicability for remotely controlled telescopes that must operate reliably. I'm getting a replacement now and I pray to god that similar errors do not occur again
Like
jhayes_tucson 22.75
...
· 
·  1 like
I had a different problem with my QHY600PH-M (random horizontal banding noise, like this guy: https://www.cloudynights.com/topic/700917-qhy168c-randomized-horizontal-banding/) and it too had to be sent back for repairs (from Chile as well...). I have a suspicion that the data during the USB transfer has neither error detection nor error correction, making me question the quality of the data and the applicability for remotely controlled telescopes that must operate reliably. I'm getting a replacement now and I pray to god that similar errors do not occur again

Your problem doesn’t look like the USB transmission errors that I’ve seen.  Where did you get the camera?  QHY supplies both a power supply and the power supply cable with the camera and you say that you got neither.  Did you put a DVM on the supply to measure the supply voltages?  Was it a switching supply?  Did you ever try the camera with the proper supply and cable?  It sounds like it ran fine when you used a battery for the power supply, which makes it sound like a potential supply issue.  I have two QHY600M-ph cameras and neither has shown the kind of problem that you experienced.   As I’ve said, the camera on my scope in Chile has worked flawlessly for about a year before this problem popped up.  Lately, it has been working just fine, which has me wondering what the heck happened.  It too could be related to a power problem.  I’m going to take some spare power supplies with me when I head back down there in another month or so.

John
Like
barnold84 10.79
...
· 
·  1 like
John Hayes:
Your problem doesn’t look like the USB transmission errors that I’ve seen.  Where did you get the camera?  QHY supplies both a power supply and the power supply cable with the camera and you say that you got neither.  Did you put a DVM on the supply to measure the supply voltages?  Was it a switching supply?  Did you ever try the camera with the proper supply and cable?  It sounds like it ran fine when you used a battery for the power supply, which makes it sound like a potential supply issue.  I have two QHY600M-ph cameras and neither has shown the kind of problem that you experienced.   As I’ve said, the camera on my scope in Chile has worked flawlessly for about a year before this problem popped up.  Lately, it has been working just fine, which has me wondering what the heck happened.  It too could be related to a power problem.  I’m going to take some spare power supplies with me when I head back down there in another month or so.

John

John,

After reading people pondering on power supply issues, I think about the power grid quality in general. I admit, I don't know about the conditions down there in Chile and how the hosting company deals with power supply for your scope. I am just projecting some personal experience from the Caribbean onto it: I would use a good and strong line filter to protect the equipment from misbehaviour or damage.
I know you thoroughly prepared for this endavour and if that point is checked already, just dismiss my comment on it.

Björn
Edited ...
Like
jdowning 0.00
...
· 
·  1 like
According to the techs at Sierra Remote ( about 60 piers)  the single most common failure mode are the switching power supplies that are included with cameras, filter wheels and the like.  These inexpensive supplies were never designed for the rigors of exposure to the elements.    I have a suggestion for remote setups.    Don't use them.   At SRO my three OTAs each have a Pegasus Ultimate V2 box for power distribution.    The power supply for each Pegasus box is an Astron 25 or 30 amp rack mounted switching power supply.   Linear would be better but more expensive.   The Astron switchers are fine - and they have a voltage adjustment pot so that the voltage at the OTA can be adjusted to just under 13 vdc.   Other similar supplies (built for radios) supply too much voltage (as high as 14.5 vdc) and are not adjustable.    I've never had a power supply problem - ever.    This is an inexpensive and highly reliable solution for remote setups.

Just my .02
Like
distantnova 0.90
...
· 
John Hayes:
Did you ever try the camera with the proper supply and cable?


Yeah, that was what it was installed with (the forum post is not by me). Also tried a different USB cable and powered hub (without one, the camera would disconnect randomly).

Björn:
I think about the power grid quality in general.


Me too, but it's difficult to judge/measure this from here. The site self-supplies with solar power (DeepSkyChile).

I don't understand however why a power issue would affect the data integrity (unless it affects the sensor). That seems like more of a lack of a proper protocol/firmware/software issue.
Like
jhayes_tucson 22.75
...
· 
Björn:
After reading people pondering on power supply issues, I think about the power grid quality in general. I admit, I don't know about the conditions down there in Chile and how the hosting company deals with power supply for your scope. I am just projecting some personal experience from the Caribbean onto it: I would use a good and strong line filter to protect the equipment from misbehaviour or damage.
I know you thoroughly prepared for this endavour and if that point is checked already, just dismiss my comment on it.


The observatory mostly runs from LiOn batteries at night but if the batteries get depleted it's possible for a diesel generator to kick in.  I don't know how things are wired but my feeling is that the line current is actually pretty clean.  Remember that they have millions of dollars worth of equipment running there each night and no one wants their gear damaged by voltage spikes.  My scope runs through a UPS to a step down transformer.  I've had one Power Werx power supply blow out but my other supplies have been fine.  I didn't equip the system to be able to remotely report power supply health so it's hard to say if the supplies are a part of the problem.  I never had a supply fail on my C14 system in New Mexico--over many years, so I didn't consider it to be significant risk.

John
Like
distantnova 0.90
...
· 
Update on my situation:

Because we experienced simultaneous telescope mount and camera disconnection failures recently, the reason may not lie with the camera after all, but with the motherboard of the PC.

The system is using an ASUS TUF GAMING B550M-PLUS WIFI II, which many people seem to be having USB connectivity and possibly power surge problems with, which may explain the irregular connectivity problems and data corruption:

https://www.pcmag.com/news/amd-locates-root-cause-of-usb-issues-on-b550-x570-motherboards

I had similar problems with my home computer, which coincidentally uses the similar ROG STRIX B550-F GAMING, after updating the BIOS there to the latest version, my external microphone now works again.

We hope this will also apply to the camera - we will continue to test during the next nights


 If any of you have similar problems, maybe check whether you have one of these motherboards
Edited ...
Like
PeterGoodhew 4.13
...
· 
John Hayes:
A couple of months ago my ATIK Horizon 2 guide camera started acting "wonky."  The background level suddenly went way up.  I mitigated it by subtracting a dark frame but it has screwed up the SNR of my guide signal.  Fortunately it still works well enough to guide.  Fortunately, I already have a replacement camera sitting at the observatory ready to be installed.  This is the second failure of a Horizon 2 in less than a year.

Just last night my main imaging camera, a QHY600PH M, went "wonky" in a different way.  It almost looks as if the 16 bit image data has been bit-shifted left by ~ 4-5 bits--but that's not what is happening. We fixed it last night by pulling and reconnecting cables but I'm out of luck tonight.  I can't get it to work correctly.  Here is a comparison of two images of the same object through a LUM filter at the same cooling temperature with the same exposure before and after the "glitch".

Screen Shot 2022-11-02 at 10.10.11 PM.png

Here are histogram plots of both images.  First, before:

Screen Shot 2022-11-02 at 9.05.45 PM.png

And second, after the failure:

Screen Shot 2022-11-02 at 9.48.08 PM.png

There is no dew and the sky is clear.  This sure looks like a camera problem.  As I said, we fixed it last night by pulling cables and reconnecting them.  I've also tried a brand new USB3 cable with no improvement.  We haven't yet measured the voltage at the camera so that might be a problem.  This particular data was taken using MDL but I get the exact same results using SGP (my normal imaging program).

My question:  Has anyone else seen this problem and is there a simple fix?  I want to exhaust all possibilities before jumping on a plane to go to Chile with a replacement camera.  With two dead cameras, I'm already mentally packing my bags.


John

John, I'm having similar problems with a new QHY600M-SBFL Mono (also at a remote site).  Did you get this resolved?  And if so, how?

Peter
Like
 
Register or login to create to post a reply.