1.43
#...
·
·
1
like
|
|---|
|
Let me explain myself, dudes. Is there some kind of rule where you see how many darks and flats are convenient for a stack? Here each one does the number of dark and flat takes that they want/consider. Some say that more quantities are better (leaving the imagination open to an indefinite practically unlimited number, 20 better than 2 and 200 better than 20). Others that the same as the number of lights you have, others that half the lights... but it is known somehow if there is some kind of consensus that from such quantity the benefit of doing more is no longer noticeable and so on with bias and flats? Has anyone read any sort of study or any dissertation? Are there charts somewhere...? Bias and flats are two other points: With dedicated cameras you can make libraries and already have the darks and the biases (whoever makes them) made at home for up to a year (I don't really understand that expiration date that runs around the network like a dogma) well of course you can go adding lights of sessions and sessions but you already have your library of darks and although you can add more number of them, maybe that it is not the most common way to do. That is why I ask if there is any study or something about it, I have not found anything and with the amount of things that are talked about pixels, efficiencies, noise and others, I am even surprised that there is nothing or that it is not easy to find. I also assume that the sensors are becoming more modern and effective and perhaps it is an increasingly less critical parameter, but it is not for that reason banal. Do you do dark-flats? This is another eternal discussion or trend. There are those who say that with a dedicated one you don't have to do bias but darks and dark flats and others who say that dark-flats don't and you have to do bias. Then there are those who say that the flats have to be less than 1 second and those who say that they have to be longer (mine made with the RBFocus flat panel are 5 seconds and I do dark-flats and do not bias just because someone told me when I asked for information when I moved from a non-modified DSLR to a dedicated cooled camera). In the end, the impression I get is that there is nothing clear and everyone does what they can or reads but there are also no sources that you can consider reliable to inform you or I do not know. Yes, there are forums and youtubers but... is your source reliable or have you also read/seen it somewhere? Hope this thread could became a discussion reference one where some of you could link webs, other threads or manuals that you trust in. Clear skies and thanks! |
3.15
#...
·
·
1
like
|
|---|
|
Hi Daniel, Just sharing what I do for calibration files Darks - 50 each for the different exposure lengths and gains for my darklibrary that gets updated every 6 months. I always should at the same temperature setting so this is not an issue. Do this on cloudynights inside so relatively easily achieved in 2 nights or so. Flats - 75 for each filter used every night, always the same exposure time using the trained flats feature in NINA together with a flat panel. The same exposure length is important as this allows me to have master darks for my flats aswell. I do this many because the flatpanel allows me to do 0.3s exposures for LRGB filters and for NB the longest exposure is 1.2s. This way shooting flats for all my filters takes a maximum of 5 minutes. Think I read somewhere 20 is probably sufficient but as it takes so little time to make more I do that. Flat for Darks - 50 and the go in the Dark library No bias I do these things after reading many forums and discussions but as for a scientific support for why I am also in the dark (pun intended) but it seems to work for me. Not an expert on this but the choice between bias or darks for flats is very heavily influenced by the camera you are using. Some cameras produce weird results when generating bias files e.g. extremely short exposures, others have strange outcomes when using darks of insufficient exposure time. For some very low dark current cameras it appears there is no difference between a dark and bias. For my ZWO ASI183MM Pro I have to use darks as it has a real heavy amp glow, but I read that for the 2600MM for instance if cooled sufficiently bias alone is enough as dark current is insignificant for this camera. I read mostly about ZWO cameras as those are what I use. If anyone has mathematical support/theory for number of calibration files to use I am also interested, but also foreseeing Uni math class flashbacks. CS Sean |
2.15
#...
·
·
3
likes
|
|---|
|
I usually do only 20 darks (library by camera/sensor and exposure time, temperature must be the same as the one used for lights) and 30 flats and 30 dark flats (these need to be done every time a session is completed and should never be reused. Temperature does not matter because these are very short exposures). I stopped doing bias because the flats and dark flats work better than bias I hope this helps Clear skies Alicia |
22.40
#...
·
·
18
likes
|
|---|
|
It's probably not as often as it seems but I feel like I answer this question on a monthly basis. First, Berry and Burnell have a good discussion about image calibration in their book, "The Handbook of Astronomical Image Processing" (https://www.amazon.com/Handbook-Astronomical-Image-Processing/dp/0943396824). They discuss the "Rule of 5" to limit added noise in a single sub to 10% of a single bias frame but they don't address the statistics of stacked images. Back in April of 2016, I worked out the statistics of calibration noise when subtracting unwanted additive signals in a stacked image. Additive signals include both darks and bias signals. You can find that thread here in post #33: https://www.cloudynights.com/topic/534493-the-statistics-of-image-calibration/page-2?hl=%20calibration. (You can safely ignore the rest of the discussion in that thread. There are a lot of incorrect statements about this stuff that muddle up the conclusions. Also don't respond on that thread--I won't see it). It is possible to modify the results to cover flat division as well, but I didn't do that. The equation shows a couple of interesting things. 1) The amount of noise reduction depends on both the number of dark frames used to calibrate each sub AND the number of light frames in the stack. 2) Past a certain point, using more dark or bias frames achieves only incremental improvement. This is the rule of diminishing returns. I've attached a plot below showing the result for a stack of up to 100 subs. We can quickly see a couple of things. 1) After about 15 subs in the stack, the noise contribution diminishes very slowly. 2) Keep in mind that for most cameras, dark signal is typically a small value, which means that keeping the added noise due to the dark subtraction to below 30% - 50% is sufficient to keep the calibration noise below the photon noise in the stack. 3). For a stack of say 30 subs, going from 16 darks to 32 darks only gains an additional decrease in noise of another 8% (or so). Using 64 darks only gains an additional 3%-4%, which will be completely unnoticeable in the result. Working with a larger stack makes things better, but not by enough to justify the effort needed to take more dark data. For those who are using exposures in the range of 600s-1200s, taking darks can require a significant amount of time. There's no reason to spend more time in the dark than necessary taking calibration data. CONCLUSION For most of us doing traditional long exposure imaging with a stacks the range of 15 -100 subs, taking more than about 16 darks is a waste of time. The same applies to bias data as well. It won't hurt anything to use 50-100 dark or bias frames, but you are kidding yourself if you think that it is improving your results. Rule for most situations: Use 16 darks to construct your master flat or master bias files and you'll be fine.  As for your other two questions: 1). Bias is important to insure that the proper offset is removed when you do calibration. No camera, even the new ones, has identically zero offset and that can be the cause of minor calibration errors. Bias is super easy to subtract so don't ignore it. 2). As long as your flat data is A) taken within the linear response region of the sensor response, B) your camera has fairly low dark current, and C) your flat exposures are less than a minute (or so) flat-darks are rarely needed. If you are doing sky flats, you might need dark flats; however, if you are using a light panel and your exposures are between 2s and 60s, you almost never need dark flats--for most modern CMOS cameras. John |
|
22.40
#...
·
·
1
like
|
|---|
Sean van Drogen: Boy, I don't know who is writing that stuff but it is simply not true. The IMX455 chip has pretty low dark current but it sure isn't zero. My typical exposure is 10 minutes with a QHY600M (which uses the same chip) and the dark current is quite obvious in the way of warm pixels. You can get away with ignoring dark calibration but you are putting the load on the stacking filter to remove all that stuff and that may be okay so as long as you understand what you are doing and you are using sufficient dithering. John |
|
7.46
#...
·
·
1
like
|
|---|
|
All the above, especially and specifically what John said above, is valid for modern CMOS cameras. Older CCD cameras would require bias frames to be taken and quite often indeed as they shall be used for cosmetic correction. Large CCDs tend to have significant time dependant variation in defects, both clusters and column defect. Personally, I've found that flat-darks can often be reused and dark libraries can last more than 1 year. With CMOS and to a lesser extent, CCDs. |
11.91
#...
·
·
2
likes
|
|---|
|
John - agree with all of what you said. But isn't the bias signal already contained in the dark frame? So using a master dark that does not have the bias subtracted from it renders the subtraction of bias irrelevant, wouldn't it? Of course, the dark frames would have to be at the same time/temperature. I do know that for some sensors, such as the one used in the 294MC/MM and possibly also the ASI 1600, the "bias" signal depends on exposure time, see for instance this analysis by John Upton: https://www.cloudynights.com/topic/636301-asi294mc-calibration-%E2%80%93-testing-notes-thoughts-and-opinions/ Subtracting a bias in these cases seems more trouble than it is worth so I simply use matched time/temperature darks. Attached, for reference his conclusions, specific of course, to this sensor:
Edit: Also a link to a Michael Covington article on the arithmetic behind calibration and why bias frames are not needed if flat darks and darks are being used and darks are not being scaled: http://www.covingtoninnovations.com/dslr/newdslr/#Arithmetic |
4.67
#...
·
·
1
like
|
|---|
|
With an asi294 I use just enough to take away the amp glow and whatever odd vignette there is with only about 10 dark, 10 flat and 10 dark flat up to 10-15 hours of light frames. You have to use them but you don't need to overwhelm the image. Simple experimenting will show when and how many you need to work. Why stack more than necessary? Too many flat frames can dampen images I think. With the asi533 color camera I do not use any. |
|
22.40
#...
·
·
1
like
|
|---|
Dan Kearl: Dan, The calibration equation is [(light+dark+bias) - (dark+bias)]/[flat+flat_dark+bias]. If flat_dark is ~0, that's not needed. Bias represents read noise+offset and if that total is ~0, then it's not needed either. However if it's not zero, the component in the denominator will cause a small error. If your calibration files look flawless, then you are good to go, but that's not always the case. I personally think that bias correction is super easy. It's trivial to take the data and then it's just a checkbox to do the calculation. How much trouble is that? The problem that John Upton is dealing with is a non-linear effect and I've heard that some cameras exhibit this behavior. The entire calibration process strictly requires a linear system. All bets are off if the camera is non-linear. I've heard that some CMOS cameras exhibit strange behavior with exposures below about 1 second and that's why I emphasized that flats must be taken within the linear range of the sensor. One problem with Upton's post is that he uses the term "Fixed Pattern Noise" with regard to bias data. That is incorrect. FPN is a specific symptom of PRNU, which is the spatial variation in responsivity between individual pixels. Any pattern that you may see in bias data is a variation in offset between pixels--not gain. John |
|
4.67
#...
·
|
|---|
|
John, I think you meant to reply to Arun H Bias frames do not work with the asi294 as far as I can tell and the Flat frames have to be calibrated with dark flat for APP to use. |
|
22.40
#...
·
|
|---|
|
Dan, You are correct. I don't know how your name got inserted--and I'm sorry that I didn't notice the error. I simply pressed "Quote Selected" on Arun's post and what you see is what popped up. Weird! (This is the second weird AB glitch that I've experienced today. I think that gremlins are messing with me.) OK, that implies that the ASI294 has some weird properties that fall outside the normal calibration process. John |
|
3.15
#...
·
·
1
like
|
|---|
John Hayes:Sean van Drogen: Well there you go. This seems to be the issue with reading random post on astro fora and not doing the proper research yourself. Thank you for pointing out my misinformation. CS Sean |
10.79
#...
·
·
2
likes
|
|---|
John Hayes: Hi John, Oh yes, it does have weird properties. The camera is sensitive to ambient temperature. The sensor has signfinicant "amp glow"/starburst. The behavior of this star burst correlates with the power of the cooler which correlates with the ambient temperature. I had the ASI294MM Pro and it turned out that even for long broad band exposures, the starburst didn't calibrate out well. The fact that I captured the darks in the house with ambient 19C and the light frames captured at ambient temp 0C was sufficient to let the sensor vary enough such that artifacts were still visible. That's what I got out after my analysis. (Therefore, I'll never ever buy a camera again with "amp glow") CS, Björn |
|
7.46
#...
·
·
1
like
|
|---|
| My ASI294MC Pro calibrates out beautifully with no issue whatsoever as reported by others. I guess the issue is with the monochrome version. |
|
11.91
#...
·
·
1
like
|
|---|
|
There is no problem with the monochrome version - the 294MM. I have taken many images now with that camera which you can see for yourselves. It calibrates out beautifully every time, not one single issue. Uncalibrated frames have amp glow, calibrated frames are clean. The things I do:
Doing this is mathematically consistent with how calibration works, accounting for how biases work in certain CMOS sensors. It is the same protocol I followed for the ASI1600 and my current ASI2600MC Pro and worked in all cases. All my darks were taken in my basement (heated to ambient home temperature) or garage, assuring that the camera was as light sealed as possible. I have taken darks at 4s, 60s, 180s, 300s, and 600s, and temps of -25, -20, and -10 C and calibrated L,R,G, B and all three NB filters. I have used it now with three different scopes - an 80mm SVX, a 92mm AP Stowaway, and a 200mm TS Optics Newtonian. In ambient temps of near 80F in summer down to -20F in Wisconsin winters. No issues. BTW, I know others who follow the same practice as I do, use the same camera and have no issues. Not understanding the math behind what is happening when you calibrate a frame and the subtleties of the linearity of CMOS sensors is probably the source of most of the issues I see. |
|
1.43
#...
·
|
|---|
John Hayes: Many thanks, @John Hayes, that's a great contribution! John Hayes: All right, not to skip bias frames. That's easy to do because I can create a bias library. I suppose that it's more or less than darks, I mean in the number of them to take. Ok, in my beginnings mates told me if the flat was with exposure more than 1 second then dark-flats were needed. One minute is a longer timeframe, and my flats are from 5s exposure. John Hayes: Excellent explained! Sean van Drogen: That was my problem too when reading information in different forums. In that case @John Hayes has not only given the explanation but has argued it well and has provided references. All my respect for his know-how and knowledge. Thank you so much and clear skies. Daniel. |
20.43
#...
·
·
2
likes
|
|---|
John Hayes:Sean van Drogen: One thing I have observed with respect to calibration of the IMX571/455 is that if you Bias calibrate your Darks you, the resulting Master Dark suffers suspect results. (Previoulsy stated clipping and this was incorrect). I've also seen similar issues when calibrating light frames with both Master Dark and Master Bias. Now I am only using a master bias for flat calibration. For light calibration I am using a Master Dark (uncalibrated) and a Master Flat (Bias Calibrated). I've found this to produce the most consistent results. I did find it interesting in your previous post that 16 darks and 16 bias were enough to properly calibrate most reasonable data sets. I've always made a master dark using 30 darks and 250 bias. Why? I dont know. Just the way I learned about it. I'm actually glad to see that you dont think it's totally necessary. Considering how large the files are for IMX455 I really hate the idea of having to capture and process so many frames for calibration. Pretty cool... |
|
22.40
#...
·
·
1
like
|
|---|
John Hayes:Sean van Drogen: Chris, If your master darks show zeros after bias subtraction that implies that A) the offset in your camera is set too low (in the electronics) or B) your camera is unstable, meaning that the bias is varying with time. It may also be possible to get that result if the temperature of the sensor is different between when you take the two sets of data. Under normal circumstances, you should not see zeros in your bias subtracted darks. I’ll have to go check my two QHY600M cameras to check that my camera doesn’t produce any zeros. I’ve never seen any symptoms indicating that this might be a problem in my calibrated images. I’ve been showing this calculation and the conclusions in various posts for the last 6 years but it takes time for the word to get around. Again, I think that if you calibrate two images—one using 30 darks with 250 bias frames and another using 16 darks and 16 bias frames, you will be very hard pressed to see much (if any) difference in the results. I should mention that another guy on CN actually made hundreds of measurements to confirm my calculations and his experimental data perfectly overlaid the prediction (shown in the plot that I posted above.) I don’t have that plot on my iPad or I’d post it. I can find it if anyone wants to see it. John |
|
22.40
#...
·
·
1
like
|
|---|
|
“Ok, in my beginnings mates told me if the flat was with exposure more than 1 second then dark-flats were needed. One minute is a longer timeframe, and my flats are from 5s exposure. “ —————————————- Daniel, Remember that dark signal grows with exposure time. Some CMOS sensors behave differently at short exposures so it is recommended to use exposures longer than a second or two when you take flats. If you properly expose the flats, the peak signal should lie in the range of 50%-75% of the peak value (in ADU). With sufficiently short exposure time—maybe under 1-2 minutes, the amount of dark current will be a tiny fraction of the total signal in the flat signal. Just as in the case of the bias signal, when the magnitude of the dark signal contained in the flat data becomes large enough, it will introduce calibration errors. The biggest need for flat-darks is when you are taking sky flats using pretty long exposures. Once you get over 1-2 minute exposures for your flats, that’s probably when to consider adding flat-darks.l I personally judge all this stuff by looking at how well the calibration process is working. Once the variation in signal across the calibration frame is less than a couple of percent, I’m good with it. It’s pretty obvious when there’s a problem and that’s when I go back to check the process to find out where the error is being introduced and that’s when I might consider adding flat-darks to see if that fixes the problem. So far, I haven’t seen any significant problems with any of my cameras (both CCD, and CMOS) introduced by not using flat-darks. John |
|
11.91
#...
·
·
1
like
|
|---|
|
I should mention... the reason I calibrate my flats with flat darks is not because of dark current, which is trivial at 4 s exposure and -10C. Rather it is to ensure that the correct Bias is subtracted. A flat-dark of the same exposure time as the flat will automatically have the correct bias encoded in. With the difference between how some CMOS sensors behave at short exposures (2 seconds or less) versus longer ones, it is just not clear to me how one can take a true bias (defined as an exposure of time nearly equal to zero) and have that be representative of a bias for a longer exposure. Seems much safer and cleaner to simply use darks of same time and temperature as the frame you are trying to calibrate. They will then have the correct bias automatically encoded, so long as you don't try to subtract a separate bias. The calibration equation is: CF = (Light - Dark)/(Flat-Flat_Dark). The correct Bias subtracts out from both numerator and denominator so long as the Master Dark has the same time and temp as the Light and Flat Dark the same time and temp as the Flat. This does mean you cannot use scaled darks, but darks of whatever time/temp are easy to get. It is not like we lack for cloudy nights! |
|
20.43
#...
·
·
1
like
|
|---|
John Hayes: I'm pretty meticulous with my calibration process. There are definitely no settings mismatches, and I use an in camera offset of 50. I should clarify, that I'm not seeing full clipping. I remembered it incorrectly. I am seeing extremely low ADU's though, and I just dont trust the result. I just subtracted a master bias from a dark to refresh my memory.:  In any event, it's likely academic. I have processed images both ways and as far as the integration is concerned, while I can measure a difference in the result, I cannot see any detriment to the data from a processing perspective. I'd be curious if you get a similar result when you subtract you rmaster bias from a dark. |
|
22.40
#...
·
·
1
like
|
|---|
|
Chris, when I get some time, I’ll look at my data. Your data looks fine to me. I’d only be concerned if the minimum value showed up as 0 and then I’d want to know how many pixels are at zero. A minimum of 1 with a median of 3 looks perfect. But maybe I’m misunderstanding something. What don’t you trust about that result? John |
|
20.43
#...
·
·
1
like
|
|---|
John Hayes: Maybe I'm just superstitious. lol... Additionally, I did a bunch of testing a while back with help from John Updton on dark scaling, and found that this sensor behaves in a very linear fashion unlike many of the CMOS predecessors. I dont think I trust a master dark with bias calibration to scale well. Again, it just doesnt look right, but these cameras do have very low thermal noise compared to the CCD cameras I have used. Perhaps I was just worried about something that was not a problem. Feelings are often confusing when it comes to this scientific stuff... |
|
1.43
#...
·
|
|---|
John Hayes: Thanks John, I assume that you're doing manual stacking. I'm using WBPP 2.4.5 to stack, following the Adam Block's videos. In that case, if I do a library of bias and stack again all my data with flats, darks and bias but no dark-falts how can I appreciate if there's any kind or improvement or not, just visually when stretching the master light with ScreenTransferFunction o there are some parameters I can look for with any process in PixInsight to compare both masterlight? With the chart you shared with us I think its clear than more than 16 subs are not necessary, 20 for those who want to have round figures. But once more just to do any kind of test with my camera to notice or not if there's any positive variation? Maybe with statistics, is there any easy way?? I very appreciate your contribution to this thread, in fact I think that all of us do. |
6.08
#...
·
·
1
like
|
|---|
| This has been a very interesting read, and I was happy just reading what everyone had to say, and I’m sure I’ll get slack for this. Is calibrating CMOS images as difficult as everyone makes it sound. I still use CCD, and all I do is darks and flats, 20 darks and 10 flats, I let CCDStack do what it needs to do and I’m done. I don’t look at the math or numbers, I visually look at each calibrated sub, if they look good I combine them to their single L,R,G,B, or S,H,O file, I do a color combine and move it to PS. If the math/numbers say there not good, but I can’t see it visually in the color image with or without enlarging to where I see pixels, why should I be concerned. Do my images turn out good, I think so, are they perfect, absolutely not. |
