3.71
#...
·
·
1
like
|
|---|
|
An interesting problem recently developed during my latest post-processing of the Medusa nebula. My standard process for processing images is to use the WBPP script in PixInsight, as it has really been improved over the years and automates the post-processing tasks in a nice, convenient way. I’ve been using it for awhile now, with 5 recent processing tasks completed with no major issues. But with the Medusa nebula things turned out a little strange. For the Medusa I managed to get 50 Ha, 32 Oii and 17 Sii subs over a period of two evenings. I took flats and darkflats (25 each) for each of the two sessions/evenings as per my usual work flow. Using the WBPP script, with a master dark of 300sec for the NB subs and the flats and darkflats taken during the evenings I processed the set of light data. Examining the final stacked images I noticed that the Ha image seemed to have very little dynamic range. Suspecting the Cosmetic Correction that I had applied I re-ran the script with no CC. The results were better, but still not up to standard. I ran the readout probe over the image taking some readings in the background and then in the brightest part of the nebula. The background was about 13 ADU, with the brightest regions around 66 ADU. I thought that maybe 300 seconds was insufficient time to get a reasonable level of data, and that maybe the case, but the 5 minute subs did show some fairly good range in the individual subs (as the following table shows). Similar probing on the Oiii and Sii stacks showed 224-392 for the Oiii stack and 118-220 for the Sii. If 300 seconds was good enough for Oiii and Sii, it should have been just fine for the Ha data. Generally, the Ha signal is the stronger, and it should have been for this object. I checked the parameters on WBPP and they seemed fine. Nothing stood out; typical settings, similar to what I usually set for my runs. PI’s Statistics process yielded the following data: Single sub Exp:300 Gain:100 O:50 Temp: -10 Filter Mean Std Dev Min Max Ha 508.0 27.5 338 62448 Oiii 519.2 30.8 352 56090 Sii 505.8 28.1 337 64355 Processed in WBPP Filter Mean Std Dev Min Max Stack count Ha 7.8 11.7 2.2 5212.7 50 Oiii 242.1 143.2 131.4 64138.7 32 Sii 128.6 153.4 52.0 65535.0 17 The sample subs seemed fine with almost identical statistics. However, the calibrated masters showed significant differences. The Oiii and Sii data was somewhat expected, but the Ha data seemed way off the mark. I then decided to take the data and run the calibration manually, without using WBPP script. The results were very interesting to say the least. The Oiii and Sii data were very close to each other even though the results using WBPP showed a difference. But the real interesting part was the fact that the Ha data was also very close to both the Oiii and Sii data. The stacked master means were very close to the value of the single sample subs for all three filters. Integration (manual) Filter Mean Std Dev Min Max Stack count Ha 507.5 11.4 496.8 5186.2 50 Oiii 521.4 14.0 433.0 6057.3 32 Sii 506.2 10.1 343.7 4370.7 17 Now I know WBPP may do some additional processing that I may not have done when I calibrated and integrated manually, but the Ha data was at least an order of magnitude better. I am at a loss as to what is going on here. I was able to complete the post-processing of the Medusa data using the manually calibrated subs and the final result was very respectable. Even the WBPP data was usable, but required much more processing finesse. Any idea on what I am seeing here? |
10.10
#...
·
·
2
likes
|
|---|
|
Hi Michael, For what it's worth whenever I see something like that, my immediate thoughts are that I forgot to either to set the right gain and offset for either my flats or lights in my acquisition sequencer, or I chose the wrong darks or flats in putting WBPP inputs together. I have done both. All can be fixed. Recently, using NINA, I forgot to set my Offset (I always use "50" for my camera = 500ADU for a Zwo 16bit) and it defaulted to an offset to 1 (or 10ADU). This made the calibrated lights look very dark after WBPP, which used a flat taken with an offset of 50 on a light frame with an offset=1 - resulting in 490 ADU subtracted from the calibrated lights, making them very dark and clipped. Fortunately the situation was repairable by using Pixelmath and Image container to add the offset ADUs back into the light frame before calibration with WBPP. In your case, I would make sure you selected the correct and identical gain/offset files in both your manual and WBPP calibrations. Hope this help - sorry if I am completely offbase. Dave |
|
3.71
#...
·
·
2
likes
|
|---|
| Thanks for the insight Dave. I did check, and all subs, flats and darks were taken with the same Gain and Offset. |
0.90
#...
·
·
3
likes
|
|---|
| Whenever something like that happens to me I will rerun without flats just to do a process of elimination. |
0.00
#...
·
|
|---|
|
Hi Michael, did you use Local Normalisation in WBPP ? Daniel |
|
3.71
#...
·
|
|---|
| Yes. Local normalization was set. |
|
0.00
#...
·
·
1
like
|
|---|
|
In this case, it may be worth testing the script without this option. I stopped myself using it without close supervision. I remember that once it completely unbalance the offset and gain between filters. daniel |
|
3.71
#...
·
|
|---|
| Daniel. Thanks for the suggestion. I will try this out and let you know. |
|
3.71
#...
·
·
1
like
|
|---|
| Unfortunately, disabling Local normalization had no effect. I did have Cosmetic Correction still on, so I may rerun once more with that disabled also. |
6.15
#...
·
·
3
likes
|
|---|
|
Michael, I followed the link to your issue posting from the collaboration of your data with Uwe. I wasn't a member of this forum so hadn't seen it before. Reading your OP was deja vu all over again. I apologize in advance for the length of this reply, but this covers an issue I spent a couple of weeks in getting to a solution. I had a similar/same issue last fall where my stacked images were almost flat in data after stacking despite looking very good unstacked. As I dug into this, I found the stacked histogram reduced significantly vs the unstacked images, especially on the Ha master. I went through every WBPP setting I could think of trying to understand what was happening. I turned the problem with some data files over the PI support forum and got some feedback very quickly. My darks had a higher ADU level than my black level on my data files. In other words, light leak on my darks. It was one of those "...duhhh..." moments; I had been in the habit of inspecting the high/low rejection frames in WBPP. Particularly the Low rejection - that frame is the "canary in the coal mine" indicator of bad calibration frames, particularly Darks. If you see much of anything there, you have some calibration file problems. Unfortunately, with the addition of the drizzle function ( which I'm addicted to), the Hi/Lo rejection frames are not part of the display for the drizzled master frame when first opened. So you have to open the non drizzled masters to access the Hi/Lo rejection frames to check for issues. As it turns out, I have a light leak, not surprising really - 300 sec exposures (or longer) for darks are going to find anything short of perfection if you are shooting Darks during daylight hours with just OTA covers. It isn't really practical to remove the camera and install the OEM sensor cover (for me) since that affects the tilt adjustments which are important on APS-C size sensors and larger. So while I also went and taped every seam and gap/crevice anywhere in the optical train, I also changed to shooting Darks after full darkness and with my Telegizmo 360 cover in place to get as close to absolute darkness around the OTA/imaging trains as I can. That did the trick and provided the optimal solution to what I thought initially was some weird problem with the imaging files or WBPP. However, I was able to "correct" a mistake I made in WBPP to allow me to progress until I had the calibration files 'fixed'. That is covered below as it touches on another issue specific to CMOS sensors and NB filter imaging. A quick check for light leaks is to check your darks (or master dark) and get the mean ADU and compare that to the mean ADU of unprocessed image files. If the level of the dark is almost equal or higher than that of your lights, you are going to get data clipping. However, there is another issue that can give similar results and has a solution embedded in your camera and WBPP settings. This is the oft reported problem of insufficient offset or lack of pedestal in your lights when using a CMOS camera and NB filters You can find an in depth discussion with Adam Block on his YouTube channel and there are some Pixel Math scripts to analyze your light frame to determine if this is part of the problem. (if there are absolute black/negative value pixels in your image, you have this problem. Increasing the offset in the cameras settings, or adding pedestal in WBPP can compensate for this. WBPP will actually analyze the light frames and add an appropriate amount of pedestal, IF that option is selected. However, the default is to NOT perform the automatic pedestal function so you have to remember to select it each time. The thing to remember is having WBPP perform an automatic pedestal correction can also protect your processing from MINOR light leak induced clipping, as well as protecting you from the infamous CMOS/NB clipping problem. If I remember correctly, I also saw some light leak influence in my FlatDarks, but not to the extent that showed up in my Darks. Pedestal won't really address much in there is a problem with the FlatDarks, but the effects of bad calibration induced clipping will always show up in the Hi/Lo rejection images, at least that has been my experience. Looking at the stats you provided in your posting, you didn't show the stats for your calibration frames/darks. However, from comparing your unprocessed results to the stacked results, it seem clear that your Darks have an ADU level close to equal to that of your Ha light frames since the stacked Ha master has an ADU mean of close to zero instead of a healthy level of pedestal shown on the Sii, and Oiii. That (to me) is an indicator there is an issue with your calibration files. Uwe's post with his processed image of your data indicated he didn't do anything different in his WBPP setup, yet he didn't have your issues. I would be willing to bet his standard WBPP set either has a very healthy fixed pedestal value or he normally selects the automatic pedestal adjustment. Pedestal adjustment can correct for a light leak/higher ADU value in the DARKS to an extent (that is what I did first until I could shoot a new set of darks). I believe if you just perform a 'quick' WBPP run with auto pedestal enabled, you'll find that most of your issue/loss of data in stacking is resolved. And/ or you can check the adu level of the Ha darks/dark master vs the Ha lights to also check if they are close/equal/greater than that of the Lights. |
|
3.71
#...
·
·
1
like
|
|---|
|
Thanks George, and as I commented in the conversation attached to the image I'll do that run in PI with auto pedestal enabled and also check my darks. Update: My master darks (for all exposure values) has a mean ADU of 500. |
|
10.10
#...
·
|
|---|
Michael J. Mangieri: That means you would have used an offset/pedestal of 50 when taking your darks, the default value on ASI cameras. |
|
6.15
#...
·
·
1
like
|
|---|
Michael J. Mangieri: Michael, That seems "odd" given the median ADU results on your Ha Master vs the initial ADU vs the results on Sii and Oiii. It almost looks as if you had auto pedestal enabled for Sii and Oiii but had no pedestal enabled on the Ha. Here is an example of my median ADUs for my Dark, Ha (raw FITS), and Ha master (w/auto pedestal) taken on IC 443 in January. Dark: 502.0 IC 443 Ha (Fits): 588.0 IC 443 WBPP Ha Master: 161.460 Since the Dark is subtracted from the FITS during calibration (along with other operations with Flats/etc), you can make a generalized assumption that WBPP Auto Pedestal added sufficient to the Ha master to double the separation between the original FITS and the DARK before subtracting the DARK from the FITS files. This left a pedestal for processing of approx 161 ADU which should insure no zero/negative value pixels in the Ha master. The stats you posted earlier seem to indicate that some sort of pedestal was being applied to Sii and Oiii in WBPP, but no pedestal was applied to the Ha, Or this could be an effect from the calibration of the FLATS/FLATDARKS into the masters. Either way, it isn't that hard to have nonsynched settings in WBPP. Each calibration option selected by the user only applies to the set of lights/masters that is highlighted at the moment. The user has to take a "positive" deliberate action to press the button to tell WBPP to apply the same option(s) to all the lights/masters being processed. I have recently tried to get into the habit of saving the WBPP processing log since I've had occasion to need to go back and look at what WBPP did in each of the steps it executes. One of the things it captures is if any pedestal or auto pedestal is applied. Anyway, the short answer is use auto pedestal for all calibration groups and you should not see this issue repeat until / unless something goes significantly wrong with your calibration files, then you get to play sleuth again. I said "calibration files" for a reason. It has been my experience that when a problem occurs about 98% of the time (maybe higher) it is because of something I did, or didn't do, or didn't do correctly, or overlooked in my calibration files. Very very very seldom has it been due to a problem with the actual imaging or image files, sequence software, or processing software. As a side note, I'm a bit surprised that there is not more ADU separation between your Darks and the unprocessed NB filter frames (which makes pedestal even more important). Your results make it appear that there is almost no ADU level difference between the DARK and the exposed Light frames. However, I have a wide/fast refractor vs. your slower/longer focal length SCT, so maybe that is typical for our setup. I'm using Antlia 3nm filters vs. your Optolong 3nm and we have the same camera. Someone with similar equipment would need to chip in with that information. |
|
3.71
#...
·
·
4
likes
|
|---|
|
Well it looks like George may have hit the nail on the head. I re-ran the WBPP script with all the exact same original data (lights, darks, etc.) but with Output Pedestal set to Auto and the results came out fine. The mean ADU for all three NB stacks was around 80-85. Subsequent post processing yielded very good results. And this time I watched the process viewer as the calibration of the lights started, and sure enough, there were messages pointing out something to the effect that 'negative values were present and pedestal value of 72 ADU applied ', or something like that. Thank you all, and especially George Yendrey, for pointing this out. I will certainly double check ALL the script settings before running it next time to make sure they are correct and appropriate for the object at hand. And I have a much better understanding of what the pedestal is all about too. |
