1.20
#...
·
·
1
like
|
|---|
|
Hi guys Sorry for the long radio silence - travelling in China. Beautiful clear weather but Beijing's B9 skies aren't exactly encouraging! Trying to answer some of the points raised above, although I'm sure I'll have missed some. I've tried to treat the data reduction in three stages (starting from stacked OSC images):
My thinking is that the first two steps *should* be fairly straightforward, with not too many decisions to make. Of course, "straightforward" still leaves plenty of room for problems and code bugs! Some issues that became apparent when I looked at the low-resolution whole-sky stack include: - There are a couple of regions where stars appear blurred or doubled up, generally indicative of a plate solving or rebinning issue. These are fairly isolated and I'll take a look when I get home. - Some images seemed to show a star colour gradient eg bluer in the middle and whiter at the edges. I'm concerned that some of my images near the south pole also look very blue. A colour gradient would imply a problem with the stacking or flat/dark correction; a uniform colour cast would imply a problem with my colour correction algorithm. Again, I'll dig into this when I get home, but if you notice any fields looking 'odd' please bring to my attention and I'll focus on them. - Michael, your comments around applying BlurXT before SPCC are interesting. I don't have PI and so can't use these tools. My code integrates the R, G and B stellar fluxes inside a radius of ~5 pixels and compares to known RGB luminosities. Generally I find measured magnitudes are within 0.3-0.4 of quoted values, and by fitting several thousand stars I should get good average correction values, but I need to check for carefully for the impact of outliers, star shapes etc. I feel that using a generous radius *should* make star blurring or shape distortions less significant, but need to test some more. Of course, I'm also dependent on effective deBeyering of the images to preserve flux. Gradients are of course the hard problem and the one where I feel much more subjective judgement may be needed, which is why I've left this correction to the end. Michael, I also noted the consistent gradient in the whole-sky mosaic for the images I collected. This may be a bug/artefact. I applied a simple automated gradient removal algorithm to all fields before stacking (there was a stronger and objectionable green gradient before I did this) and I suspect this may be responsible for the diagonal grey gradient that is so obvious. Again, something for me to dig into when I get back. I also haven't applied any gradient equalisation between fields yet. One challenge that does need to be managed is the rapid growth in image size as more and more fields are stacked. I'm nervous about an approach that requires 'global' corrections to be done on multiple fields (eg your steps 6-8 Michael) as this may be computationally impractical. My approach has been to do alignment, colour and basic gradient corrections on individual fields. Gradient matching *is*performed globally but at low resolution eg by calculating background R, G, B values in tiles measuring maybe 0.5 x 0.5 degrees and correcting for differences where fields overlap. The wide field (40 mm) data would also come into play here, confirming background RGB values on a similar tile scale. Hopefully I can snag some time over the Christmas break to push forward on some of these points. I'll share results as I go, but in the meantime am keen to hear about any issues or anomalies that you pick up in the images I've already uploaded. James |
3.94
#...
·
|
|---|
|
I had some fun with Gradients and playing with Process and Image Containers on my latest data, for me this seems to be another way to cope with our data, at least for the steps before merging a mosaic, in that area I am looking (again) in a solution based on AstroPy. Here's what I got with minimal effort, total time I (better: my computer) worked on this were 10-15 minutes. No Postprocessing done, only GraXpert, BlurX correct only, Image Solver and then SPCC at the end. Then combining the fields with Photometric Mosaic in default settings. Then to show off details I removed stars and did a little NoiseXterminator. The seams between frames are visible, but I think that most of this visibility is because the upper and the lower frame have only 50% of the integration time of the middle frame and for that are more grainy, a price to pay when you 'only' have two cameras and do images on both sides of the Meridian.... For me it is a joy to look at the stars in the field, I did not see issues with duplicated/half stars when inspecting the seams. Will now look at some data from Brian to see if this also works in more demanding areas of the skies.... Michael  Here's the Pano of the 3 images:  |
|
3.94
#...
·
·
1
like
|
|---|
|
I took 12 tiles from Brian and ran them through the process as described above, I like what I see with the stars everywhere, cannot judge the colors (I am colorblind) but I think things look good besides a few fields where Photometric Mosaic failed on me. I really hopy that AstroPy (or @James Tickner ) can come to the rescue, I think the remaining gradients should be solveable.. @Brian Boyle how bad is was I created? (Besides the fields with slightly different background?) I had to crop quite a bit to be able to stay below the 20Mb Limits for uploads, the original and the single parts are also available on my Network drive: https://www.mycloud.ch/s/S0012A31EE7B15509D2A9051E1B72E54FE47FEC8168  |
0.00
#...
·
·
2
likes
|
|---|
|
Hi Michael, regarding gradients, the field looks very good. Yes, there are some, but they are rather minor. As for color, at least for me there seems to be a green/yellow cast. It is rather uniformly over the whole image, so I it has something to do with the process. CS Gerrit |
|
3.94
#...
·
·
1
like
|
|---|
|
Thanks Gerrit! Seems I need to do more experiments, glad there are some days with bad weather ahead.. One question, do I remember correctly or didn't you also do some widefield images in the 30-50mm range? If yes, are they available anywhere? Michael |
|
0.00
#...
·
|
|---|
|
Hi Michael, Just some days with bad weather for you? I haven't had the opportunity to image for the last 3 months..... Wasn't even able to do some equipment tests yet. As for my images, yes, I am doing Widefield with 50mm and aps c. Some even under dark skies. Some of my final images can be seen on my account in forum.astronomie.de (user mame: Gerrit Erdt). Take a look, if they fit your needs, I will provide you all the raw data and information you need. CS Gerrit |
16.52
#...
·
|
|---|
| @Michael Ring That mosaic looks good. Inspired by your attempt, I also tried to mosaic these fields. The steps were WBPP-> Resample at 10arcsec/px -> Dynamic Crop [7.5 x 5 deg] -> Image Solve -> GraXpert -> MosaicByCoordinates -> GradientMergeMosaic (average mode) -> GraXpert [again] -> BXT -> STF stretch -> Histogram Transformation [black point 0.5%] The cropping was done to remove the worst effects of my tilt in the corners. The seams are clearly seen [particularly in the bottom left F112] I checked F112, but it didn't have any peculiarities in either colour or image structure. Then I tried to combine just F112 and F142 to the North, in Photometric Mosaic and the seam disappeared. So clear the seams here are a function of GradientMergeMosaic. Unfortuately using PM on more than a few fields is not only very laborious, but the program on really works for fields that are parellel to each other in x,y coord system of the camera. Hopeless for curved projections away from equator. This still gives me a lot of confidence that @James Tickner will work. Much of the reduction is possible in PI, but it is the final merging step where the current PI routines fall down. Of course, I still cant completely rule out data quality as an issue - but once we can rule out the merging process as causing any remaining seams then we can get a better idea of QC.  |
|
16.52
#...
·
|
|---|
And here is the same mosaic, but this time using PhotometricMosaic (pairwise in strips of constant RA) instead of GradientMergeMosaic. I even added a mild star reduction, greater stretch (2% black point), and a touch of enhanced saturation. This is the first time I have been able to stretch a mosaic this far. There are still "seams" but not aligned with the edge of panes. This is likely to be an artefact of the way PM works (looking for panels that are "square" to the final mosaic. Again it is encouraging that the data allows us to get this far, particularly given the PI's mosaicing routines clearly have limitations and were never designed for mosaics of this scale. CS Brian |
|
3.94
#...
·
·
1
like
|
|---|
|
GradientMergeMosaic seems pretty unusable at the moment, it is slow and not good, attributes that do not work well together. Are you able to do some Panels that include the Milky Way and perhaps also some empty areas with your Sigma Lens from areas where you already have all 135mm data available? The GraXpert guys worked with my data from the Orion region and the results are again very, very promising. It was enough to have 1hr panels at 40mm to calibrate out the gradients of the 135mm fields so for your skies even 30min of exposure should be enough. One thing I saw in my experiments was that Photometric Mosaic did not produce good results when background levels on neighboring fields were very different, the result of merging those panels had a bad color cast. After subtracting the difference the fields blended well. I will experiment more in the next days Michael |
|
16.52
#...
·
|
|---|
Michael Ring: Hi Michael, I have not done anything with my Sigma lens at the moment. 3hour nights and lots of cloud have limited my AP for the moment. I can do more panels with the Milky Way - although the results above also include the Milky Way.. It just takes a lot of time with Photometric Mosaic. And i don't have the computational skills to write my own code. CS Brian |
|
1.20
#...
·
·
1
like
|
|---|
|
Hi everyone I've uploaded a new version of the public global sky map. Compared to version 1 this has a few improvements: - I've fixed up the embarrassing R<->B swap in the star magnitudes used for colour calibration. Stars are now looking more natural I think! - A few fields had poor quality plate solutions due to relatively few stars being included. This led to poor constraints on the 3rd order distortion corrections, which mainly manifested as stars being a few pixels out of position in the image corners. This affected the colour calibration algorithm, and also led to 'blurring' in the mosaic where stars from two fields overlapped. - I've tidied up the code used for gradient matching between the thumbnail resolution images used to build the whole-sky map. There's one annoying issue with star colours shifts which I'll bring up in the QC forum. Regards, James |
|
16.52
#...
·
|
|---|
James Tickner: Hi James That revised map is looking really good. Brian |
|
1.20
#...
·
|
|---|
|
After what can only be described as an inordinate delay, I've finally assembled the code to produce a first prototype mosaic of the south polar region. The assembled image is available in my Google drive (link https://drive.google.com/file/d/1fQrIPaH9IfYVJUAHFwgSwffQPN5Ag1m2/view?usp=drive_link) with the filename 202403_south_polar_mosaic.tif. This includes fields 1-46 and covers the region from the south pole to approximately DEC -67 deg. The image is saved as an unstretched 32-bit TIF file which is approx. 17k x 17k pixels in size (2 GB approx). I'd encourage anyone who's interested to download and see what you can produce! The main parts of the workflow have been discussed at length in this and other fora, but in brief:
The main change from my previous approach is to delay the rebinning until the end of the process. This keeps images in their original format and resolution as long as possible, and should facilitate introducing other steps (eg star removal, other gradient extraction tools, BXT etc) as required. Usefully, these steps can be run field-by-field and so avoid trying to process giant mosaics. From a quick inspection, the alignment between fields seems OK. Seams between images don't seem too obvious, although I haven't tried stretching really hard. Colour balance is the hardest thing to judge accurately. There does seem to be some 'shadowing' around the brighter objects (eg 47 Tuc) which I'm putting down to the gradient removal - something for me to tweak later. All feedback is appreciated. In particular, I'd love to hear about any overlap artefacts (doubled stars), colour issues, seams or gradient issues which I'm sure will be there at some level. |
|
1.20
#...
·
·
1
like
|
|---|
Just to whet the appetite, here's a scaled-down JPG version of the mosaic. |
|
16.52
#...
·
·
1
like
|
|---|
|
Hi @James Tickner That is wonderful. I did a few quick processing steps in PI. BXT, stretch and star removal. Some quick initial observations 1) The field-to-field matching looks really good. 2) The appears to be a strong a large scale (grey) radial gradient, which made it difficult to see large-scale structure. So I used GraXpert to take it out - but only for he purpose of looking a bit deeper at the image. Note that I used a rectangular cut-out of the area to make my life a little simpler, 3) After GraXpert, I applied BlurXT, STF and StarXT. And this is what I got  After SCNR and flipping horizontally [I now recognise this sky very well] I got this  There is still a large scale gradient not removed by GraXpert . In part this may be due to the different set-ups used across this strip - but if we restict to one part that the result looks pretty funky and detailed.  I realise that much of the blotchiness is probably due to what GraXpert did I realise that much of the blotchiness is probably due to what GraXpert did [and if you put back in the stars the effect is pretty horrible. But I think it shows that the field-to-field matching for the background is pretty good, and that another tweak - as you indicated - is needed. Don't really know where the gross greyscale radial background cam from though..... Brian |
|
1.20
#...
·
|
|---|
|
Mmmm, very interesting @Brian Boyle! Notwithstanding some of the gradients and blotchy colours, I think this is the first time that I've appreciated the extent and complexity of the IFN near the south pole. Major features like the Dark Doodad and the Chamaeleon Cloud almost disappear - there's just so much going on! The radial gradient and colour blotches are interesting. I'm guessing that these come from the iterative gradient matching step that I perform at the end. In this step, I fit linear R, G and B gradients (ie colour = a + bx + cy where x = pixel column and y = pixel row and a, b and c are fit coefficients) to the difference between a chosen field and all overlapping fields. After performing the fit for each field in the mosaic the linear gradients are subtracted. The process is iterated until convergence, at which point background levels should match. The downside of the iterative approach is that it is poorly constrained around the edges of the mosaic where there are no more overlapping fields. I'll try to add some constraints to encourage the fits to maintain the correct overall brightness. The colour blotches might be a similar artefact - I'll experiment some more. And I'll fix the 'flip' bug - the program that I'm using (MATLAB) maps array to images in an odd way. I've built another version of the mosaic with the iterative gradient matching omitted. I suspect this will show more unevenness between tiles, but should get rid of the radial gradient and maybe some of the colour patches. Interested to see what you find with the same image processing treatment. If notthing else, it should help confirm if I'm on the right track. It will take a while to upload, but I'll use the same filename with '_noiter' added at the end. Thanks! |
|
16.52
#...
·
·
1
like
|
|---|
|
Hi all, I attach a progress report on the Southern Sky mosaic at 40mm using @James Tickner centres. Thanks to @Michael Ring for encouraging me to buy the Sigma Art 40mm  Here is the result of 3 nights and 15 of the 29 fields with centres of dec 0 or below. Fog/moon after 2am or so has hampered my ability to get fields later than 20hours or so, but it is not so bad. I might have a few more clear nights this lunation to complete a few holes. Since I can't write code, I rely on what already exists, so this was created as follows Observing 40 x 120s using Sigma Art 40mm wide open at f1.4 Blink and Subframe selector to remove poor (mostly cloud affected) frames. [each field has at least 30 subs] WBPP GraXpert BXT [correct only] Stretch [using PixelMath with expression to replicate STF] Write out to jpg Use PTGui to stitch using Fisheye geometrey [All PI/APP routines are utter s**t] Clearly the stretching has not worked so well between the fields with the galactic plane and those at high high galactic latitudes. But I am pretty amazed it has done so well with such a crude approach. [I also tried with differnet crops - but this problem still remains]. If anyone wants to try to create a better mosaic I can put all the files on DropBox. Note that I have resampled down to 2arcmin/px - the original is at 30arcsec/px - for the purposes of this post. I probably have enough data to drizzle to get to 15arcsec/px, but my elderdy laptop to simply to slow to do this. Either way, I think. this is encouraging for the ABC survey mosaic. What do you think? |
|
16.52
#...
·
·
1
like
|
|---|
| PS Observing field centres with Dec=0 with a 40mm lens and full-frame sensor presents a bit of a timing challenge at latitude 45 South and mountains that rise to 20 deg in elevation in the North... But at least I have the benefit of a dark sky and long nights at this time of year. |
|
3.94
#...
·
·
1
like
|
|---|
|
Congrats @Brian Boyle , the pano already now does look really nice and detailled! Some day I will have to come to the south to enjoy the skies you see with my own eyes… It would be good if you could upload at least some of the raw stacks, then I can restart playing with the ideas on gradient removal. Did you use the fields mapping that @James Tickner did for me? In that case the overlap of frames should be quite high with your fullframe camera which should make it easier to adjust the differences between fields. Thanks for all your efforts! Michael |
|
16.52
#...
·
·
1
like
|
|---|
Michael Ring: Hi Micheal, Thanks. Just started observing tonight.. will upload the files to dropbox tomorrow, as I might have a few more by the end of the tonight. [But it will take me some time to process them]. The field centres are the first list that @James Tickner sent, based on a FF camera. I think you might have used the 2nd APS-C sized field centres. Either way, we have covered a lot of the sky at this focal length. Brian |
|
3.94
#...
·
|
|---|
|
I wish you good Photon hunting tonight! How fast is your Internet? As a fun project I would love to stack and drizzle one frame of your pano if that is acceptable effort for you. |
|
16.52
#...
·
|
|---|
|
Hi Michael Happy to make everything available to the team... if you want to process for the start, I will also need to supply bias, datk, flat as well as the subs. I presume that means uploading them all to dropbox too. Or is there are smarter way? |
|
3.94
#...
·
|
|---|
|
For a start Dropbox is perhaps easiest way to do things. Flats and Bias are a must, darks are not strictly necessary with modern cameras Michael |
|
16.52
#...
·
|
|---|
|
Hi Michael In the meantime, I have dropped all the processed fields (WBPP) into the directory Widefield on the DropBox. Have fun! Brian |
|
16.52
#...
·
|
|---|
|
Hi Michael In the meantime, I have dropped all the processed fields (WBPP) into the directory Widefield on the DropBox. Have fun! Brian |
