| Would you be interested in contributing towards an AB all sky survey? | |
|---|---|
| No. I wouldn't find such a survey useful. | |
| No. Satisfactory data already exists for me elsewhere. | |
| No. I would find such a survey useful, but I don't have the time, location or equipment to contribute. | |
| Yes, I would be interested in taking part. One or two fields maximum. | |
| Yes, I would be interested in taking part. Prepared to do multiple fields. | |
| Login to vote and view results. |
 #...
·
·
1
like
|
|---|
James Tickner: Alternatively, what if the participants to the project put the images in an AstroBin group, and I add a button to groups to download all images and associated plate-solving details, and this resampling can be done offline with a script of your writing? |
16.52
#...
·
·
1
like
|
|---|
|
As always, I am overwhelmed by the creativity and knowledge of Astrobin users. Thanks in particular to @James Tickner and @Astrogerdt - and of course @Salvatore Iovene - for their really helpful contributions. I propose we adopt @Astrogerdt detailed observing and processing pipeline as the working model. To imaging requirement I suggest we add - all observation above a zd of 60 and with moon down [or max phase 25%]. Reluctantly [!], I also agree with @James Tickner on taking the mosaicing out of the hands of the individuals. I say reluctantly because my initial thought was to avoid all the complexity and hard work needed to mosaic-up the whole sky, and simply produce separate fields, like the pages of some (now) antiquated atlas. But I completely accept that a whole sky approach is so much better. Based on my own success [or lack of it] with mosacing via a number of different routes, I think @James Tickner is spot on in not leaving it to individuals - certainly not if they are like me. It also has a efficiency gain, in that I can probably doing "patches of fields" with multiple overlapping panes without necessarily having too worry too much to centre each group on a survey field centre. Thanks to @Salvatore Iovene for offering AB as a platform for the individual pane sharing, but we still need someone (or a team) to QC and mosaic up into the whole sky. Without that effort, the survey - at least in this form - won't happen. This is a massive task. And it not something I would necessarily expect anyone to volunteer for. So if any one does come forward, we would owe them a massive thanks. The only thing that could possibly be said in its favour, is that it might be an interested astroimaging project for someone stuck up in the North over the coming summer lunations. Please keep the constructive comments coming, I feel we have already made significant consensual advances. We have a draft pipeline and method for allocating patches of sky. We have over 100 interested potential participants (imaging or otherwise). We may need to fine-tune the pipeline, but probably out biggest hurdle right now is the QC/mosaicing, and significant resources needed for this. I would also like to thanks all those of you who have been so constructive with your comments to date. Given all our different backgrounds, experiences and equipment we will never end up with a protocol/pipeline that anyone of us would have individually adopted, but that is the power of collaboration. Also, I would advocate that we may best learn by doing, and that early start (i.e next dark lunation) would help us iron out bugs/issues before we get too far in. This is largely driven by my own personal experience, and in particular I recall a large imaging survey in was involved in during the 80s and 90s. At the start of the survey, our (small team was told was told by many, many people smarter than us, that what we were doing was crazy and not possible. Indeed, the survey went through a number of observing and processing pipeline changes in the early days. It took us six years to find out first distant supernova, and a further four years to uncover the acceleration of the Universe. I won't pretend that this survey is going to have the same impact, but the wonderful feeling of collaboration achieving something much bigger than one could do alone is very much the same. Here's to more great advice and offers of help Thank you, Brian |
0.00
#...
·
|
|---|
|
I am not sure if QC really must be that much manual labor. Off the top of my head, I can think of the following quality determinators for each field:
I don't know if my thinking is way too simple, but I think those are some important points of QC, and they can be at least partially automated, or at least to an extent that it reduces manual work. Or am I completely off? CS Gerrit |
1.20
#...
·
|
|---|
After reading your workflow, @Astrogerdt I see I need to bite the bullet and invest in a copy of PI! Most of the ideas I suggested above are already implemented by them  In particular the SPCC colour correction tool would remove the need for users to collect separate images of the calibration field as I suggested before. My only concern is that SPCC calibrates to star colours which are obviously broadband. If different camera have different spectral sensitivity then narrow band emissions are going to appear at different brightness. Suppose two users collect neighbouring images of an Ha emission nebula, one using an astro OSC and the other an unmodded DSLR. Even after the star colours are correctly calibrated, the Ha emission will be much fainter in the DSLR images compared to the astro OSC. Reading the documentation for MosaicByoordinate it seems that this is exactly the functionality we need to tranform the user's images onto a regular RA/DEC grid. The input options are shown in the image below:  So if I understand correctly, we would ask users to set What is people's experience with this tool - does it work as I imagine? I guess the final compositing of the master whole sky image will have to be done separately (unless someone has a computer to handle 8 Gpix images in memory!), but with aligned and registered field images this is Edited: reading more about PI's projection options, Mercator is probably not a good choice - maybe Orthographic or Plate-Carree would be better, depending on which introduces the most distortion on a 12 x 12 degree field. It also seems that PI rotates the coordinate system to centre the selected point before applying the transformation (looking at the examples in the ref documentation). Whilst this makes sense for a single image, it will complicate the compositing into the master whole-sky image, but I think this can be managed if the transform equations are known. |
|
1.20
#...
·
|
|---|
Does it cover the required area? That is solved by plate solving Some comments:
|
|
7.90
#...
·
|
|---|
|
I fear we're starting to lose sight of the forest from the trees here. If you think that mosaicking is going to be easy because of MosaicbyCoordinates than think again as in one instance I couldn't make an mosaic of 6 frames taken with the same telescope over the course of a couple of months from a pristine location. YMMV. So, going back to basics there should a thought on the foundamental issues here, namely what kind of skies are acceptable and what aren't, what is the minimum height on the horizon these images are taken at, what is the maximum moon phase to be allowed and how far from the moon you need to be, what is the required minimum magnitude of the integrated light, and what if the image scale and field width isn't the one required (in which case I'm most likely out). Also, what are the optical requirements and the imager requirements (personally, I wouldn't mix OSC/RGB with DSLR with red cutoff and wouldn't accept LRGB, just RGB). And that is for starters. |
|
0.00
#...
·
·
2
likes
|
|---|
| @James Tickner Yes, PixInsight is truly amazing, its capabilities are stunning every time I use it! SPCC uses the transmission curves for better accuracy, but of course it can't correct for the use of modified/unmodified cameras in one survey. Unfortunately, I see only one possibility to solve this issue, namely to exclude unmodified DSLRs even if that is a significant amount of people. I used an unmodified DSLR myself for many years, and imaging H Alpha objects with it is just pointless, even in dark skies. It requires so immensely much more integration time, that it becomes nearly impossible. And since practically all the sky is filled with H Alpha, there are no fields where this would not diminish the results. Unfortunately, I can not contribute to the discussion about the MosaicByCoordinate script, as I never used that. One possible solution to the gradient problem could be to have people with high quality all sky cameras in dark sky location on board with the survey. I never had one, so I don't know if my idea is stupid, but I read about a multiscale gradient reduction method developed by the PixInsight team. Basically, you take your normal frames (lets say f=100mm) and then additionally frames with a much larger FOV (say, f=20mm). Doing a gradient reduction on the 20mm image will be equally hard, but on a local level (your desired frame covers only a small fraction, so it can be considered local), it will be practically gradient free. All sky cameras or other very wide field setups could provide such data. The method is described in more detail here: https://pixinsight.com/tutorials/multiscale-gradient-correction/ Maybe this could solve some of the gradient problems in a final step after the mosaicing process. Contributors would still have to perform gradient reduction on their own fields, but it would lessen the impact of this step and thus take a burden from the QC team. Reg@andrea tasselli's considerations about acquisition conditions, I made a quick table for calculations based on sky brightness and f/ ratio:  I guess from a pure SNR and quality point of view, it is irrelevant wheter the data were captured in bortle 1 or bortle 4 skies, providing they are deep enough and processing is done well enough. Using this table, one can calculate the needed exposure time to achieve the same SNR under different conditions, with SQM=22 and f/=2 being the baseline (I derived that from talk earlier, but that can be easily changed). Let's say one images from SQM=19.8 and f/=4, then the exposure time has to be 7.59 * 4 = 30,36 times longer. As you can see, this heavily favours dark sky, so maybe we should think about 21.5 being the norm. That should be up tof further debate. This way, we can have some consistency and better planning abilities for acquisition. Moonphase, distance and altitude are a whole new topic we have to cover. CS Gerrit |
|
16.52
#...
·
|
|---|
| @andrea tasselli @Astrogerdt thanks for your thoughts (and your helpful table) Two observations: 1) A few posts ago, I did propose we include a moon down or illumination <25% to mitigate the gradient problem. And also zd<60 criteriion No-one has commented on that so far. Can I presume there is general agreement on this? At least asa starting point. I did a test field the other night: 4 panes using my Nikor 200mm f2 + ZWO 2400MC with 20 x 300 secs in 45% moon (2 panes were done with moon up, one as it was setting and one with moon down). I followed @Astrogerdt pipeline. I chose Field 70 in the survey, at it fulfilled the zd requirement above. And it was suitably anonymous. Here is the stretched mosaic'ed image. FIeld70_ht.jpg As you can see it hasn't stiched together very well. Whether that was the moon or my incompetence, I don't know. All I do know it is the first mosaic I have failed to stitch well in quite some time. I tried both MosaicByCoordinates folllowed by GradientMergeMosaic in PI and APP's mosaicing. It is the latter that is shown here, because the former was even worse. 2) The draft survey parameters were quoted in SNR at a give mag, so sky brightness is naturally taken into account. It works out at roughly 2hours per pane in Bortle 2, 3hours in Bortle 3 and 4 hours in Bortle 4. In truth this is less driven by desired depth (we will push it as far as we can) by more by sheer practicality. We can ask people to give up all their dark, clear time for the greater good. So where are we now? If this latest discussion reinforces anything it is the following [square brackets denote values which we can still debate constructively]. Variable exposure times are a must to reflect different sky conditions, but will limit observations to Bortle [4] or better. [It is unlikely that un-modded DSLRs will reach the required depth around Ha, and so astronomical cameras only] Minimise moon light, moon down or [25%] illumination at most. Thanks Brian |
|
0.00
#...
·
|
|---|
|
Hi Brian, Nice image! To me, it doesn't seem like there are clear borders, just huge gradients over due to the single panels. This should be at least easier to solve. I guess if we really struggle to create a reasonable mosaic, we could ask Juan from the PTeam for help. With such a huge project, I guess he will be willing to help us out with his knowledge. If we continue to struggle, we could try that. Your Bortle 4 = 4 times exposure time fits well with my table, so if someone is short on time or wants to fit in as closely as possible, he can use those values to calculate the needed time. I can't really comment on your moonlight considerations, as my Bortle 6 skies are complete trash, regardless of the moonlight. Most of the time, I do not only take moon illumination, but also the angle to the target into consideration. I try to image as far away from the moon as possible. Should we include something about that into our criteria list? CS Gerrit |
|
16.52
#...
·
|
|---|
|
hi Gerrit, I think you are right about my images - I will investigate my pipeline a little more.... Brian |
|
16.52
#...
·
|
|---|
I have added the following to @Astrogerdt pipeline
Happy to receive comments - no response means you are happy with them. I will also work on the basis that the field centres are now fixed. The method I would like propose for field selection is - at least initially - a very simple one. It uses the existing astrobin capabilities, so no-one has to do any extra work, butI would be great @Salvatore Iovene could indicate that his is comfortable with this approach 1) Following each full moon, we use a post to the Astrobin Forum to announce the availability of a RA limited range of ABC survey fields. 2) The RA range will be three hours either side of the LST in the middle of night at new moon in the lunation. 3) Users can then simply post a message to that forum, claiming a field (or fields) to be done during that lunation 4) Once completed, observers simply post the image to their page.
This allows us to maintain an archive of the individual panes, crowd source a mosaicing solution and begin QC. What we still don't have is someone willing to mosaic fields together. Inspired by the above, this could be crowd-souced too. Those interested in the large-scale mosaic this can simply follow those people who have booked fields (particularly neighbouring fields!) so that when they post their images they can request the individual panes and start experimenting. We have specified fields and observing/linear processing protocols as tightly as appropriate, and now we are using the strengths of the Astrobin platform and community to host and crowd source the acquistion, QC and ultimately the mosaicing of the ABC survey fields. Grateful for people's thoughts on this. Brian |
#...
·
·
1
like
|
|---|
Brian Boyle: That's fine but I recommend creating a group here: https://www.astrobin.com/groups/ (Actions menu -> Create new group) This way all members of the group receive a notification when there is a new topic. You could organize this by creating a new topic for each month, instead of keep posting in the same one. |
|
16.52
#...
·
|
|---|
| Thanks for the advice @Salvatore Iovene |
3.94
#...
·
|
|---|
|
Time to order that Samyang lens, I only have APS-C sensors or unmodded Nikon Cameras and six sub-panes with my 200mm F4 look very unrealistic at SQM of 21.6. I guess one big issue will be that people wil need to give back panels because of bad weather and this communication could be a bit difficult in a forum. How many tiles do you plan to communicate per moon cycle? It may be good to have a website with a status of the tiles for the current cycle to prevent ending up with a Swiss cheese style panorama. One idea could be to use the postings to the group to get an updated status of imaging. Thinking about a few ways to do that atm… Also, for people using APS-C it should be possible to share tiles, not sure if this works well but given the very low number of days with clear skies in the last 6 months it will be a challenge to get all 4 sub-tiles done in one month with the below 25% moon rule. One thing to think about would be to right now define a stretch goal of doing Dual Narrowband in the rest of the time on the same tile. When the number of tiles per moon cycle is not to high there should also be a possibility to overbook the most relevant tiles to make sure that the most relevant parts of the night sky will be complete after a cycle. Also contributors should agree to keep the individual frames for at least a year as processing might change over time. Also, I do not want to pollute my Astrobin Profile with the data for the Survey, how do you guys plan to do that, only share the private link to the group or do you create a folder for the results? Michael |
|
7.90
#...
·
|
|---|
| I don't see orientation requirements but I suppose if the field required is covered it shouldn't be much of a concern, or is it? Am also concerned that we're trying to bite more than we can chew (e.g. SNR=30). Furthermore field correction hasn't be specified, nor allowable distortion. |
|
1.20
#...
·
|
|---|
|
Regarding the challenge of allocating fields, I set up a Google Docs sheet here https://docs.google.com/spreadsheets/d/12s0AsefhP8gbVvGIOQRpp_k0HBroK3LV-lLpL6gXLDE/edit?usp=sharing This shows all of the fields, which ones have been 'claimed' (booked) and which ones have been completed. Users can 'book' a field by leaving a comment including their Astrobin ID in the 'Field ID' column. Actually, they can leave any comments here and respond to others' comments, so they can even 'negotiate' over fields if they want, or cancel their booking. On a regular basis I (or anyone in the coordinating team) can use the bookings to mark panels as Claimed or Complete. Of course, this doesn't stop us regularly advertising upcoming panels on the Astrobin forum as well, but I think this approach can provide a near real-time feedback for users on which fields are available. The 'Good Now' filter selects fields whose midnight culmination date is within 45 days of today, which is roughly equivalent to Brian's condition of fields culminating within 3 hours of midnight. This only really applies to low-to-mid DEC fields of course - fields close to the poles will probably be best imaged from high latitudes during local winter regardless of their RA. @andrea tasselli Responding to your most recent points:
I noted your comments (and Brian's) about problems using this tool. I am not a PI user so I have not tried the tool, only read the description in the documentation. What are problems you've found? If it is with the plate solving and alignment then this could prevent us using this tool at all. But if the problem is with seams or gradients then this is a different challenge. I'm interested to hear your experiences. @Brian Boyle The only clarification I'd add to your instructions would be to specify which images we need and which are optional (at the moment this is a bit ambiguous I feel). Must have: linear images (flat field/dark corrected, stacked, gradient removed etc as per the pipeline) Highly desired: stitched field image (as per discussion above, assuming MosaicByCoordinate can be shown to meet our needs) Optional: final image (stretched, noise reduction, sharpening etc etc) to the users' taste - not needed for the survey |
|
7.90
#...
·
|
|---|
James Tickner: First and foremost, I've used it the last time 2 years ago so something of what I'm about to say might have been improved. Caveat emptor spelled here the issues I've found: 1. At high galactic latitudes the plate solving might fail (and thus MosaicByCoordinates). 2. For unknown reasons the feathered areas might be subject to signficant local distortions (vortexes). One may have to resort to manual mosaicking. Reducing the extent of the overlap area also helps. 3. The above might be related to widely different PSFs in the feathered areas between overlapping panes. This is more of a supposition than a fact but the presence of field aberrations in the (local) PSFs does not help with transitions between one pane to the next. 4. Highly elaborate colour corrections (such as PSCC and PCC) might add an additional layer of hurdles in the transition zone between panes. I'd strongly recommend using only the simplest of colour correction procedures, that is CC in PI. This said I used it successfully for a rather largish mosaic of M31/110 (28 tiles), which you can see here (note that those tiles were taken in B6/7 skies): https://www.astrobin.com/6q60h0/?nc=collection&nce=2681 |
#...
·
·
1
like
|
|---|
Michael Ring: Unfortunately this would require some changes to AstroBin first. Right now images in the staging area cannot be published to groups, so either you guys do not publish images to the group, but simply share the link to private images in a forum topic in the group, or you change your gallery settings to begin using collections: https://welcome.astrobin.com/image-collections |
|
3.94
#...
·
|
|---|
| Hmm.. Is there a way to only add a folder to the homepage and to keep the images visible or to make one of the folders of a collection the default page? My homepage looks pretty empty when I switch to collections..... |
#...
·
|
|---|
Michael Ring: Unfortunately there is no way at the moment, but I could add support for invisible collections relatively easy. Your page looks empty if you switch to collections because you have no collections. Most people do collections for galaxies/nebulae/planets, etc. |
|
3.94
#...
·
|
|---|
| Honestly I am not a big fan of the collection pages, it makes sense for people with tons of images but not when you only have a limited number of pictures. I am fine with this collection folder beeing visible, but when the price to pay is to loose the easy access for pictures via the homescreen then I will will share the link to my Survey data. |
#...
·
·
1
like
|
|---|
Michael Ring: I can see if I can lift the requirement that images shared to groups must be public. I don't recall right now if there was a technical reason for that, I will need to check the code. In any case, you could always keep your images in the staging area, and instead of sharing them to the group the canonical way, you could send links to the person who coordinates the effort. |
|
0.00
#...
·
|
|---|
| @Brian Boyle I included your suggestions into my processing guide, which is now the second draft. It can be accessed via the same link as before, just for clarity, here it is again: https://docs.google.com/document/d/1_IdBbbtzMk3pUuYUPi0JJWMvjsvRtPmj/edit?usp=sharing&ouid=102793495713995642568&rtpof=true&sd=true I also added a rule for astronomical darkness (sun below -18°), just to make sure of this. Is that, and the rest of the second draft, OK for everyone? Regarding distortion correction in the mosaic scripts, I don't know if that is applied automatically, but AlignByCoordinates script can definitely do that. So there is a solution for that. If we find out, that the mosaic script can't do that, then I will add that into the processing guide. CS Gerrit |
|
16.52
#...
·
|
|---|
|
Hi All, Good discussion here, and it has thrown up a few issues about image sharing. A few things that are important 1) Using existing AB capabilities as far as possible i.e. not involve Salvatore in any more work. 2) Maximise visibility of images to all, with minimum work required to do so 3) Respect users rights not to pollute their own profile. [I didn't realise this was a thing, but I guess there are those who might worry about their Astrobin index which is fair enough.] Sticking to these principles, how about we simply setup a new user account for the survey? The downside of this is that the password would need to be shared between those posting to it, unless an account could be created with image universal read/write access only [slightly violating principle 1]. Even then, security issues might not allow this to happen. @Salvatore Iovene would this be an option If an account requires a subscription to be paid, I will happily do this. If such an account was possible, then we could, at least initially, have images sorted into dec strip collections for ease of access. How does that sound? On the orientation requirement @andrea tasselli raises a good point - and then more-or-less answers it I agree, that is would be desirable and most efficient to have sensor with a PA at 0, 90, 180 or 270, but I wouldn't necessarily insist. But I suggest we leave it to the individual to work out what works best for them, including perhaps even covering a patch on fields with one set of mosaiced panes. [With a 200mm system and a 10 x 7 degree field, the requirement for 11.5 x 11.5 fields with 15% overlap between individual panes might see me booking an RA range in neighbouring Dec strips.Finally here is a 20 pane mosaic I did last month with my wide-field. The panes were oriented at all forms of suboptimal directions and APP creaked a little in trying to stitch it together. But it worked Ok in the end. The larger mosaic (version C) didn't work so well, I think. https://www.astrobin.com/w39jpa/#r0 The average exposure time per pixel around 40 x 300 secs. Mosaicing was done with APP, and the imaging was taken consistent with @Astrogerdt pipeline procedure. This is fortunate, because these images were taken well before any of this discussion happened. I still do not know how we are going to put this all together. MosaicByCoordinates in PI does a great job with the astrometry, but GradientMergeMosaic does a very poor job with the stitching. [At least, I do a very poor job with GMM. If others experiences differ, please let me know.]. APP appears to do a much better job with the stitching, but it is very much a black box. I suspect that the astrometric solution at the field edges is not well constrained, and it appears to put quite a lot into the green channel as part of the multi-band blending process to remove the seams. Given what exists elsewhere is the astro-processing library, I am a little surprised that we don't appear to have a better solution yet. I might be that we do, and I don't know about it - or I know about it and am not using it properly - but since no-one has yet come forward, I don't think it is just me. CS Brian |
|
0.00
#...
·
|
|---|
| @Brian Boyle would you mind sharing some of the panels you used for the mosaic you posted (which is, by the way, absolutely stunning!)? Maybe I can give it a try over the next few days, I am also learning for some important exams, so that may take a while. I never did mosaics before, so no guarantee here, but maybe I can find something useful or some advice online, when I have the data available. CS Gerrit |
