Pixinsight file sizes?

2.41 Sean Mc #... · view views · 1 like
Howdy! google fu is failing me the question is: why are my pixinsight masters so ridiculously large? I’m shooting with the 294mm pro in bin 1. Fike size straight out if the camera is around 91mb. integrated masters (darks, flats, lights etc) are around 365mb. If i 2x drizzle, they are 1.2gigs each. do i have some setting messed up in pixinsight? Why the heck is an integrated master 4x the size of the original image? disk space is running out FAST THX ALL! Like

2.41

#...

· 1 like

Howdy!

google fu is failing me

the question is: why are my pixinsight masters so ridiculously large? I’m shooting with the 294mm pro in bin 1. Fike size straight out if the camera is around 91mb.

integrated masters (darks, flats, lights etc) are around 365mb. If i 2x drizzle, they are 1.2gigs each.

do i have some setting messed up in pixinsight? Why the heck is an integrated master 4x the size of the original image?

disk space is running out FAST

THX ALL!

0.90 cashew #... · view views · 1 like
Integrated masters are usually 32Bit floating point -> 2x original size. If you drizzle 2x then file size goes up 4x. You can try saving it as a compressed xisf. Like

1.20 Ron Crouch #... · view views
Well the 1st thing is the processed files immediately grow from 16bit to 32bit. You really do not want to change that back to 16bit or your combined results are going to really suffer. At least 32bit is really needed to be able to properly render the color or grayscxale depth. Then the 2x drizzle doubles once again! I use a qhy294mono pro and raw files are 4164x2795 unbinned. That is 27.745 MB or 181.96 mb. Processed subs grow to 45.487MB. Drizzle that 2X and there you are at 91MB. Like

1.20

Ron Crouch

#...

Well the 1st thing is the processed files immediately grow from 16bit to 32bit. You really do not want to change that back to 16bit or your combined results are going to really suffer. At least 32bit is really needed to be able to properly render the color or grayscxale depth. Then the 2x drizzle doubles once again!

I use a qhy294mono pro and raw files are 4164x2795 unbinned. That is 27.745 MB or 181.96 mb. Processed subs grow to 45.487MB. Drizzle that 2X and there you are at 91MB.

8.59 Jon Rista #... · view views
If you are integrating to 64-bit float, then that right there should account for the 4x data size growth. You are going from 16 bits (2 bytes) per pixel, to 64 bits (8 bytes) per pixel. You might need 64-bit float...it depends on how many subs you are stacking. For a master dark, I would think 32-bit would be fine, but for deep light stacks, sometimes you want to use 64-bit. But that would only apply to the final stack, not the intermediates...for most of the intermediates you'll generally want to use 32-bit float. Of course, if you drizzle, and THOSE are 64-bit float...then yes, you would indeed use a lot of space per file. Like

8.59

Jon Rista

#...

If you are integrating to 64-bit float, then that right there should account for the 4x data size growth. You are going from 16 bits (2 bytes) per pixel, to 64 bits (8 bytes) per pixel.

You might need 64-bit float...it depends on how many subs you are stacking. For a master dark, I would think 32-bit would be fine, but for deep light stacks, sometimes you want to use 64-bit. But that would only apply to the final stack, not the intermediates...for most of the intermediates you'll generally want to use 32-bit float.

Of course, if you drizzle, and THOSE are 64-bit float...then yes, you would indeed use a lot of space per file.

0.00 Jim Morse #... · view views
Simple math. Drizzle 2x2 and you increase size 4x. After doing all of your processing you can rescale the image to get it back under control size wise. I usually do a 50% rescale before uploading to Astrobin. That gets the image back under 25MB. Like

0.90 Jeff Rothstein #... · view views
Hi Sean. You might consider drizzling at 1x, which gets you the subtle benefit of the drizzle without upsizing the image. Best, Jeff Like

0.90 Ian Parr #... · view views · 3 likes
That is exactly how it works. 2x drizzle quadruples to file size. Have you considered deleting the Calibration and registraton files I suspect 99.9% of most mortal users never need. These are the system generated sub-folders under the working folder that I use a script to delete immediately after the WBP completes. Calibrated Registered cosmetized Logs There also a lot of superfluous files created in the master folder alongside the actual drizzle files you actually want to work with e.g. LN_* etc I use a script to move the ones I do not want to a JUNK folder on another drive and rename the ones I want to something much shorter. e.g. masterLight_BIN-1_3388x2712_EXPOSURE-180.00s_FILTER-Red_mono_drizzle_2x_autocrop.xisf masterLight_BIN-1_3388x2712_EXPOSURE-180.00s_FILTER-Green_mono_drizzle_2x_autocrop.xisf masterLight_BIN-1_3388x2712_EXPOSURE-180.00s_FILTER-Blue_mono_drizzle_2x_autocrop.xisf masterLight_BIN-1_3388x2712_EXPOSURE-180.00s_FILTER-Red_mono_autocrop.xisf masterLight_BIN-1_3388x2712_EXPOSURE-180.00s_FILTER-Green_mono_autocrop.xisf masterLight_BIN-1_3388x2712_EXPOSURE-180.00s_FILTER-Blue_mono_autocrop.xisf masterLight_BIN-1_3388x2712_EXPOSURE-180.00s_FILTER-Red_mono_drizzle_2x.xisf masterLight_BIN-1_3388x2712_EXPOSURE-180.00s_FILTER-Red_mono.xisf masterLight_BIN-1_3388x2712_EXPOSURE-180.00s_FILTER-Green_mono_drizzle_2x.xisf masterLight_BIN-1_3388x2712_EXPOSURE-180.00s_FILTER-Green_mono.xisf masterLight_BIN-1_3388x2712_EXPOSURE-180.00s_FILTER-Blue_mono_drizzle_2x.xisf masterLight_BIN-1_3388x2712_EXPOSURE-180.00s_FILTER-Blue_mono.xisf LN_Reference_Light_BIN-1_3388x2712_EXPOSURE-180.00s_FILTER-Red_mono.xisf LN_Reference_Light_BIN-1_3388x2712_EXPOSURE-180.00s_FILTER-Green_mono.xisf LN_Reference_Light_BIN-1_3388x2712_EXPOSURE-180.00s_FILTER-Blue_mono.xisf masterLight_BIN-1_3388x2712_EXPOSURE-180.00s_FILTER-Red_mono_drizzle_2x_autocrop.xisf -> RED.xisf masterLight_BIN-1_3388x2712_EXPOSURE-180.00s_FILTER-Green_mono_drizzle_2x_autocrop.xisf -> GREEN.xisf masterLight_BIN-1_3388x2712_EXPOSURE-180.00s_FILTER-Blue_mono_drizzle_2x_autocrop.xisf -> BLUE.xisf Also when you open those files they contain an unwanted weighting component I a pretty sure most people have no use for other than to perhaps direct cropping. So I close the extra weightings files and then use this in the Process Console to just save. The XISF files size halves. save * --nomessages --noverify Another neat trick is to use an Image Container that contains a list of open views by their Image Identifier and drag that list of images to a saved instance of the ViewID process that has been configured to change the long winded Indentifier to something more obvious and readable that is 100% consistent with the new file name. That Image Container can also be dropped on a Gradient Correction (*new) or GraxPert process icon to quickly get those jobs done. My next step is to repeat for BlurXTerminator for Star Correction (only) and I only use Non Stellar correction later on the combined image. var P = new Script; P.filePath = "$PXI_SRCDIR/scripts/mschuster/ViewIdReplace/ViewIdReplace.js"; P.md5sum = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"; P.parameters = [ // id, value ["version", "1.4"], ["pattern", "_integration_autocrop"], ["replacement", ""] ]; P.information = ""; All this does is save a huge amount of time and diskspace. Seriously someone should write a Cleanup script or they should add an option to the WBP to allow these extra disk hogs to be automatically removed. I am an Excel VBA developer so I use that. There may be users who like to dive down the rabbit hole of logs and calibration files after WBP but I suspect for most users they just want to get stuck into the end results and get on with it. Maybe someone has shares in drive manufacturers or has a panic attack if those files get deleted but there is allways therapy available. Good Luck Cheers Ian Like

0.90

Ian Parr

#...

· 3 likes

That is exactly how it works. 2x drizzle quadruples to file size. Have you considered deleting the Calibration and registraton files I suspect 99.9% of most mortal users never need. These are the system generated sub-folders under the working folder that I use a script to delete immediately after the WBP completes.
Calibrated
Registered
cosmetized
Logs
There also a lot of superfluous files created in the master folder alongside the actual drizzle files you actually want to work with e.g. LN_* etc
I use a script to move the ones I do not want to a JUNK folder on another drive and rename the ones I want to something much shorter.
e.g.
masterLight_BIN-1_3388x2712_EXPOSURE-180.00s_FILTER-Red_mono_drizzle_2x_autocrop.xisf
masterLight_BIN-1_3388x2712_EXPOSURE-180.00s_FILTER-Green_mono_drizzle_2x_autocrop.xisf
masterLight_BIN-1_3388x2712_EXPOSURE-180.00s_FILTER-Blue_mono_drizzle_2x_autocrop.xisf
masterLight_BIN-1_3388x2712_EXPOSURE-180.00s_FILTER-Red_mono_autocrop.xisf
masterLight_BIN-1_3388x2712_EXPOSURE-180.00s_FILTER-Green_mono_autocrop.xisf
masterLight_BIN-1_3388x2712_EXPOSURE-180.00s_FILTER-Blue_mono_autocrop.xisf
masterLight_BIN-1_3388x2712_EXPOSURE-180.00s_FILTER-Red_mono_drizzle_2x.xisf
masterLight_BIN-1_3388x2712_EXPOSURE-180.00s_FILTER-Red_mono.xisf
masterLight_BIN-1_3388x2712_EXPOSURE-180.00s_FILTER-Green_mono_drizzle_2x.xisf
masterLight_BIN-1_3388x2712_EXPOSURE-180.00s_FILTER-Green_mono.xisf
masterLight_BIN-1_3388x2712_EXPOSURE-180.00s_FILTER-Blue_mono_drizzle_2x.xisf
masterLight_BIN-1_3388x2712_EXPOSURE-180.00s_FILTER-Blue_mono.xisf
LN_Reference_Light_BIN-1_3388x2712_EXPOSURE-180.00s_FILTER-Red_mono.xisf
LN_Reference_Light_BIN-1_3388x2712_EXPOSURE-180.00s_FILTER-Green_mono.xisf
LN_Reference_Light_BIN-1_3388x2712_EXPOSURE-180.00s_FILTER-Blue_mono.xisf

masterLight_BIN-1_3388x2712_EXPOSURE-180.00s_FILTER-Red_mono_drizzle_2x_autocrop.xisf -> RED.xisf
masterLight_BIN-1_3388x2712_EXPOSURE-180.00s_FILTER-Green_mono_drizzle_2x_autocrop.xisf -> GREEN.xisf
masterLight_BIN-1_3388x2712_EXPOSURE-180.00s_FILTER-Blue_mono_drizzle_2x_autocrop.xisf -> BLUE.xisf
Also when you open those files they contain an unwanted weighting component I a pretty sure most people have no use for other than to perhaps direct cropping. So I close the extra weightings files and then use this in the Process Console to just save. The XISF files size halves.

save * --nomessages --noverify

Another neat trick is to use an Image Container that contains a list of open views by their Image Identifier and drag that list of images to a saved instance of the ViewID process that has been configured to change the long winded Indentifier to something more obvious and readable that is 100% consistent with the new file name. That Image Container can also be dropped on a Gradient Correction (*new) or GraxPert process icon to quickly get those jobs done. My next step is to repeat for BlurXTerminator for Star Correction (only) and I only use Non Stellar correction later on the combined image.

var P = new Script;
P.filePath = "$PXI_SRCDIR/scripts/mschuster/ViewIdReplace/ViewIdReplace.js";
P.md5sum = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx";
P.parameters = [ // id, value
["version", "1.4"],
["pattern", "_integration_autocrop"],
["replacement", ""]
];
P.information = "";

All this does is save a huge amount of time and diskspace.
Seriously someone should write a Cleanup script or they should add an option to the WBP to allow these extra disk hogs to be automatically removed.
I am an Excel VBA developer so I use that.
There may be users who like to dive down the rabbit hole of logs and calibration files after WBP but I suspect for most users they just want to get stuck into the end results and get on with it. Maybe someone has shares in drive manufacturers or has a panic attack if those files get deleted but there is allways therapy available.

Good Luck
Cheers
Ian

0.90 cashew #... · view views
Jon Rista: You might need 64-bit float...it depends on how many subs you are stacking. For a master dark, I would think 32-bit would be fine, but for deep light stacks, sometimes you want to use 64-bit. But that would only apply to the final stack, not the intermediates...for most of the intermediates you'll generally want to use 32-bit float. I was wondering which criteria can be used to estimate when it's better to switch to 64-bit. Would I be able to see it in the streched image? Effects like posterisation? Like

0.90

cashew

#...

Jon Rista:
You might need 64-bit float...it depends on how many subs you are stacking. For a master dark, I would think 32-bit would be fine, but for deep light stacks, sometimes you want to use 64-bit. But that would only apply to the final stack, not the intermediates...for most of the intermediates you'll generally want to use 32-bit float.

I was wondering which criteria can be used to estimate when it's better to switch to 64-bit.
Would I be able to see it in the streched image? Effects like posterisation?

5.49 David Rosenthal #... · view views
You also might be saving the rejection maps as part of the masters. Check the settings Like

4.21 Alex Ranous #... · view views
If you're really tight on space, you can configure PixInsight to always compress .xisf files. If you go to the Format Explorer tab and select xisf, you and edit the preferences and specify the data be compressed. I'm sure you'll get a performance hit due to all the compression/decompression going on, but I haven't measured it so I don't know how much of a hit it will be. Like

0.00 Ballard #... · view views
Hi Sean My .fits subframes are each ~120meg (16 bits using a ASI 6200mm camera which is 62 megapixel binning 1x). normally integrate the images and save them as 32bit floating point .xsif files which come out to ~239meg each. When I combine the integrated LRGB frames they can get large ~ 720meg each. So it does use a lot of space. Are you setting the 64 bit option in the integration as this would double the size. When you use 2x drizzle that will quadruple the image size which is what is probably what is causing the size increase. Since I keep most of the calibrated, Registered and integrated files I will eat up 50 to 100gig or more per image. So In my case I bought an 8 port NAS server and put 8 eight tera byte drives into it but keep 1 on hot standby and use the NAS in RAID 6 for double redundancy so get around 40 terabytes available. I also keep separate backups on regular 18 terabyte drives which I purchased when they went on sale. Like

0.00

Ballard

#...

Hi Sean
My .fits subframes are each ~120meg (16 bits using a ASI 6200mm camera which is 62 megapixel binning 1x). normally integrate the images and save them as 32bit floating point .xsif files which come out to ~239meg each. When I combine the integrated LRGB frames they can get large ~ 720meg each. So it does use a lot of space. Are you setting the 64 bit option in the integration as this would double the size. When you use 2x drizzle that will quadruple the image size which is what is probably what is causing the size increase.

Since I keep most of the calibrated, Registered and integrated files I will eat up 50 to 100gig or more per image. So In my case I bought an 8 port NAS server and put 8 eight tera byte drives into it but keep 1 on hot standby and use the NAS in RAID 6 for double redundancy so get around 40 terabytes available. I also keep separate backups on regular 18 terabyte drives which I purchased when they went on sale.

2.41 Sean Mc #... · view views
Ballard: Hi Sean My .fits subframes are each ~120meg (16 bits using a ASI 6200mm camera which is 62 megapixel binning 1x). normally integrate the images and save them as 32bit floating point .xsif files which come out to ~239meg each. When I combine the integrated LRGB frames they can get large ~ 720meg each. So it does use a lot of space. Are you setting the 64 bit option in the integration as this would double the size. When you use 2x drizzle that will quadruple the image size which is what is probably what is causing the size increase. Since I keep most of the calibrated, Registered and integrated files I will eat up 50 to 100gig or more per image. So In my case I bought an 8 port NAS server and put 8 eight tera byte drives into it but keep 1 on hot standby and use the NAS in RAID 6 for double redundancy so get around 40 terabytes available. I also keep separate backups on regular 18 terabyte drives which I purchased when they went on sale. Aha. This is my point. You’re using a higher res camera with an individual sub of 120mb and getting a master that’s 239. without drizzle my individual sub is 91mb and my L master is 360mb. of course I can’t do ANYTHING else in pixinsight since wbpp is running :/ not even check settings. Pretty sure i have 32bit on, not sure about saving the rejection map as part of the master. I’m assuming there’s no harm in not saving the rejection map? Like

2.41

Sean Mc

#...

Ballard:
Hi Sean
My .fits subframes are each ~120meg (16 bits using a ASI 6200mm camera which is 62 megapixel binning 1x). normally integrate the images and save them as 32bit floating point .xsif files which come out to ~239meg each. When I combine the integrated LRGB frames they can get large ~ 720meg each. So it does use a lot of space. Are you setting the 64 bit option in the integration as this would double the size. When you use 2x drizzle that will quadruple the image size which is what is probably what is causing the size increase.

Since I keep most of the calibrated, Registered and integrated files I will eat up 50 to 100gig or more per image. So In my case I bought an 8 port NAS server and put 8 eight tera byte drives into it but keep 1 on hot standby and use the NAS in RAID 6 for double redundancy so get around 40 terabytes available. I also keep separate backups on regular 18 terabyte drives which I purchased when they went on sale.

Aha. This is my point. You’re using a higher res camera with an individual sub of 120mb and getting a master that’s 239.

without drizzle my individual sub is 91mb and my L master is 360mb.

of course I can’t do ANYTHING else in pixinsight since wbpp is running :/

not even check settings. Pretty sure i have 32bit on, not sure about saving the rejection map as part of the master. I’m assuming there’s no harm in not saving the rejection map?

8.59 Jon Rista #... · view views · 1 like
Jon Rista: You might need 64-bit float...it depends on how many subs you are stacking. For a master dark, I would think 32-bit would be fine, but for deep light stacks, sometimes you want to use 64-bit. But that would only apply to the final stack, not the intermediates...for most of the intermediates you'll generally want to use 32-bit float. I was wondering which criteria can be used to estimate when it's better to switch to 64-bit. Would I be able to see it in the streched image? Effects like posterisation? It is highly doubtful you would need to use 64-bit float outside of teh final integration itself. Your intermediates shouldn't need to be 64-bit. As for how to know when to switch to 64-bit... Well, first, as far as I gathre now, WBPP seems to use 64-bit by default, which IMO is a mistake. That would indeed balloon data file size by 4x from 16-bit raw FITS files or whatever source data you have. (Drizzling is NOT needed here to make the intermediates 4x larger, all that is needed is 64-bit float!!) I used to have a formula for calculating when I was stacking enough subs to warrant 64-bit. Its been a while and I don't remember it off the top of my head. I usually switched once I got up to a couple hundred subs or more. Another reason to switch to 64-bit is if you are doing HDR processing, and combining short exposures with longer exposures. You'll most likely need 64-bit in those cases. I do know that PI now supports these PSF based weighting algorithms, and I don't yet know enough about them to know if that might do some kind of HDR-like scaling. If so, then you might need 64-bit if you use that as well. Key thing though, is you really should NOT need 64-bit files for everything...only the final integration, unless you are using a ton of files for your masters. Like

8.59

Jon Rista

#...

· 1 like

Jon Rista:
You might need 64-bit float...it depends on how many subs you are stacking. For a master dark, I would think 32-bit would be fine, but for deep light stacks, sometimes you want to use 64-bit. But that would only apply to the final stack, not the intermediates...for most of the intermediates you'll generally want to use 32-bit float.

I was wondering which criteria can be used to estimate when it's better to switch to 64-bit.
Would I be able to see it in the streched image? Effects like posterisation?

It is highly doubtful you would need to use 64-bit float outside of teh final integration itself. Your intermediates shouldn't need to be 64-bit.

As for how to know when to switch to 64-bit... Well, first, as far as I gathre now, WBPP seems to use 64-bit by default, which IMO is a mistake. That would indeed balloon data file size by 4x from 16-bit raw FITS files or whatever source data you have. (Drizzling is NOT needed here to make the intermediates 4x larger, all that is needed is 64-bit float!!)

I used to have a formula for calculating when I was stacking enough subs to warrant 64-bit. Its been a while and I don't remember it off the top of my head. I usually switched once I got up to a couple hundred subs or more.

Another reason to switch to 64-bit is if you are doing HDR processing, and combining short exposures with longer exposures. You'll most likely need 64-bit in those cases. I do know that PI now supports these PSF based weighting algorithms, and I don't yet know enough about them to know if that might do some kind of HDR-like scaling. If so, then you might need 64-bit if you use that as well.

Key thing though, is you really should NOT need 64-bit files for everything...only the final integration, unless you are using a ton of files for your masters.

8.59 Jon Rista #... · view views
Jim Morse: Simple math. Drizzle 2x2 and you increase size 4x. After doing all of your processing you can rescale the image to get it back under control size wise. I usually do a 50% rescale before uploading to Astrobin. That gets the image back under 25MB. He was quadrupling before drizzling. He went from 91mb raw frames, to 365mb just after integrating his masters. Then, drizzling quadrupled again to 1.2gigs. Obviously, the drizzling at 2x means 4x as many pixels. The original quadrupling though, is most likely a bit depth thing. WBPP enables 64-bit float by default now. Like

8.59

Jon Rista

#...

Jim Morse:
Simple math. Drizzle 2x2 and you increase size 4x. After doing all of your processing you can rescale the image to get it back under control size wise. I usually do a 50% rescale before uploading to Astrobin. That gets the image back under 25MB.

He was quadrupling before drizzling. He went from 91mb raw frames, to 365mb just after integrating his masters. Then, drizzling quadrupled again to 1.2gigs. Obviously, the drizzling at 2x means 4x as many pixels.

The original quadrupling though, is most likely a bit depth thing. WBPP enables 64-bit float by default now.

2.41 Sean Mc #... · view views
How do i set wbpp bit depth? I can’t even figure out how to check what it’s set to. last integration just crashed most of the way through. 8 hours in. Task manager wouldn’t even end the pixinsight task :/ stupid windows 11. i have 160 frames of 91mb each. Masters (without drizzle) are coming out to 360mb each. It seems excessive. I have an i71300k with 32g ram and 3 ssd drives. Like

0.90 Ian Parr #... · view views
8 hours! Shudder. Sadly no CUDA (yet) for WBP. I have 128 GB ram and a 32 Gb Ram disk (R where i create multiple TEMP folders and then set these up under Directories and Network Preferences. I found creating multiple folders even on the same Drive can make quite a bit of difference but even a 32 Gb Ram Disk can hit the wall working with large 2xDrizzled data sets from my ASI2600MM Pro but Pixinsight just complains when it looses a swap file but does not crash. So I can close un-necessary images and just keep the one I was working on to try again. Windows has a lot of back ground processes like telemetry etc and I use Chris Titus's WinUtil to dramatically improve performance when those un-necessary process are turned off. He has a YouTube site which is pretty good but it basically comes down to this shortcut on my desktop to open the utility and get rid of the bad resource hogs. I suggest at least creating a System Restore Point before that and have a system backup of Drive C done just in case but haven't heard of any catastrophes from using it on a stable system. C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -ExecutionPolicy Bypass -Command "Start-Process powershell.exe -verb runas -ArgumentList 'irm https://christitus.com/win \| iex' The other option might be to divide and conquer by running WBP for each channel seperately and then aligning the final masters. Before getting CUDA to work (it needs an NVIDIA GPU ) it would take 15-20 minutes to run things like BlurXTerminator but with CUDA that dropped to around 60 seconds! There is also Fast Integration on pre-cropped data sets. Trial WBP with just a few subs to work out the crop required and apply it to the raw subs before jumping in head first. Adam Block has a video on YouTube for that. https://youtu.be/04wJtg7Vm08?si=3B_yWDzfRiRdYXQD Good luck. Like

0.90

Ian Parr

#...

8 hours! Shudder. Sadly no CUDA (yet) for WBP. I have 128 GB ram and a 32 Gb Ram disk (R

where i create multiple TEMP folders and then set these up under Directories and Network Preferences. I found creating multiple folders even on the same Drive can make quite a bit of difference but even a 32 Gb Ram Disk can hit the wall working with large 2xDrizzled data sets from my ASI2600MM Pro but Pixinsight just complains when it looses a swap file but does not crash.
So I can close un-necessary images and just keep the one I was working on to try again.

Windows has a lot of back ground processes like telemetry etc and I use Chris Titus's WinUtil to dramatically improve performance when those un-necessary process are turned off. He has a YouTube site which is pretty good but it basically comes down to this shortcut on my desktop to open the utility and get rid of the bad resource hogs. I suggest at least creating a System Restore Point before that and have a system backup of Drive C done just in case but haven't heard of any catastrophes from using it on a stable system.

C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -ExecutionPolicy Bypass -Command "Start-Process powershell.exe -verb runas -ArgumentList 'irm https://christitus.com/win | iex'

The other option might be to divide and conquer by running WBP for each channel seperately and then aligning the final masters.

Before getting CUDA to work (it needs an NVIDIA GPU ) it would take 15-20 minutes to run things like BlurXTerminator but with CUDA that dropped to around 60 seconds! There is also Fast Integration on pre-cropped data sets. Trial WBP with just a few subs to work out the crop required and apply it to the raw subs before jumping in head first. Adam Block has a video on YouTube for that.

https://youtu.be/04wJtg7Vm08?si=3B_yWDzfRiRdYXQD

Good luck.

2.41 Sean Mc #... · view views
Yeah i have a 4070ti and cuda enabled. It really helps with blurx. i still don’t know why my masters are so big. Like

8.59 Jon Rista #... · view views
Sean Mc: How do i set wbpp bit depth? I can’t even figure out how to check what it’s set to. last integration just crashed most of the way through. 8 hours in. Task manager wouldn’t even end the pixinsight task :/ stupid windows 11. i have 160 frames of 91mb each. Masters (without drizzle) are coming out to 360mb each. It seems excessive. I have an i71300k with 32g ram and 3 ssd drives. Sorry, I just read about the 64-bit in some forum posts. Now that I'm looking at WBPP, I don't see an option...which means it may be hard-coded into the script. I haven't spent any time looking at WBPP's code lately, so I am not sure where this might be set. I am curious why the option is not exposed...you would think it would be pretty important. The use of 64-bit MAY only be for the stack, and if that is the case, then honestly, that is probably fine. Since you only end up with one file from the ImageIntegration process (plus maybe some rejection maps). I would only be truly concerned, if all the intermediate files were 64-bit as well. It might be worth asking about that on the PI forums... If WBPP DOES use 64-bit for everything, IMHO it would be extremely wasteful. Like

8.59

Jon Rista

#...

Sean Mc:
How do i set wbpp bit depth? I can’t even figure out how to check what it’s set to.

last integration just crashed most of the way through. 8 hours in. Task manager wouldn’t even end the pixinsight task :/ stupid windows 11.

i have 160 frames of 91mb each. Masters (without drizzle) are coming out to 360mb each. It seems excessive. I have an i71300k with 32g ram and 3 ssd drives.

Sorry, I just read about the 64-bit in some forum posts. Now that I'm looking at WBPP, I don't see an option...which means it may be hard-coded into the script. I haven't spent any time looking at WBPP's code lately, so I am not sure where this might be set. I am curious why the option is not exposed...you would think it would be pretty important. The use of 64-bit MAY only be for the stack, and if that is the case, then honestly, that is probably fine. Since you only end up with one file from the ImageIntegration process (plus maybe some rejection maps).

I would only be truly concerned, if all the intermediate files were 64-bit as well. It might be worth asking about that on the PI forums... If WBPP DOES use 64-bit for everything, IMHO it would be extremely wasteful.

2.41 Sean Mc #... · view views
Looks like it’s using a 32bit float wile running ¯\_(ツ)_/¯ Like

8.59 Jon Rista #... · view views
Sean Mc: Looks like it’s using a 32bit float wile running ¯\_(ツ)_/¯ So, are ALL of your intermdiate files, 365 megs each? Or are only certain intermdiates that large? I can't think of another reason why a raw 91meg source file, would end up being approximately four times larger, unless the pixel data was using 4x as many bits. Some of the files will be header space, and its possible that XISF might allocate header space differently than FITS. But, only the raw pixel data could really account for a 4x change in file size. Only other thing that might account for such a difference, is if your original frames are somehow compressed? I am not sure how many pixels your sensor has. You could multiply the pixel count by the bit depth, then divide by 8 bits per byte to come up with an expected file size. If that computed number comes out to around 2x your actual file size, then maybe some kind of compression is involved. Outside of that....32 bit float should only be doubling your file size. Just to double check...you ARE SURE that your camera's native bit depth is 16 bit? Like

8.59

Jon Rista

#...

Sean Mc:
Looks like it’s using a 32bit float wile running ¯\_(ツ)_/¯

So, are ALL of your intermdiate files, 365 megs each? Or are only certain intermdiates that large? I can't think of another reason why a raw 91meg source file, would end up being approximately four times larger, unless the pixel data was using 4x as many bits. Some of the files will be header space, and its possible that XISF might allocate header space differently than FITS. But, only the raw pixel data could really account for a 4x change in file size.

Only other thing that might account for such a difference, is if your original frames are somehow compressed? I am not sure how many pixels your sensor has. You could multiply the pixel count by the bit depth, then divide by 8 bits per byte to come up with an expected file size. If that computed number comes out to around 2x your actual file size, then maybe some kind of compression is involved. Outside of that....32 bit float should only be doubling your file size.

Just to double check...you ARE SURE that your camera's native bit depth is 16 bit?

2.41 Sean Mc #... · view views
It’s actually a 14bit sensor. asi294mm pro bin1 fits are 91mb each. master lights (and darks and flats) are 365mb each. i reset wbpp to default and it’s still the same. No drizzle. ¯\_(ツ)_/¯ Like

0.90 Ian Parr #... · view views
ASI 2600 MM Pro generated files from last night Light_NGC3372_180.0s_Bin1_2600MM_RED_gain100_20240309-002626_-9.6C_0008.fit 52,194,240 masterLight_BIN-1_6248x4176_EXPOSURE-180.00s_FILTER-RED_mono.xisf 121,975,096 masterLight_BIN-1_6248x4176_EXPOSURE-180.00s_FILTER-RED_mono_drizzle_2x.xisf 903,686,144 WBP with RGB 53 fits takes me about 10-15 minutes with 2x drizzle on my Desktop gaming machine with a 24 core Thread Ripper with 128 Gb Ram. My trusty old i7 laptop with 32 Gb ram and fast M2 SSD drives would take hours and hours and ... zzzzzz. The combined RED files are saved succinctly as RED.xisf and after I remove the weightings the file drops from 903,686,144 with about a 10-15% crop to 354,485,728. RGB Combined it comes out as RGB.xisf 1,063,432,608 As someone who started professionaly with a IBM PC AT with a 286 processor and 640 Kb ram in the mid eighties, and a $10,000 20 Mb Tall Grass hard drive when everyone else there used 5 1/4 inch floppies, these numbers never fail to amuse me :-) I hear CD's are making a comeback and will soon hold 1 petabyte. On that basis alone we should be planning for Terabyte fits files and Petabyte combined data sets and we will need at least an Exabyte of ram to run Pixinsight. Edited ... Like

0.90

Ian Parr

#...

ASI 2600 MM Pro generated files from last night

Light_NGC3372_180.0s_Bin1_2600MM_RED_gain100_20240309-002626_-9.6C_0008.fit 52,194,240

masterLight_BIN-1_6248x4176_EXPOSURE-180.00s_FILTER-RED_mono.xisf 121,975,096

masterLight_BIN-1_6248x4176_EXPOSURE-180.00s_FILTER-RED_mono_drizzle_2x.xisf 903,686,144

WBP with RGB 53 fits takes me about 10-15 minutes with 2x drizzle on my Desktop gaming machine with a 24 core Thread Ripper with 128 Gb Ram.

My trusty old i7 laptop with 32 Gb ram and fast M2 SSD drives would take hours and hours and ... zzzzzz.

The combined RED files are saved succinctly as RED.xisf and after I remove the weightings the file drops from 903,686,144 with about a 10-15% crop to 354,485,728.

RGB Combined it comes out as RGB.xisf 1,063,432,608

As someone who started professionaly with a IBM PC AT with a 286 processor and 640 Kb ram in the mid eighties, and a $10,000 20 Mb Tall Grass hard drive when everyone else there used 5 1/4 inch floppies, these numbers never fail to amuse me :-)

I hear CD's are making a comeback and will soon hold 1 petabyte.

On that basis alone we should be planning for Terabyte fits files and Petabyte combined data sets and we will need at least an Exabyte of ram to run Pixinsight.

Edited ...

0.90 Chris Ashford #... · view views · 2 likes
Sean Mc, The 294MM is a 47MP camera which, at 2bytes (16 bits) per pixel, should give you over 90MBytes per .fits file. Regardless of the dynamic range of the image itself (you may have selected gain to give you a 12 bit or 14 bit image for instance), whatever computer is controlling your rig will create a 16 bit file. That's the size of file that computers are designed to operate with. I have a 294MC Pro which gives me a resolution of 41442822 (bin 2), and an image size of 11.7MP - comes out at just over 22 MBytes. But in your case with the MM version you must be using the unlocked bin1 mode which gives you a resolution of 82885644 pixels....right? Hence your file size is 4 times my file size. Pixinsight increases your image dynamic range from 16 bits to 32 bits so that when stretching and editing your images you get better resolution and avoid problems like "posterization" of your data. That doubles the size of your files. So, expect your .xisf files to be more than ~180MBytes. You can configure PI to create 64 bit images if you feel your target needs even higher dynamic range (like the Orion Nebula for instance) - in that case you'd end up with ~360Mbyte files. Make sure you don't have "Create 64 bit image" selected in WBPP. Then, if you drizzle, that quadruples the size of your file. Question - do you need to use drizzle? Drizzle only helps you when you are undersampling. By that I mean you are "undersampling" if your sensor pixel size is bigger than your scope optical resolution. Since the 294MM in bin1 mode pixel size is 2.315um (very small) you might find that drizzle doesn't yield any benefit to your image. This will save you quite a lot of memory if you stop using drizzle which can also be turned off in WBPP. FYI, James Lamb has made a great video which explains optical versus sensor resolution: (1091) Telescope Resolution, FWHM, and HFR - YouTube Finally, if you have selected "Create Rejection Files" in WBPP Integration settings then your masters will increase in size by a factor of 3. If you are really short of computer memory then consider using your camera in bin2 mode to reduce the raw file size, and make sure that "Create 64 bit images" isn't ticked, untick "Create RejectionFiles" and stop using drizzle. Delete intermediate files of old image processing and consider getting yourself some sort of archiving system to free up space. For storing data I personally only keep the raw files and delete everything else once I have finished processing an image. Before starting image processing I always check that I have 200GB of memory available. I use two 8 TB regular disk drives for archiving to clear files off my fast M2 SSD drives. I hope this helps!! Good Luck Chris Ashford Edited ... Like

0.90

Chris Ashford

#...

· 2 likes

Sean Mc,

The 294MM is a 47MP camera which, at 2bytes (16 bits) per pixel, should give you over 90MBytes per .fits file. Regardless of the dynamic range of the image itself (you may have selected gain to give you a 12 bit or 14 bit image for instance), whatever computer is controlling your rig will create a 16 bit file. That's the size of file that computers are designed to operate with.

I have a 294MC Pro which gives me a resolution of 4144*2822 (bin 2), and an image size of 11.7MP - comes out at just over 22 MBytes. But in your case with the MM version you must be using the unlocked bin1 mode which gives you a resolution of 8288*5644 pixels....right? Hence your file size is 4 times my file size.

Pixinsight increases your image dynamic range from 16 bits to 32 bits so that when stretching and editing your images you get better resolution and avoid problems like "posterization" of your data. That doubles the size of your files. So, expect your .xisf files to be more than ~180MBytes. You can configure PI to create 64 bit images if you feel your target needs even higher dynamic range (like the Orion Nebula for instance) - in that case you'd end up with ~360Mbyte files. Make sure you don't have "Create 64 bit image" selected in WBPP.

Then, if you drizzle, that quadruples the size of your file. Question - do you need to use drizzle? Drizzle only helps you when you are undersampling. By that I mean you are "undersampling" if your sensor pixel size is bigger than your scope optical resolution. Since the 294MM in bin1 mode pixel size is 2.315um (very small) you might find that drizzle doesn't yield any benefit to your image. This will save you quite a lot of memory if you stop using drizzle which can also be turned off in WBPP. FYI, James Lamb has made a great video which explains optical versus sensor resolution: (1091) Telescope Resolution, FWHM, and HFR - YouTube

Finally, if you have selected "Create Rejection Files" in WBPP Integration settings then your masters will increase in size by a factor of 3.

If you are really short of computer memory then consider using your camera in bin2 mode to reduce the raw file size, and make sure that "Create 64 bit images" isn't ticked, untick "Create RejectionFiles" and stop using drizzle. Delete intermediate files of old image processing and consider getting yourself some sort of archiving system to free up space.

For storing data I personally only keep the raw files and delete everything else once I have finished processing an image. Before starting image processing I always check that I have 200GB of memory available. I use two 8 TB regular disk drives for archiving to clear files off my fast M2 SSD drives.

I hope this helps!! Good Luck

Chris Ashford

Edited ...

8.59 Jon Rista #... · view views
Sean Mc: It’s actually a 14bit sensor. asi294mm pro bin1 fits are 91mb each. master lights (and darks and flats) are 365mb each. i reset wbpp to default and it’s still the same. No drizzle. ¯\_(ツ)_/¯ Ah! Ok. Then that could account for the difference. At the very least, your intermediates will be 32-bit float. That would at the very least, increase your intermediates to around 200 megs each. It is also possible that PI is including other data alongside the main data. For MASTERS...and, I wasn't aware it was ONLY the masters involved here (there can be a lot more intermediate files than just masters)...PI will often include rejection maps. So when you integrate a master dark, master flat, and master light...if you used a rejection algorithm, the rejection maps may also be embedded in the same .xsif file. So, that could account for the additional space usage. If it really is JUST the MASTERS...then, that is just three files. If you go through the full integration workflow with PI, there are many other tools involved, and there can be intermediates from each. Calibration will produce intermediate individual light frames. Cosmetic Correction will produce another set of intermediates. So will Star Registration. If you do any kind of evaluative process, and have it generate weighted outputs, that's another set of intermediates. Etc. ImageIntegration will then take the final fully pre-processed set of lights, and integrate them, producing a SINGLE file. The single files produced from ImageIntegration are not really the big issue. If you are concerned about disk space, you need to be looking at the sheer volume of intermediate light frames. If you stack 100 lights, then in the end, you could end up with the original 100, 100 more from calibration, 100 more from cosmetic correction, 100 more from registration, at the very least. That is 400 files in total, each 91 megs, or 36.4 gigs. I never delete my original raw lights, but after I'm done with an image, I will delete the intermediates. With the originals, I can pre-process again if I want. Further, you can often compress your individual raw frames into .zip files to compress them (or a solid RAR, and compress them even more!) to save additional space long term. A compressed solid RAR will basically allow the compression algorithm to look across files for similar repeatable patterns, which can greatly increase the compression ratio. You can't extract individual files from a solid, you can only extract the entire thing, but it can save a lot of space. Like

8.59

Jon Rista

#...

Sean Mc:
It’s actually a 14bit sensor.

asi294mm pro bin1

fits are 91mb each.

master lights (and darks and flats) are 365mb each.

i reset wbpp to default and it’s still the same. No drizzle.

¯\_(ツ)_/¯

Ah! Ok. Then that could account for the difference. At the very least, your intermediates will be 32-bit float. That would at the very least, increase your intermediates to around 200 megs each. It is also possible that PI is including other data alongside the main data. For MASTERS...and, I wasn't aware it was ONLY the masters involved here (there can be a lot more intermediate files than just masters)...PI will often include rejection maps. So when you integrate a master dark, master flat, and master light...if you used a rejection algorithm, the rejection maps may also be embedded in the same .xsif file. So, that could account for the additional space usage.

If it really is JUST the MASTERS...then, that is just three files. If you go through the full integration workflow with PI, there are many other tools involved, and there can be intermediates from each. Calibration will produce intermediate individual light frames. Cosmetic Correction will produce another set of intermediates. So will Star Registration. If you do any kind of evaluative process, and have it generate weighted outputs, that's another set of intermediates. Etc. ImageIntegration will then take the final fully pre-processed set of lights, and integrate them, producing a SINGLE file.

The single files produced from ImageIntegration are not really the big issue. If you are concerned about disk space, you need to be looking at the sheer volume of intermediate light frames. If you stack 100 lights, then in the end, you could end up with the original 100, 100 more from calibration, 100 more from cosmetic correction, 100 more from registration, at the very least. That is 400 files in total, each 91 megs, or 36.4 gigs. I never delete my original raw lights, but after I'm done with an image, I will delete the intermediates. With the originals, I can pre-process again if I want. Further, you can often compress your individual raw frames into .zip files to compress them (or a solid RAR, and compress them even more!) to save additional space long term. A compressed solid RAR will basically allow the compression algorithm to look across files for similar repeatable patterns, which can greatly increase the compression ratio. You can't extract individual files from a solid, you can only extract the entire thing, but it can save a lot of space.

2.97 Roman Pearah #... · view views
The master files are actually bundles of multiple views, including the image itself, along with rejection maps (both high and low). For Drizzle masters, it will contain the Drizzle weights. For autocropped masters, the crop mask. You'll note that when you open these in PI, several views appear. So you should see the size reflect the number of bundled views (3 views = 3x the file size). You can disable the inclusion of these files in the WBPP settings. Edited ... Like

2.97

Roman Pearah

#...

The master files are actually bundles of multiple views, including the image itself, along with rejection maps (both high and low). For Drizzle masters, it will contain the Drizzle weights. For autocropped masters, the crop mask. You'll note that when you open these in PI, several views appear. So you should see the size reflect the number of bundled views (3 views = 3x the file size). You can disable the inclusion of these files in the WBPP settings.

Edited ...

	Distinct awarded users	Total awarded images
Image of the day
Top picks
Top pick nominations

	Image of the day	Top pick	Top pick nominations	Total submitted
Deep sky
Solar system
Extremely wide field
Star trails
Northern lights
Noctilucent clouds
Landscape

	Image of the day	Top pick	Top pick nominations	Total submitted
Backyard
Traveller
Own remote observatory
Amateur hosting facility
Public amaeteur data
Professional, scientific grade data
Mix of multiple sources
Other
Unknown

Image Index:

Contribution Index (beta):

Pixinsight file sizes? Pleiades Astrophoto PixInsight · Sean Mc · ... · 34 · 1132 · 3

Image Index:

Contribution Index (beta):

Pixinsight file sizes? Pleiades Astrophoto PixInsight · Sean Mc · ... · 34 · 1132 · 3

Sorry

Sorry

Sorry :-(

Sorry

Sorry

Sorry

Sorry :-(

Sorry

Sorry

Sorry

Sorry :-(

Sorry

Sorry

Sorry

Sorry :-(

Sorry

Sorry

Sorry

Sorry :-(

Sorry

Sorry

Sorry

Sorry :-(

Sorry

Sorry

Sorry

Sorry :-(

Sorry

Sorry

Sorry

Sorry :-(

Sorry

Sorry

Sorry

Sorry :-(

Sorry

Sorry

Sorry

Sorry :-(

Sorry

Sorry

Sorry

Sorry :-(

Sorry

Sorry

Sorry

Sorry :-(

Sorry

Sorry

Sorry

Sorry :-(

Sorry

Sorry

Sorry

Sorry :-(

Sorry

Sorry

Sorry

Sorry :-(

Sorry

Sorry

Sorry

Sorry :-(

Sorry

Sorry

Sorry

Sorry :-(

Sorry

Sorry

Sorry

Sorry :-(

Sorry

Sorry

Sorry

Sorry :-(

Sorry

Sorry