0.00
#...
·
 |
|---|
|
First of all there is a great bunch of people here helping and I appreciate that immensely!! What do most of you do for storage and what files should I be storing. I don't have that many images yet but over 2T of data. That's lights, darks, flats, bias and some final images. Also I need to upgrade computers but the new Win11 and Apple machines are out of my budget. I currently have an older Dell duo core with 12g of ram 2T HDD and a USB 2T HDD. I've got my eye on a newer refurb Dell with 32g of ram a 1T SSD NVME and a 2T HDD and Win10Pro, it won't run Win11. I think the plan with Microsoft is to discontinue support for Win10 in about 3 years and I'm hoping prices may come down some by then. Thanks and Clear Skies, Zuukz |
11.01
#...
·
·
1
like
|
|---|
|
Great question, and one I've been asking myself too as my data drive is almost at its limits. Personally, I am backing up all my captures to an external drive and removing them from my main processing rig. I'm keeping calibrated frames that I have selected via PixInsight's SubFrameSelector, but discarding the raw images, the registered, calibrated, measured directories etc.. I have a 2TB NVme drive for my main processing data drive and it is always getting full if I don't archive data/clean-up temporary files. |
2.11
#...
·
·
2
likes
|
|---|
|
I always thought I had enough storage woth a 2TB hard drive. I know have 3 external 2TB hard drives and I am building a NAS with 4 hard drives all 8TB each. However, I save everything and back up everything in triplicate. I also save calibrated fits files along with the raw fits incase I want to add to the data in following years. CS Dan. |
7.75
#...
·
·
4
likes
|
|---|
|
Data is precious - in particular if you consider what amount of time you invest getting and processing it. So saving on data storage is the wrong spot in my opinion. After all we are doing all of this for the data and the final image. One of the unique features of our hobby is that every year you can collect additional data on the same subject, so taking good care about your data is crucial in my opinion. I keep 3 copies of my data:
So even if the house burns down at least I still have my data ... With respect to astrophotography I save
I don't store intermediate files created by PI in the process of image integration like cosmetic corrected or registered images. I would be able to recreate them if necessary. Data is growing fast, my QHY 268m creates 50 MB fits raw files, so 20 GB of Data for one object is no exception. With my 8 TB SSD for storage in my PC plus additional 4 TB NVMe SSD for processing (about 50% free each) I still have some leeway though. NAS can get extended without issues anyway. Clear skies Wolfgang |
|
7.90
#...
·
·
1
like
|
|---|
| I only keep the final data files and delete every raw file older than 2 years. Life is too short to waste on collecting data than, in the end, will never ever get used again. I know, I have data 20 years old on CDs and they are there just as mementos of glory days long gone. |
|
3.15
#...
·
·
1
like
|
|---|
|
Cold storage is relatively cheap but want some redundancy so a NAS was the solution and then the first NAS filled up so another bigger one added. CS Sean  |
0.00
#...
·
·
1
like
|
|---|
| Great thread! Out of curiosity: You are not considering cloud storage? I use OneDrive only, and upload all projects - lights, darks, flats, bells'n whistles - there, after discarding bad subs, creating the final image, etc. Then again I'm fairly new to this, so I may have missed out on some key considerations among those in the know. |
|
7.75
#...
·
·
2
likes
|
|---|
Great thread! Out of curiosity: You are not considering cloud storage? I use OneDrive only, and upload all projects - lights, darks, flats, bells'n whistles - there, after discarding bad subs, creating the final image, etc. Then again I'm fairly new to this, so I may have missed out on some key considerations among those in the know. My third copy using CrashPlan by Code 24 is a cloud backup solution. Obviously you need a good internet connection to cope with the data volume. Nothing wrong with using cloud storage. Clear skies Wolfgang |
0.90
#...
·
·
3
likes
|
|---|
|
I use private NAS in Hybrid RAID (old symphony one from 2013 with automatic backup from my computer) for saving : - light - master dark - master flat - master offset - pixinsight project with the final image - configuration files of software (including NINA sequence) This is not perfect but protect me from HDD crash. |
1.43
#...
·
·
1
like
|
|---|
|
Hi everyone, In my case I use 2 external HDD (not SSD nor NVMe) and I use the SSD from the laptop just for the OS, the applications, and some documents and of course I have a folder in where I have the files that I will stack. In the external HDD there are files for storage purpose only. Clear skies! Daniel. |
0.00
#...
·
·
1
like
|
|---|
| For me, I don't store anything on my computer. Don't want to clog it up. Everything goes on my external 8TB hard drive. This includes all the calibration libraries, PI work and raw image files coming off the camera. Each morning I also copy the night's image files to a second portable SSD drive. There's no processing work on that one, just the raw images. |
|
1.43
#...
·
·
1
like
|
|---|
|
I forgot to say that my laptops are not the best option. The older one is a MackBookPro i5 4 cores 16GB RAM from 2012 The newer is a Dell Windows 11 i7 4 cores 16 GB RAM from 2021 But I think that the best option will be a desk computer, something powerful like a pc gaming or something like that. It would be fun to do a poll to find out the trend of colleagues with their equipment. Let's do it. Here you are: https://www.astrobin.com/forum/c/astrophotography/other/do-you-use-a-laptor-or-a-desk-computer-for-satacking-and-processing/ Clear skies! Daniel. |
7.63
#...
·
·
1
like
|
|---|
| I backup my files from my laptop to two 5TB portable drives on a semi-regular basis, normally when I complete a target's final PI processing or when I've collected a few night's worth of data, whichever comes first. I save all light subs (after culling the rejects) and each session's master flat, master dark flat, master light as well as the final PI image. Once the masters are created I discard all of the original flats and dark flats. I also backup my XL logsheet of my imaging history, my library of master darks and other misc. files that I have accumulated. I've considered cloud backup but apart from uploading my final JPG images to my Smugmug photo site there's just too much data to make uploading it all convenient. |
4.95
#...
·
·
2
likes
|
|---|
|
Its interesting to see all the different configs people use. I knew immediately when I got into this hobby that data storage would be kind of a big deal. I have a minor IT background and built my own systems since 1995. Going into this I hadn't built I system in nearly 10 years so I had a bit of a learning curve as far is new tech goes, particularly on the storage side if things. My 10 year old system was a 3rd gen I7 quad@ 3.2GHZ with 32gb DDR1600. It was designed originally for gaming and had dual GTX 660ti's. It had a total of 750gb SSD's on 2 drives along with another 4tb in SATA HDs and an additional 1tb Seagate portable. This system was fine for my initial DSLR processing with DSS, GIMP and Affinity photo. When I purchased my GEM45 and ASI2600 I decided to step into Pixinsight and really felt it was time to modernize. I purchased a Clevo laptop(bare bones laptop) with 11th Gen I7 running Ubuntu Linux exclusively with 2tb NVME. This is how it benches PI: CPU Identification CPU vendor ............. GenuineIntel CPU model .............. 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz System Information Platform ............... Linux / Ubuntu 22.04.1 LTS Operating system ....... 5.15.0-47-generic #51-Ubuntu SMP Thu Aug 11 07:51:15 UTC 2022 GNU/LinuxCore version ........... PixInsight Core 1.8.9-1 (x64)Logical processors ..... 8 Total memory size ...... 31.136 GiB Execution Times Total time ............. 01:24.97 CPU time ............... 01:18.00 Swap time .............. 00:06.95 Swap transfer rate ..... 2385.722 MiB/s Performance Indices Total performance ...... 5536 CPU performance ........ 4853 Swap performance ....... 13213 This is the system I have been processing PI with for the last 8 months. If I was a little more motivated (or fully retired) I could probably improve the performance by tweaking the swap system, but my I only have so much learning capacity at my age. I added a 5TB Seagate portable HD for backup. I am still using an inexpensive laptop outdoors for capture and guiding but plan on moving to Astroberry at some point (already have my box, but need to learn KStars/EKOS before the full leap), but in the meantime move files with a 1TB Sandisk Extreme portable SSD. Additionally, I just recently built an I9, 64GB DDR4-4400, 2TB NVME, Radeon 6900xt desktop that I am currently running Win11 on, primarily as a gaming system. But I got the itch to try it out for my astro endeavors and have a 2nd NVME drive on the way to dual boot Linux (not sure if I will stick with Ubuntu distro). I plan to test out PI with the GPU processing techniques I have briefly read about. I probably got away fro the OP question a bit. So far I tend to store everything, lights, calibration frames. I have started building a dark library, which will save some space in the future. But for the most part, I delete nothing but junk lights like focus problems, guiding farts, Elon Musk's photo bombs, and so on. If there is any hill I will die on as far as recommendations, I would encourage saving money and adding stability/speed/support by shifting to a Linux platform. The modern GUIs make it a pretty smooth transition from Windows/Mac. I dabbled with Unix based systems in the late 90's/early 2000's and it is so much easier to work with for the hobbyist plebe like myself now than back then. |
|
0.00
#...
·
|
|---|
Thanks all, this is some great information. I'm not a total computer idiot but I can do just enought to have to call a tech  Chime in on this, I'm looking at a Dell with Win11 (I klnow my Apple freinds here may be shaking their head) I have other software to run that won't on an Apple. Here,s the specs:Dell Inspiron 3910 Business Desktop Computer I 12th Gen Intel 6-core i5-12400 Processor 32GB DDR4 2TB SSD + 2TB HDD Intel UHD Graphics 730 DisplayPort HDMI DVD-RW Bluetooth Win11 and then add a USB 6T backup. The technology changes so rapidly now that I don't think I could ever be on the "cutting edge" but it's time to retire the old Dell Optiplex 780 circa 2009. I'd love to get Pixinsight but my old machine can't run that software. Guess I should go back to my old Fortran programming days CS Zuukz |
|
7.75
#...
·
·
1
like
|
|---|
Jay Stanley: The main question when evaluating a PC is always: What is your main use case for it? In case it's astrophotography image processing I would recommend at least an i7 cpu and some more RAM. The i5 with 32 GB will do the job too, but the performance boost of an i7 and more RAM will be noticable if budget allows. Clear skies Wolfgang |
|
0.00
#...
·
|
|---|
| Thanks Wolfgang, I wish I could but that configuration would run about a grand extra and not in the budget for now. I haven't commited to any purchase yet so maybe I should save longer but it sure would be nice to have something better than this old duo core 12 gig of ram. |
|
7.75
#...
·
·
1
like
|
|---|
|
Everything else being equal the price difference should not be that much. You may check the AMD Ryzen 7 options too. Starting with an I7/Ryzen 7 with 32 GB is another sensible option. You just have to check with the dealer that there are still free RAM slots available for a later RAM upgrade (this means you need 2x16 GB RAM instead of 4x8GB). However, as I already pointed out, the i5 with 32 GB will do the job too, just somewhat slower. It will significantly outperform your old setup. Even with the i5 I would check whether a future RAM upgrade is possible. Clear skies Wolfgang |
3.34
#...
·
·
1
like
|
|---|
| I use Western Digital RAID drives for all my basic storage. I have about 6 of these, each in the 4-8 TB range. I have a separate 1TB SSD that is used for post processing and short term storage. Once completed, image projects are moved to RAID. I also have separate copies of key stored data on separate storage systems. |
 #...
·
·
1
like
|
|---|
|
I keep all of my 'lights' and 'flats' on a 4TB SDD. I have a library of darks and biases on another 4 TB disk. The OS and all programs are on a VME drive, and all other personal data is on a separate SSD. All of this gets backup up to a 12 TB external drive. Roger |
0.00
#...
·
·
1
like
|
|---|
|
I'm very much concerned about backups of my raw data and data that has been processed by PI. On-site backups provide some redundancy but are still vulnerable to fires, lighting, hackers, etc. I was already doing cloud backups to a local datacenter for my non-astro data so including my astro data was merely incremental. However, even the datacenter went down once and had to institute its own backups to AWS. The biggest bottleneck with cloud backups is data upload speed. Even with a 800 mbs or 1 gig cable download speed, upload speed is usually only around 40 mbs. Backing up the day's data can take hours. The only solution is a fiber connection and those are only now starting to become available here in the U.S. for the last mile and are also very expensive. It seems that ensuring reliable backups is an endless battle. Michael |
|
7.75
#...
·
·
1
like
|
|---|
|
The offline backup already protects you against everything except theft and influence of the elements. So while there is some gap in your protection with a one day delay of an online backup, for private data it's acceptably low. Probability that the event strikes exactly while you're transferring data is low if you don't live in Chile with 300 nights of clear skies and data. In the worst case you're losing (part of) your last session which is bad but not catastrophic for amateurs. For a business obiously other rules and risk assessments apply. Clear skies Wolfgang |
22.76
#...
·
·
2
likes
|
|---|
|
I have a 2TB SSD that acts as a raw data buffer on the PC that runs my remote scope. I have it set up to automatically upload images as they are acquired to a primary 48TB NAS system. The primary NAS is backed up on a daily basis to an identical off-site secondary NAS. This approach allows cleaning out the PC drive on the scope periodically without any concern about losing anything. I've also upgraded the service at my home to get 30Mb/sec upload speed so that I can more easily get at my data over the web. Once fiber is available in my neighborhood, I'll sign up for 100+Mb/sec upload speed to make it really easy to access my data from anywhere. For local processing, I keep everything on a 5TB USB drive. They cost roughly $100 and typically last 3-5 years before I have to buy another one--and I keep everything. The QHY600PH that I run produces ~130MB files so that's a LOT of data! Fortunately, my calibration files (flats, darks, bias) are very long lasting so they don't take up too much room. I try to manually back up my USB "working drive" to a second one every so often but I have to admit that I'm not as rigorous as I should be to keep everything constantly backed up. It's good to answer this question because it makes me actively think about this weakness in my data storage strategy. It has been a little too long since I backed up my current working drive. John |
0.90
#...
·
·
1
like
|
|---|
|
I keep lights and calibration frames on external hard drives. Calibration frames are masters only since I reuse the dark frames after the first integration run and the flat and flat darks until something changes with the optical train. Those are kept with the individual projects that use them. The raw uncalibrated lights are kept because as the image processing software continues to evolve there is always a reason to go back and reprocess. My imaging computer is an AMD with 32 cores,128GB RAM and 2TB NVME drives. The extra power is needed given the size of the frames. |
|
0.00
#...
·
·
1
like
|
|---|
John Hayes: John, where are you located that you can get 30GB/s upload speed? Is this business class, if so, who is your provider? |
