Storage and Computers Other · Jay Stanley · ... · 33 · 1160 · 1

StuartT 4.69
...
· 
·  1 like
my laptop only has 1TB SSD so I only keep the data I am currently working on on that. Everything else I archive to a 5TB external drive. But I am shortly going to put all this in the cloud. As someone else said, data is precious and google backs up more reliably than me :-)
Like
DarkStar 18.93
...
· 
·  1 like
Michael Lehv:
John Hayes:
I've also upgraded the service at my home to get 30GB/sec upload speed so that I can more easily get at my data over the web.  Once fiber is available in my neighborhood, I'll sign up for 100+GB upload speed to make it really easy to access my data from anywhere.


John, where are you located that you can get 30GB/s upload speed? Is this business class, if so, who is your provider?

I asume this is a typo: should be MBit/s not GBit/s. Since the mainstream high speed network cards end at 10 GBit/s.
For higher speeds you need fiber optic multiplexing adapter. For their costs (and infrastructure) you get almost one CDK as give away for free. 😜
Like
gfunkernaught 2.41
...
· 
·  1 like
I'm the oddball in astrophotography as in I can get really good images with unorthodox methods.  I use the same calibration frames for all of my sessions, with a library for each camera/OTA setup, and various exposure lengths that image.  I use DeepSkyStacker and enable Dark Optimization to account for difference in sensor temps for the lights/darks.  Since my two rigs are f/6 and f/7, I almost exclusively shoot 5min subs, with 10min and 60sec-2min subs thrown in whenever I go to a dark site, and thus I have less subs than someone who shoots at less than 5min.  Planetary imaging files are also a huge chunk of my AP library.  I have an 8TB RAID 5 NAS served from my Ubuntu server, and 828GB of data (which could probably be reduced) since I started in Feb 2020.  I've tried PI and couldn't get interested enough to learn it, and found that it loves to create files files files.  Not necessary in my opinion when there are other options and routes to getting good results.  

Yes you can spend the money and get even more storage, but file management also counts big time.  Oh and don't use SSDs if you plan to build/use a NAS, SSDs aren't meant to be constantly written to, well written to more than they are read from.  I don't know how fast PI processes data, but DeepSkyStacker processes at less than my 1Gbps connection, so even if I had my images on a local SSD, it wouldn't matter.  

Also for backups:  Use an external HDD to periodically backup your data, and turn it off when not in use.  I have an external drive connected to my server and it runs nightly backups, then the drive spins down and sleeps when idle.
Edited ...
Like
jhayes_tucson 22.64
...
· 
·  2 likes
Michael Lehv:
John Hayes:
I've also upgraded the service at my home to get 30GB/sec upload speed so that I can more easily get at my data over the web.  Once fiber is available in my neighborhood, I'll sign up for 100+GB upload speed to make it really easy to access my data from anywhere.


John, where are you located that you can get 30GB/s upload speed? Is this business class, if so, who is your provider?

Michael,
Yes, I had to switch to a business account.  It was a little nutty.  It cost the same, they left the same equipment at my house and all that changed was the account number.  The maximum residential transfer rate that I could sign up for was 20 Mb/sec (up) and by switching to a business account, I got 30 Mb/sec (up).  The download speed stayed the same (as I recall I'm signed up for 300Mb/s).  They tell me that fiber should reach my neighborhood in another year or two and that gives symmetric up/down speeds.

John
Edited ...
Like
jhayes_tucson 22.64
...
· 
·  3 likes
Ruediger:
Michael Lehv:
John Hayes:
I've also upgraded the service at my home to get 30GB/sec upload speed so that I can more easily get at my data over the web.  Once fiber is available in my neighborhood, I'll sign up for 100+GB upload speed to make it really easy to access my data from anywhere.


John, where are you located that you can get 30GB/s upload speed? Is this business class, if so, who is your provider?

I asume this is a typo: should be MBit/s not GBit/s. Since the mainstream high speed network cards end at 10 GBit/s.
For higher speeds you need fiber optic multiplexing adapter. For their costs (and infrastructure) you get almost one CDK as give away for free. 😜

Yes, thanks for the catch.  That was a typo.  I'd love to have 30GB/s speeds in any direction!  I meant to say "30Mb/sec".  Note the lower case 'b' for "bits."  I tried to clean this up throughout the thread and I apologize for the error.

John
Like
IsabelaChase 0.00
...
· 
Hey Zuukz! I'm rocking a refurbished Dell myself. For storage, consider a good SSD for faster access to frequently used files. Keep the 2T HDD for the bulk. Backups on an external drive are a must. I organize project folders to keep things tidy. Good luck!
Like
menardre
...
· 
·  1 like
I probably go overboard storing files.   The following are the files that I keep:

I have a library of all my biases and darks for each temperature and each exposure setting. The library consists of 50 images at each setting and the 'master' stacked and processed final.

I keep all of my subframes and all of my flats for each image.

I use PixInsight for processing. Once I get to the point of having a stacked, aligned, and integrated image, I keep all of the processed images. For instance I keep the initial stacked/aligned/integrated image as the 'Master'. I then keep each version of the PixInsight processing steps such as DBE, SPCC, Stretching, CurvesTransformation, etc. That way I can always go back and reprocess starting for any given PixInsight step. I do not keep all of the intermediate files such as all of the Debayer or star alignment files that are used to develop the 'Master'.

I then routinely backup all of the files to a remove HDD.

Roger
Like
CCDnOES 5.61
...
· 
Just a note on this - I read that they expect the currently cheap SSD prices not to last and significant increases are expected in 2024 so stock up now. 

PC World Memory
Like
ODRedwine 1.51
...
· 
My storage disk is called " X-morsus-tera" .  A pigeon construct for 10-terabytes.

CS
David
Like
 
Register or login to create to post a reply.