File management: What files do I need to keep?

[MikeOates]

I am finding that this hobby of image taking and processing just eats up hard disk space, and I am running out of room, and I have about 9TB and it's filling up fast.

So my question is, which files can I safely delete and not wish that I hadn't in the future. I am not referring to data files, they can always be reworked when I learn how to process better or better software or methods become available. It's the calibration files, bias, darks, flats, they seem to take up more space. If I have created master Bias, darks, flats, are those going to be enough so I can delete all the individual frames and just keep the masters?

I am just trying to find out what people tend to do.

Thanks,

Mike

PS. Of course when I said above "better software", I do mean improvements to PI

[bitli]

Hi,
9 TB seems to be a lot to me - I tend to re-use the same dark/bias for a while (may be too long), and I put these raws on a separate drive. But I tend to keep the raw, just in case. I am still way below 1GB. I guess that raw bias/dark are the only candidate for elimination (may be flats too if you are confident that you processed them correctly). The question is which percentage of your data is this.  Unless the saving is very significant, I would keep all data (maybe offline), unless it is obviously bad.  It proved usesull to have the raw to reprocess images, even with different versions of PI.  Or you may once want to try another software, just to see how much PI is better  .
Whatever you do be sure to have a very clear naming/folder convention, otherwise you will not find your data when needed. Some self advertisement: you may use the script FITSFileManager to rename and sort the data by date/temperature/binning...

Currently I do the processing of a single image on an SSD of 120GB, and it has largely enough room to process one image (when done I move them back to the HD). By curiosity, do you have very large images or a lot of them (or both)? Do you have seaparate dark/bias for each nigh or do you reuse them?
-- bitli

[Andres.Pozo]

I take new bias/dark frames twice a year. After generating the master images I keep the RAW images but I compress them.

For the light and flats frames, I keep the raw images and the masters result of the preprocessing. I usually delete the intermediate images (calibrated and registered).

The idea is to never delete a raw image to be able to redo the full processing in the future if there is any kind of improvement on the software or to be able to combine them with new images.

[NGC7789]

I take new bias/dark frames twice a year.

It would seem to me that one would need to keep the darks, bias and flats (at least the masters) associated with a given set of lights. If future advances in software prompt reprocessing from the beginning a current set of calibration frames may not match lights taken years ago. Remember too that those advances may have to do with the creation of the masters themselves so it may be prudent to keep the individual files.

I agree that there's no need to keep the intermediate images.

[stevek]

Keep the calibrated light frames (in case you want them for a project later on) but delete the flats and BIAS when you are sure the lights are properly calibrated.  They have done their job after you have calibrated the lights.

[MikeOates]

Thank you all for the ideas. I was forgetting about all the intermediate files, so getting rid of those will save a lot of space. With DSLR imaging you end up with far more, & larger files, so I will concentrate on pruning those. Then see if I need to reduce further files from the CCD camera.

9TB does sound like a lot of space, but half of that is a duplicate for backup and there are lot of other files on there, from photoography and other stuff.

Mike

[chris_todd]

I do mostly DSLR as well, and the files are huge, so once I've generated masters (lights, dark, flat), and I'm reasonably satisfied that the dark and flat masters are good, I'll throw away all the intermediate files for darks, flats, and lights, as well as the dark and flat original RAW files (with my DSLR, I take fresh darks and flats every session, so I have never seen a need to keep the originals as long as the integration and calibration looks good).

For the lights, I'll archive the original RAW files to a DVD so I can get them off the hard drive, and have a safe permanent backup.  I suppose I could do the same with the dark and flat RAW, but like I said, I rarely bother.

I have master BIAS I've generated for every ISO I'm likely to use, and I only kept the masters (I threw away the orginal RAW and all the intermediates).

So after image calibration and integration, the only thing I have on my 3TB drive is the master dark, flat, and light frames for each image or imaing session, and my master bias for each ISO.  So far, after almost two years, I only have about 1 TB filled on my 3TB drive.

[Nocturnal]

A word of warning about using DVDs for backup. They tend to be pretty finicky about being read with the same drive that wrote them. I've had several instances where disks I wrote with drive A could not be read with drive B. And naturally I only used drives that supported the same + or - type. I've also had corrupted or unreadable files on writable DVDs. To summarize, I don't use them for anything important anymore. Their capacity and write speed is also quite pathetic so I now keep copies on hard drives.

[papaf]

Take a look at some of those pretty inexpensive NAS. Pretty fast, at least faster than a DVD, and secure too.

[MikeOates]

Take a look at some of those pretty inexpensive NAS. Pretty fast, at least faster than a DVD, and secure too.

Thanks, yes I will look into that option. I never trust DVD's as Nocturnal pointed out, they are often not readable on other drives and as time passes the data degrades.

Mike

[NGC7789]

...a safe permanent backup...

DVDs, especially ones burnt in a home burner, are not permanent. They have a life expectancy of about 10 years under ideal storage conditions and are vulnerable to heat and particularly sunlight. Hard drives also are not permanent. The reality is that for the large sizes of modern media files there is no archival media (that is one that will last for 100+ years). The best counter for this is redundancy. The rule of thumb is the 3-2-1 strategy. Three copies of the data (the original counts), on two different media with one copy off site.

Regardless of the storage strategy organization becomes a big issue too. If you do decide to go back a reprocess old images you have to be able to find the light and calibration frames that go together.

[Nocturnal]

Yes, data organization is an issue. I store all my stuff in an AstroImages directory with 'captures' and 'projects' underneath. Captures are organized by date, projects by current, done, redo and hopeless. I use fast local SSD storage for whatever files I'm working with and use SyncToy to 'contribute' files from that local SSD to my Windows Home Server NAS (attached via GbE so it's reasonably snappy). On the NAS the data is mirrored so it is protected against single media failure. Since that does not protect against accidental deletion I take full backups of my NAS to 2 SATA disks that I keep separate from the NAS. I rotate those disks. The one thing I need to add is keeping one of those disks outside the house. Actually in the observatory would be a fine place. Now that I use TrueCrypt to encrypt the entire volume I feel comfortable doing that.

So yeah, data management can get quite involved, even at home. Working on enterprise storage boxes for EMC, HDS and Netapp doesn't help that