Folder organization

[Diane Miller]

I'm just starting up the learning curve here and have a long way to go slogging through all the information.  But a basic thing I haven't found is the logical way to organize folders for a processing operation.  So far, I copy the relevant subs from the internal drive devoted to AP to the desktop, as the OS drive is a faster SSD.  I create a main folder (object name) which contains subfolders for lights, flats and biases (which will contain the master bias).  I'm just using a one-shot color camera -- Canon 7D Mk II.  Not using darks yet as I have read about issues with Canons. 

Beyond that I'm confused about what is the logical approach.  Thanks for any help. 

[lucchett]

Hi,
I have a calibration folder and an object folder.
In The object folder I have:
-R,G,B,L,HA,rejected, ETC folder s with lights
I put calibrated, cosmetic corrected and aligned lights in the same corresponding folders

- Master folder, in which I have the different combines
- final folder, in which I put the final results or very relevant intermediate images

In the main object folder I have my project file, that contains all the steps and intermediate images.
I start the project from the combines.

Andrea

[Diane Miller]

So you have folders for calibration, objects (and main objects??), master, final and a "project file" -- is that another folder?

Objects contains various lights
Master contains combines

I'm confused -- sorry. 

[Stu]

I do it a bit differently.  I have an SSD C: drive and a large D: drive.  I have an "imaging folder".  Within that folder I have "calibration file folder" and "imaging projects".  In the calibration file folder I keep the calibration files labelled by date.  I have a lot of space so I just keep the individual subs.  In the "imaging Projects" folder I'll actually copy the calibration master and put it in the relevent imaging project master folder.  Then within each imaging project I keep the relevent light and flat subs and the PI project.  Perhaps inefficient but works for me. 

[Diane Miller]

OK -- good on a top-level Calibration folder, which for me would only be a master bias, and probably subs, re-done every 6 months or so.  I'm afraid of Darks, as a Canon DSLR user (7D Mk II.)

I have other top-level folders for each object and within each is a different sets of lights + flats, by date.

I assume a project is a set of lights, flats and master bias for an object. 

To process, I'm copying one of those date runs to the desktop on my SSD system drive.  (Not up to trying to combine different nights yet -- just on the first baby step here.)  That's where I'm not sure how best to organize.  I've been making a folder for the object, and in it subfolders for lights and flats.  I assume I could just reference the master bias where it is on the other drive -- or if better I can easily copy it to another subfolder in the object folder. Or just copy the file -- don't see that it needs a subfolder. 

So on my desktop:

Object name -- this would be my Project folder?
   Under it:
      Lights
      Flats
      Bias Master (don't really need a folder for it, though?)

As I'm at the very first step here I'm not even sure what outputs I'll have as I go through the calibration stages and later -- guess I'll have to start cutting a path through the woods to really find out.  Really asking the question to help clarify the big picture.

But I'm guessing that logically I'd make a subfolder under Lights for calibrated Lights, and subfolders under that for subsequent stages?  Or maybe just rename and let them go in the same folder?

Thanks for any clarification!

[jkmorse]

I'll toss in my two cents worth.  I have mine organized as follows with each indent a nested subfolder:


1. Images Folder
 A. Imaging System Folder [Separate folders for each CCD/OTA combination; e.g., "STT8300 - e130D"]
     I.  Image Capture Date with label of target (e.g., "06Oct2015 - Veil West NB") [Note this requires a transfer from my field laptop to my desktop processing Computer]
       a. Master Calibration (Flats, Bias, Dark) folder [appropriate Bias and Dark copied from my Calibration Master Folder]
       b. Raw lights folder
          I.    Excluded subs folder
          II.   Calibration working folder (usually delete content once I get the particular stack integrated)
          III.  Process Folders
                A.  Process 1
                B.  Process 2
                C.  Process 3 (etc., for as many different processing attempts I make)

I also have a separate bundle of folders to hold all my calibration subs (Flats, Darks and Bias in folders labeled by dates), as well as for the calibration Masters for each of my CCDs.

I keep all of this on my 3Tb HDD but have a separate SSD where I keep a copy of any image capture folder and its subfolders that I am working on at the moment.

FWIW,

Jim

[jkmorse]

Note, it is also helpful when doing your processing to label each new image with the process step just completed so you can keep track if you want to back track.  So, for example, you might have a file named:

VeilWest_Ha_LF7050_DCrop_DBE_Reg_Decon_HT

meaning my Ha stack, integrated using Linear Fit with settings of 7.0 and 5.0, that I have dynamic cropped, run Dynamic Background Extraction on, registered (i.e., star aligned) with your other master stacks, performed a deconvolution run and finally stretched using Histogram Transformation.

Again, FWIW,

Jim

[Diane Miller]

Thanks -- that sounds like what I'm struggling toward, and am partway there!

[Stu]

Just to answer your "project" question.  In PI you can "save project" where it will save everything that you've done, including all the files, process icons, workspaces, settings, etc, so the next time you open it up it'll be exactly how you left it (including to where on the screen you left everything).  So if I'm doing different processes, I'll also do a "Process 1", "Process 1", etc.  The only drawback is that the resulting amount of data you dump on to your hard drive can be HUGE, especially if you have a lot of clones.  I try to keep everything under control by deleting the stuff I don't want. 

But heck, external storage is cheap. 

I'll also add that periodically I back up everything on two separate hard drives. 

[jkmorse]

+1 Stu on that backup, especially for Raw data.  You can always reprocess but you can never get those raw subs back if you lose them.

[Diane Miller]

I'm very careful about backups so don't expect problems there.  The original files get copied to another drive to be worked on, although I realize I won't hurt them.  (Only maybe insult them, at my current level of understanding.)   

I've preached backups (along with how to do them) to a lot of photography people till I'm blue in the face, and personally know 3-4 who have lost everything because they didn't listen.

[chris.bailey]

I do mine a little differently. I run various scope/camera combinations so have a folder for each, under that I separate 'projects' by imaging season, which is typically Sept through to April so my current one is 15-16. Under that I keep a current calibration masters folder but then put a copy of the current calibration masters in each project folder so I know I have a set of calibration frames in with the images they apply to. So my folder layout is then

FSQ85_SX814
   15-16
      Sept2015 - Current Calibration Masters
      SeptOct2015 - Veil Mosaic
         Cal (this is the folder that contains a copy of the calibration masters current at that time)
         Ha
            Rejected
         SII
            Rejected
         OIII
            Rejected

Like Jim I will then save the straight integrations e.g. Veil_Mosaic5_HaIntegration and then major process steps Veil_Mosaic5_HaIntegration_Crop_DBE.

I often find I go back to data at a future date, either adding to it or just re-processing as new methods crop up. I find the important thing is to develop a system of folder hierarchy and file naming and stick to it.

As you say, backup, backup and then backup again. Big portable hard drives and NAS's are now cheap so few excuses for not doing it.

Chris