Author Topic: BatchPreprocessing issue  (Read 3676 times)

Offline Thorsten Lockert

  • Newcomer
  • Posts: 36
    • Personal Home Page
BatchPreprocessing issue
« on: 2013 January 02 19:40:01 »
I have found one mis-feature with BatchPreprocessing. It affects, at the very least, flats, and probably lights.

If you add in image files that have the same *filenames* (from separate folders, e.g. folders by date), the calibration step will write the calibrated images to a new folder. However, if you have a file named e.g. "2012-12-29/flat-Blue-1x1-30C-001.fit" and one named "2013-01-01/flat-Blue-1x1-30C-001.fit" these both gets written as "calibrated/flat/flat-Blue-1x1-30C-001.fit" into the destination folder. Then, when the integrated master is created, such images get used multiple times.

What I saw in my case was 26 images in 2012-12-29 and another 101 images in 2013-01-01 (I have an EL panel, so getting a good number of flats is not much of an issue)... So I ended up with 101 images in the "calibrated/flat" folder, while ImageIntegration reported that it had used 127 images. Looking carefully at the log, the images with sequence numbers 001 through 026 had each been used twice.

I have not checked if this also happens with lights, but I assume it will if they otherwise are the same. Bias and dark frames appear not to be a problem as they are not being calibrated (even, for darks, when "Optimize dark frames" are enabled...?).

I have been going maybe a bit overboard (?), making my masters from integrations of 127 raw frames (bias, 90 minute darks and flats for each filter) -- the only really time consuming ones to acquire being the darks, which must be done over multiple nights.

Thorsten

Offline Thorsten Lockert

  • Newcomer
  • Posts: 36
    • Personal Home Page
[1.8RC3] Re: BatchPreprocessing issue
« Reply #1 on: 2013 January 28 16:31:31 »
Just tried this using a set of Hydrogen Alpha lights, collected with ACP Scheduler... The input looked like:

Code: [Select]
$ ls -R
Flaming Star Nebula-HAlpha (13-14)/     Flaming Star Nebula-HAlpha (7-8)/
Flaming Star Nebula-HAlpha (15-16)/     Flaming Star Nebula-HAlpha (9-10)/
Flaming Star Nebula-HAlpha (3-4)/       master-cal/
Flaming Star Nebula-HAlpha (5-6)/

./Flaming Star Nebula-HAlpha (13-14):
Flaming Star Nebula-S001-HAlpha filter-R001-Ha.fts

./Flaming Star Nebula-HAlpha (15-16):
Flaming Star Nebula-S001-HAlpha filter-R001-Ha.fts

./Flaming Star Nebula-HAlpha (3-4):
Flaming Star Nebula-S001-HAlpha filter-R001-Ha.fts
Flaming Star Nebula-S001-HAlpha filter-R002-Ha.fts

./Flaming Star Nebula-HAlpha (5-6):
Flaming Star Nebula-S001-HAlpha filter-R001-Ha.fts
Flaming Star Nebula-S001-HAlpha filter-R002-Ha.fts

./Flaming Star Nebula-HAlpha (7-8):
Flaming Star Nebula-S001-HAlpha filter-R001-Ha.fts
Flaming Star Nebula-S001-HAlpha filter-R002-Ha.fts

./Flaming Star Nebula-HAlpha (9-10):
Flaming Star Nebula-S001-HAlpha filter-R001-Ha.fts
Flaming Star Nebula-S001-HAlpha filter-R002-Ha.fts

./master-cal:
Master_Bias_4096x4096_Bin1x1_Temp-30C.fit
Master_Dark_4096x4096_Bin1x1_Temp-30C_5400s.fit
Master_Flat_4096x4096_Bin1x1_Temp-30C_Ha.fit

So, a total of 10 different light images. The resulting output, after loading this all into BatchPreprocessing (and marking the calibration frames as masters), looked like...:

Code: [Select]
$ ls -R
Flaming Star Nebula-HAlpha (13-14)/     Flaming Star Nebula-HAlpha (9-10)/
Flaming Star Nebula-HAlpha (15-16)/     calibrated/
Flaming Star Nebula-HAlpha (3-4)/       master/
Flaming Star Nebula-HAlpha (5-6)/       master-cal/
Flaming Star Nebula-HAlpha (7-8)/       registered/

./Flaming Star Nebula-HAlpha (13-14):
Flaming Star Nebula-S001-HAlpha filter-R001-Ha.fts

./Flaming Star Nebula-HAlpha (15-16):
Flaming Star Nebula-S001-HAlpha filter-R001-Ha.fts

./Flaming Star Nebula-HAlpha (3-4):
Flaming Star Nebula-S001-HAlpha filter-R001-Ha.fts
Flaming Star Nebula-S001-HAlpha filter-R002-Ha.fts

./Flaming Star Nebula-HAlpha (5-6):
Flaming Star Nebula-S001-HAlpha filter-R001-Ha.fts
Flaming Star Nebula-S001-HAlpha filter-R002-Ha.fts

./Flaming Star Nebula-HAlpha (7-8):
Flaming Star Nebula-S001-HAlpha filter-R001-Ha.fts
Flaming Star Nebula-S001-HAlpha filter-R002-Ha.fts

./Flaming Star Nebula-HAlpha (9-10):
Flaming Star Nebula-S001-HAlpha filter-R001-Ha.fts
Flaming Star Nebula-S001-HAlpha filter-R002-Ha.fts

./calibrated:
light/

./calibrated/light:
Flaming Star Nebula-S001-HAlpha filter-R001-Ha_c.fit
Flaming Star Nebula-S001-HAlpha filter-R002-Ha_c.fit

./master:
light-FILTER_Ha-BINNING_1.fit

./master-cal:
Master_Bias_4096x4096_Bin1x1_Temp-30C.fit
Master_Dark_4096x4096_Bin1x1_Temp-30C_5400s.fit
Master_Flat_4096x4096_Bin1x1_Temp-30C_Ha.fit

./registered:
Ha/

./registered/Ha:
Flaming Star Nebula-S001-HAlpha filter-R001-Ha_c_r.fit
Flaming Star Nebula-S001-HAlpha filter-R002-Ha_c_r.fit

Note that there are only two images in each of the calibrated/light and registered/Ha folders, while there had been 10 input images (with lots of filename duplication, but unique within each folder).

Thorsten

Offline pfile

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 4729
Re: BatchPreprocessing issue
« Reply #2 on: 2013 January 28 16:54:14 »
i think i reported this long back in the BPP thread but it must have fallen thru the cracks. prior to using BackyardEOS i had a bunch of DSLR filenames called IMG_XXXX.CR2 and the counter would reset every night resulting in many images with the same names.

can ACP append a unique sequence # to the end of your images, and keep incrementing from night to night? not a solution but sort of a workaround.

rob

Offline Thorsten Lockert

  • Newcomer
  • Posts: 36
    • Personal Home Page
Re: BatchPreprocessing issue
« Reply #3 on: 2013 January 28 17:03:37 »
can ACP append a unique sequence # to the end of your images, and keep incrementing from night to night? not a solution but sort of a workaround.

It appears to only have sequence numbers within observation runs, not across them. So, either I'll need to do further preprocessing before PixInsight, or BatchPreprocessing needs to learn to put files with like names (but from separate folders) into uniquely named files (or subfolders).

In the short term I will do the former, and I might look into adding changes to BatchPreprocessing to do the latter unless someone beats me to it.

Thorsten

Offline rwodaski

  • Newcomer
  • Posts: 10
Re: BatchPreprocessing issue
« Reply #4 on: 2013 January 28 17:40:45 »
My trick is to alter the ACP filespec to include date/time; that guarantees unique filenames. There is a text file where you can tell ACP how to format filenames.