BatchPreprocessing and Exposure Times

[wadeh237]

I've been using PixInsight for a while, but doing calibration through integration steps in another piece of software.

I'm finally migrating my calibration through integration steps to PixInsight.  I understand how to do it manually with ImageCalibration, StarAlignment and ImageIntegration, but I would like to understand a bit more about how BatchPreprocessing works.

My main question is regarding grouping light frames with different exposure times.  I have an ASI1600MM-cool camera.  This camera is known to have amp glows that are non-linear with respect to time.  As a result, I cannot use dark frame optimization.

In the scenario I am working with now, I have a large set of light frames taken with red, green and blue filters.  All of the images were taken at the same camera temperature.  A portion of the light frames were taken with 30 second exposures, and the remaining frames were taken with 60 second exposures.  All are binned 1x1 and taken with the same gain and offset.

I have a 1x1 binned master bias at the same camera temperature, gain and offset.  I also have two master darks at the same temperature, gain and offset.  One of them is made from 30 second darks and the other was made from 60 second darks.  Finally, I have matching flat masters.  The dark masters have not been bias calibrated, and the flat masters have not been bias or dark calibrated.

I've added all of the calibration frames in BPP and they are correctly grouped in their respective tabs (ie. binning, exposure times and filter groupings are correct, and the master frames have a star icon).

I have add all of the light frames.  They are correctly grouped under binning and filter, but are not separated by exposure time in the BPP user interface.  The exposure times are reflected in the filenames, and I have verified that the exposure time is present in the FITS headers.  For the 30 second exposures, the FITS keyword EXPOSURE is set to 30, and in the 60 second exposures, the FITS keyword EXPOSURE is set to 60.

Finally, in the BPP interface, the dark frame exposure tolerance is set to 10 seconds, and I have selected "User master bias", "User master dark" and "Use master flat".

I believe that if I run BPP as described above, it should calibrate the images correctly.  My concern is that when I look at the resulting log file, it appears that it is only using the 60 second dark master.  Here are the relevant lines from the beginning of the log file:

[2018-11-29 21:50:08] * Searching for a master dark frame with exposure time = 60s -- best match is 60s
[2018-11-29 21:50:08] * Master bias: D:/Astro/Calibration/Bias Master gain_76 offset_40 bin1x1.FIT
[2018-11-29 21:50:08] * Master dark: D:/Astro/Calibration/Dark Master gain_76 offset_40 bin1x1 -10C 60 seconds.FIT
[2018-11-29 21:50:08] * Master flat: D:/Astro/Flats/Flat Master - Flat_0.008sec_gain_76_offset_40_1x1_Blue.FIT

I've searched the log file and can find no reference to the 30 second dark master.  I've looked that the statistics for the calibrated images and they look reasonable - but the difference in dark current between 30 and 60 seconds is small enough that it would be really difficult to tell which dark frame was used.

For this particular project, it probably doesn't make a huge difference.  But I want to understand what's going on, so that when I do a project with a combination of narrow band and broad band filters (which would have significantly different exposure times), I can set it up correctly.

Thanks for any suggestions,
-Wade

[wadeh237]

To add to this, I just created some test files.

I started by copying some raw FITS files from the above set (to ensure that all the FITS headers match).  I then used PixelMath to set all the master bias pixels to .001, all the 30 second dark master pixels to .051, and all the 60 second dark master pixels to .251.

I then did the same thing to one of the 30 second light frames to set its pixels to .1015 and one of the 60 second light frames to set its pixels to .5025.

Finally, I ran BPP with both the 30 and 60 second masters listed.  The resulting calibrated files made it clear that it used the 60 second dark for both light frames.  The calibrated 30 second frame was all black (every pixel zero).  The calibrated 60 second frame was about .26.

I say "about" for the 60 second calibrated pixels because I was trying to simulate light vignetting with a simulated flat frame.  I did something wrong with that, though, so my fake vignetting survived the calibration.  I wanted to prove for sure whether the master darks and flats are expected to be calibrated.

There are lots of tutorials and cookbook style instructions out there for BPP, but I was surprised that I could not find any straightforward documentation that lists the script's assumptions about the calibration masters.

I took a look at the script itself, hoping that it was well commented.  But a cursory look at the code didn't reveal the answers.  At some point, I may actually dig into the code to see what it's doing, but I don't know the PixInsight API well enough to do that quickly.

So it seems like I will need to run BPP separately for exposures of different lengths in my scenario (I'm wondering if it would be different if I optimized the dark calibration).  I'll have to keep playing with the flat calibration to understand it better.

If anyone knows for sure how this works, I would appreciate hearing it.

Thanks,
-Wade

[wadeh237]

If anyone reads this thread in the future and is looking for the answer, I did find it stated concisely here:

https://pixinsight.com/forum/index.php?topic=10254.msg66664#msg66664

To summarize, if you want to use BPP with pre-made master calibration files, it assumes:

• The master bias is a simple integration
  The master dark is also a simple integration (not bias calibrated)
  The master flats are bias and dark calibrated, and integrated

I also reached the same conclusion through a fair amount of experimentation (before coming across the above link while looking at something else...)

-Wade