BatchPreprocessing with provision for flat darks

[M Covington]

For those of us who regularly take flat darks rather than bias frames (the two are actually equivalent with many cameras), here is my humble attempt to improve the BatchPreprocessing script.  This version gives you a checkbox, "Bias frames are flat darks," which, when checked, causes the bias frames to be used not only as bias but also for dark subtraction of the flats.

This works whether or not you are optimizing darks.  When you are optimizing darks, it is not so necessary, because the regular darks are scaled to match the flats, but at least in theory (and I think in practice) this calibration is still a little better this way.

Presented to the PixInsight community with my compliments; feedback and improvements welcome!

This script is internally named "BatchPreprocessingFD" to avoid conflict with the original.  But I would like to suggest picking up this change into the distribution version, if the developers agree that it is useful.

UPDATE - Do not download this file - a corrected version was posted later.  Scroll down to that.

[M Covington]

Revised:

• New feature: cosmetic correction (with simple default settings) can be performed without creating an instance icon.
  
• Improvements to the wording of a couple of pre-existing error messages.

I will be submitting this to the GIT repository.

UPDATE - Do not download this file - a corrected version was posted later.  Scroll down to that.

[paulsartory]

Hi Mark (it is Mark?)

I use an ASI1600mm that has particular linearity issues relating to calibration.

So a quick question....just to make sure I understand your BPP modification and the application of FDs.

I need to calibrate my flats with bias frame only before calibrating my lights.......I think this is what you FDs do.

I do NOT want to bias calibrate my darks or lights. These are matched in temp, duration and gain.

btw when I try to start it from a workspace icon I get the following error:
*** Error [000]: C:/Users/pauls/Downloads/BatchPreprocessingFD(1)/BatchPreprocessingFD-engine.js, line 1906: Error: Parsing integer expression: conversion error:
undefined

Cheers
Paul

[M Covington]

When you got that error message, was "Bias frames are dark flats" checked or unchecked?

As to your other question, PixInsight "optimizes" (scales) dark frames (if you check Optimize) and needs bias frames for that.

Can you tell me what exposure time you used for each set of frames?  Apart from the error message (which I haven't gotten), I'm wanting to work out how this fits your workflow.

Clear skies,
Michael

[M Covington]

I think your crash is due to corruption in the Pixinsight.ini file, which is stored in %Appdata%\Pleiades (usually).
Would you mind e-mailing me (at mc@uga.edu) your PixInsight.ini file?

[paulsartory]

Hi Michael,

Thanks for your rapid response.

Typical workflow is:

For lights and darks

CCD Gain set at 139 (equivalent to unity gain) offset 21

NB light frames 120secs @-10c

Master dark created using image integration module from uncalibrated dark frames (no bias calibration)

For flats

CCD Gain set at 300 (equivalent to x6.5) offset 50. I need this gain using a flipflat on a f8 refractor to to reach 20,000adu5scs for NB filters in less then 5secs

NB Flats taken -10c (cooling probably not necessary)

Create master bias from bias frames taken at same gain and temperature as flat

Bias calibrate flats using image calibration module

Create master flat using image integration module

Calibration

At this point I calibrate the light frames manually using the image calibration module using the appropriate master dark and flat (but no bias) and not checking the "optimize" and "calibrate" boxes associated with each master.

BatchPreprocessingFD module settings

I would like to use the BPP if I can for speed and ease of convenience (automatic directory creation etc)

I populate the tabs as follows...Lights, matching master dark and appropriate filter master flat.

Check "Bias frames are dark flats", "Use master dark", "Use master flat".
Uncheck "optimize dark frames"

I am hoping that this will achieve what I do manually. however I do notice some over correction of amp glow in the integrated stretch result compared with my manual process.

With regard to the desktop process icon I have tried both checked and unchecked with same result.

I have sent the INI file to the email address you provided

Paul
 

[M Covington]

Did you put any frames in the bias frames tab?  If so, what?

[paulsartory]

No, I don't put any frames in the bias tab.

[M Covington]

Ah.  "Bias frames are flat darks" means you have put flat darks in the bias tab.  I may need to add a pop-up message.

[paulsartory]

Ok...that makes sense...especially if I read the pop up more carefully

A question though...is the master bias subtracted from the master dark (I create the master dark manually outside of the BPP script)?

Paul

[M Covington]

Ok...that makes sense...especially if I read the pop up more carefully

A question though...is the master bias subtracted from the master dark (I create the master dark manually outside of the BPP script)?

Paul

I just checked, and as far as I can tell, the master dark is not bias-subtracted.  This information is in the log that is written during processing.

[paulsartory]

That's good to know... I need to do some test runs......but first a couple more clarifications....

Using the script and a master bias and uncalibrated flats with  "Bias as flat dark" checked ....... I can generate my (bias subtracted) master flats?

If I use these master flats and master darks and NO master bias and uncheck "Bias as flat dark" and "Optimize dark frames" can I calibrate my lights without bias subtracting my master darks?

I have copied your script to the scrip directory under pixinsight and regenerated it but I still get the error when I execute a desktop instance?

If I can't resolve instance execution problem can I achieve the same solution by populating the original BPP script using the master darks and the bias subtracted flats (generated by your script) with "optimize darks" unchecked?

I very much appreciate the time you have taken in explaning your script...even if I don't end up using it

Paul

[M Covington]

I think it will work as you describe.  If you are not using bias, there is no reason to use my script rather than the original one.

If you are still getting the error message, please tell me exactly what you are doing that leads to it, and whether the original script has the same problem.

Thanks,
Michael

[paulsartory]

Thanks Michael,

I tested using the master flats generated from your script and they calibrated the lights perfectly.

Even if I do not use your script I now better understand how it and the original works.  It would be nice (hint..hint) if there were more options to apply different bias and dark's for different gains reported in the fits header and to chose how calibration is applied to flats, darks and lights...especially with the idiosyncrasies of CMOS chips which are becoming much more prevalent.

Anyway back to the error issue.....I simply drag an instance icon (triangle) to the desktop, close the script, double click on the process icon and then click on the global (circle) icon on the script console that pops up....and up comes the BPP window.....no problems with original script.

Cheers
Paul

[M Covington]

Can you give me step-by-step instructions to reproduce the error message?  Does it occur only when no frames have been put in as bias?