Finally resolved problem with flat frames in BatchPreProcessing

[avarakin]

I have seen some users here having problems with flat frames not working as expected during BatchPreProcessing.
I also had several images where flat frames were not working as expected.
The issue I saw was that sometimes master flat frames were created with very dark corners, which was causing calibrated light frames to have very bright corners.
After some investigation, I was able to trace this issue to the way how flat frames are integrated by the script - they are integrated using AdditiveWithScaling normalization, which apparently shifts the levels of the resulting flat frame.

I applied a fix on my installation of PI to resolve this issue and I see much better results with the images I previously had problems with.

Here is how to apply the fix:

1. Locate this file on Linux (on Windows it should be somewhere else) and edit it as root:
/opt/PixInsight/src/scripts/BatchPreprocessing/BatchPreprocessing-engine.js

2. Find this function:
StackEngine.prototype.doIntegrate = function( frameGroup )

3. Locate this fragment in this function:
case ImageType.FLAT:

4. Make the following change on 2 following lines:
      II.normalization          = ImageIntegration.prototype.NoNormalization;
      II.rejectionNormalization = ImageIntegration.prototype.NoRejectionNormalization;

Disclaimers:
1. This fix worked for me but may not work for you, so use it at your own risk
2. I am not fully understanding the reason why the original implementation is not working properly
3. I am not sure if this kind of post is allowed in this area, so please feel free to move it to appropriate section of the forum
4. I noticed that saved instances of the script can no longer be opened due to checksum error, which is not a big deal for me

Alex

[pfile]

this is odd - i thought that proper normalization for flats was multiplicative, not additive. seems to be a bug in the script!

rob

[Juan Conejero]

After some investigation, I was able to trace this issue to the way how flat frames are integrated by the script - they are integrated using AdditiveWithScaling normalization, which apparently shifts the levels of the resulting flat frame.

That's not true. See lines 1362-1379 of BatchPreprocessing-engine.js:

   switch ( imageType )
   {
   case ImageType.LIGHT:
      II.normalization          = ImageIntegration.prototype.AdditiveWithScaling;
      II.rejectionNormalization = ImageIntegration.prototype.Scale;
      II.weightScale            = ImageIntegration.prototype.WeightScale_IKSS;
      break;
   case ImageType.FLAT:
      II.normalization          = ImageIntegration.prototype.Multiplicative;
      II.rejectionNormalization = ImageIntegration.prototype.EqualizeFluxes;
      II.weightScale            = ImageIntegration.prototype.WeightScale_IKSS;
      break;
   default:
      II.normalization          = ImageIntegration.prototype.NoNormalization;
      II.rejectionNormalization = ImageIntegration.prototype.NoRejectionNormalization;
      II.weightScale            = ImageIntegration.prototype.WeightScale_MAD;
      break;
   }


These lines are part of the StackEngine.doIntegrate() function. Flats are integrated using multiplicative output normalization and flux equalization for rejection. Your modification disables normalization completely, which invalidates pixel rejection and tends to produce a suboptimal master flat frame in SNR terms.

The issue I saw was that sometimes master flat frames were created with very dark corners, which was causing calibrated light frames to have very bright corners.

These problems are caused by bad image acquisition practices and/or file format issues (typically, importing alien data = raw frames stored by other applications as FITS files in floating point formats or large integer formats using undocumented ranges). Modifying the BatchPreprocessing script this way is not a valid solution to these problems (although it may seem to 'work' in some specific cases).

This modification of the BatchPreprocessing script is not recommended.

[hcostache]

These problems are caused by bad image acquisition practices and/or file format issues (typically, importing alien data = raw frames stored by other applications as FITS files in floating point formats or large integer formats using undocumented ranges).

Is there a way to check a FITS file for this kind of format/range problems?

Regards,
Horia
 

[pfile]

yes, if you go into the format explorer and double click "FITS" you will get a dialog box where you can tell PI what to do when it encounters a FITS file with out-of-range floating point values. if you set this to "Always Ask" you'll get a dialog box whenever a "bad" fits file is opened. note though that this can be a pain if you open a bunch of fits files in Blink or in ImageCalibration, etc. you'll repeatedly get dialog boxes when out-of-range files are encountered. generally once you get your acquisition and calibration flow going properly this is no longer a problem.

rob