Batch Calibration with RBI Mitigation

[jeffweiss9]

  Apparently calibrating images from cameras using RBI mitigation requires a different workflow, namely avoiding the use of bias frames with your darks; i.e. not scaling darks because the RBI trap leakage noise source does not scale with temperature as does normal dark current.  I have a brand new FLI Microline ML-8300 and the whole RBI mitigation thing is new to me and I'm finding it difficult to get clear information.  I'm posting here because I'd like to continue to use the PI Batch Calibration preprocessing script with this camera but need to know how to do that or even if it is possible.
   I use a library darks always with the same temperature as my lights so I don't think that not scaling darks should be a problem. I do worry about sequencing of when you can take your darks, flats because trap leakage noise decays only slowly after the chip is exposed to a bright source. If the camera is automatically doing the RBI Preflash/Flush/Mitigation (I've seen different terms), do I need worry about that?  If the solution for calibration in PI is to use my library darks (to be recollected WITH RBI mitigation, although I took them without) and not use bias frames, the Batch Calibration script will execute with just a diagnostic warning that there are no bias frames but it goes to completion fine.  The results do look improved over using bias frames with my library darks on the test frames I took. Can someone confirm that is a correct procedure and that the Batch Calibration script is actually operating properly?  If not, what else must be done differently back in manual calibration with the ImageCalibration tool (like I used to do it before the script using also flat-darks but no bias frames) or can that alternate workflow be performed by the Batch Cailbration script (as is, or with future mods)?
Thanks for your help.
-Jeff

P.S. I'm using TSX-CAO daily build 7032 that had the last FLI upgrade to the FLI X2 driver and I believe it is supposed to be supporting RBI mitigation (I see the parameters in Camera Setup).

[mschuster]

Hi Jeff,

Maybe the thing to do is a small test using the ImageCalibration process and compare the results to what you get with the BatchPreprocessing script.

I would run a test with no flat fielding, just to simplfy the test, and concentrate on the bias+scaled dark versus dark only question.

To use ImageCalibration with no master bias, you need to make sure your darks were not bias subtracted initially when your master dark was created. So the master dark must contain bias. Then in ImageCalibration, uncheck "Master Bias", check "Master Dark" and uncheck "Master Flat". Uncheck all three "Calibrate" options. Uncheck the "Optimize" option under "Master Dark". And supply your Master Dark file name of course. This setup should work as long as your frames don't have overscan regions.

Set up this way, ImageCalibration will simply subtract your master dark from each of your lights, clipped to the range 0 to 1, as usual.

Also, I would recommend comparing master darks with no RBI preflash to master darks with RBI preflash using various preflush schemes. RBI preflash can add a lot of noise. Sometimes this added RBI preflash noise can be worse than the original RBI problem was to begin with in my opinion, especially on the dimmer targets.

Thanks,
Mike

[jeffweiss9]

Thanks, Mike-
    I just did that test comparing Batch/no bias with ImageCalibration/no bias, where the Masterflat used for both was remade with no bias subtraction so it still contains the bias. The resulting integrated images with the same ImageIntegration parameters look very similar, although the BatchCalibration image's histogram is shifted right with ~ 2.5x increased spacing from the black point.  If I adjust the black point in Histogram Transformation, the two histograms look very close.  Looking at the images themselves, however, the BatchCalibration result (with no bias frames) looks slightly better than the one from ImageCalibration/nobias since it has slightly better contrast. The ImageCalibration result does look slightly superior to the BatchCalibration run using bias, if not quite as good as BatchCalibration without bias.  So I guess I can conclude that the BatchCalibration result does look quite close, if not exact, to the ImageCalibration results with no bias used on either.  So from this one experiment, it would seem that running the BatchPreprocessing calibration script without bias frames gives results that are quite close to what you get with ImageCalibration/StarAlignment/ImageIntegration with your settings.  The BatchPreprocessing has the distinct advantage of doing all that plus CosmeticCorrection in a single script over multiple filters so that would seem to be the way to go.
-Jeff

[pfile]

are you worried about RBI that's the result of a long dark exposure itself, or RBI from long "real" exposures that precede your darks?

in other words, is the thermal signal alone enough to cause RBI in a 8300 class sensor? my understanding is that the wells have to have a lot of charge in them in order for it to leak into the substrate, but i could easily be wrong about that.

rob

[jeffweiss9]

Rob-
  That's a separate question but my speed reading of articles I found on the web by Richard Crisp and some good RBI mitigation explanations on the QSI and Moravian Instruments websites (I'm a newbie on this subject and just started reading on it today), suggest that significant trap leakage relative to dark current occurs at all signal levels but increases with bright signals, long exposures and/or low temperatures.   My impression was that the problem came from the subs themselves causing the trap leakage which then decays slowly, especially at low temperatures, leaving a ghost of the brighter stars and certain chip structures in the next sub.  You could wait the 10 or 20 minutes between subs for it to decay away or you could use the RBI preflash to flood the substrate with penetrating IR (and clear the pixel wells afterwards to clean them up) which gives equilibriated (constant) but elevated levels of trap leakage noise from what was there previously but now that noise is uniform so that it is successfully subtracts out in dark subtraction. You then fight the elevated levels by setting a very low temperature in the camera.  Or the third possibility would be NOT to do RBI mitigation and forget about it, as there are folks claiming they do that and only rarely see any sign of RBI in an image.

My impression from the extensive reading of a few hours today was that it was better to set it and do it (at least for an FLI camera, as there is also variation per camera), at least on lights and darks, but then the calibration needs to be done without bias frames because darks will not scale due to the presence (at all signal levels) of this trap leakage noise that doesn't scale with temperature as assumed by PI.  Since I only use darks from my library at the same temperature as the lights, I don't see the non-scaling as a problem, but wanted to know if the BatchPreprocessing script worked properly with no bias frames.  I think this comparison that Mike suggested says that it does.   But then my expertise is 2 or 3 hours old on this subject and I'd like to hear more confirmation that this is on the right track.
-Jeff

[pfile]

yes, i think i misunderstood your question, i didn't parse "RBI mitigation" as "RBI preflash"...

sorry about that.

[mschuster]

Hi Jeff,

I can't explain the different locations and scales of the integrations and what was rejected or not. All the process settings have an effect of course, some of them must have been different. Or maybe the masters themselves got generated with different settings.

The no bias calibrations are simple, just a dark subtraction and a normalized flat division. It is easy to double check this in PixelMath on one or two frames, and make sure your calibrated frames actually have the values they should.

Regards,
Mike

[mschuster]

Hi Jeff,

Here is some more info, I don't know if it will help. For a no bias calibration, I believe the standard formula subtracts the master dark and then divides by the master flat, with a normalization equal to the mean of the master flat. PixelMath for this is

(light - masterDark) * (mean(masterFlat) / masterFlat)

I tried this in PixelMath (with rescale off) and the result equals the output of ImageCalibration (setup with no master bias) exactly.

So this is what the no bias ImageCalibration does, and it matches what I believe is the standard no bias calibration formula.

I also tried BatchPreprocessing using my master dark and flat (and checking the appropriate options). BatchPreprocessing generated a "no bias" warning message, but the results equaled ImageCalibration and PixelMath exactly.

To make this work, master dark and master flat need to be generated properly. Just to be clear, the master dark is an integration of darks that HAVE NOT BEEN bias subtracted, with no normalization and weights equal to one. The master flat on the other hand is an integration of flats that HAVE BEEN master flat-dark subtracted, with multiplicative normalization and weights equal to one. So to do this right you need a master flat-dark also (integrated like the master dark was). These guidelines are documented in Vicent's tutorial on calibration.

For my setup, I use 2400s darks (that match my 2400s lights) and 24s flat-darks (that match my 24s panel flats), all at the same temp and binning. No biases at all.

It appears that BatchPreprocessing does not support both darks and flat-darks with no biases at the same time, to generate masters as well as calibrate and integrate darks, flats and lights all in one run, but I may be mistaken.

Thanks,
Mike

[jeffweiss9]

Great, Mike-
Yeh, I'm pretty sure you can't do all that with one BatchPreprocessing run. I'm recollecting my darks now to try this out again, as I recalled that the darks I had were made the day the camera arrived and were not taken with RBI mitigation, or the same mitigation, as used in the FLI driver in TSXP/CAO for my lights.
-Jeff

[jeffweiss9]

Mike-
  I recollected my darks with RBI mitigation on and repeated the test to find that using the prepared MasterDark and MasterFlat (made process icons for those steps), then BatchPreprocessing (with 'optimize dark frames' unchecked, that I had not done previously),
I now get identical calibrated lights with BatchPreprocessing and ImageCalibration. So in the future after icon-generating the MasterDark and MasterFlats for each filter, at least BatchPreprocessing will automatically handle all the filters and take things through to the end. That will still save a lot of the total time. 
  I have been convinced, however, that I can use a MasterBias instead of making MasterDarkFlats with little effect on accuracy given the flats are taken at 2/3 to 3/4 of well depth.  That saves a bunch if you use different exposures for flats as I do.
  Thanks for your help on this.
Clear skies,
-Jeff
 P.S. To do the whole process with an 'RBI option' in the BatchPreprocessing script that would do all this would be a major reorganization of it, I think. Probably too much to ask for.