Flat calibration & integration question

[Duncan]

I know the issue of the calibration tool flagging "No correlation between the master dark and target frames" has come up a lot, but I have a specific question that I wasn't able to find an answer to in the archives.

Specific situation:
I'm calibrating a set of flat frames, and I get the error message on some of the frames (maybe 25% or so).
Mono sensor, I have "Ignore CFA" set.
I have both dark and bias masters. The dark master is a relatively short exposure, close to exposure time of the flats.

Question:
Should I exclude the calibrated frames that show the warning message from the subsequent integration, or include everything?

[Niall Saunders]

Hi Duncan,

Perhaps you might initially consider breaking your process down into smaller chunks.

First, if your exposure times are very short (as you say, typical for Flat Frames) then you might simply consider excluding Bias Frames altogether.

Can you integrate 'just' the Flat Frames - without calibrating them in any way?

Check the FITS header, or properties, of the Master Dark - is the image size identical (in both x- and y-coords) to each and every raw Flat Frame? Perhaps easier - is it identical to those Flats that are throwing the error?

If you group the Flat Frames that throw an error, and then try to Calibrate those, as a group, do you then get a 100% failure rate? Similarly, do you get 100% success if you calibrate the other (good) 75% of images?

Try some of these, and then let us know how you got on.

Hope this helps.

[Duncan]

Niall,
many thanks for the reply. I tried out a few different combinations as you suggest, and the result is completely consistent, that is, running calibration just on the failing/not failing subsets doesn't make any difference.

All of the frames are properly conforming, and, just to be clear, I can integrate the resulting calibrated frames without any issues, it's just that I have a warning on some number of the flats that the dark correlation failed.

I have run calibration on my full set of flats at this point, and it turns out that only 1 or zero of each of the sets of R, G & B flats (which are binned 2x2) generate this warning. For those I just excluded the failing frames from the subsequent integration.

Curiously, I also redid calibration on the luminance flats with a longer exposure dark, and didn't get the warnings. My calibration library hasdarks at 2s, 300s and 600s for each binning. I was using the 2s dark master for the flats (the actual flat exposure) when I got the warnings. Re-running with the 300s dark master, I don't get the warnings.

I'm wondering if the problem is that the 2s dark master has noise that is dominated by readout noise, and this is somehow messing with the correlation?

[Geoff]

With short flat frame exposures I would calibrate the flats with bias frames only and ignore the darks.
Geoff

[Niall Saunders]

Good point Geoff,

However. having now had three hour's sleep, I can visualise things more clearly . . . in fact, with 2s Flats, you could virtually get away with no calibration whatsoever.

If you had acquired enough flats (and, at 2s exposure times, there is little reason not to do so) then simply Integrating these would effectively smooth out the 'background' readout noise.

It doesn't really matter if you now have a non-zero 'noise floor' - in fact the creation of a Master Flat should actually avoid any 'zero' (or even 'very close to zero') values altogether. Remember the application of Flats to an Image is a 'division' process, and zero-values don't fit well into that scenario.

I actually prefer to establish the MAX value of all pixels in a Master Flat, and then to ADD a constant value to the Master Flat such that the 'brightest' pixel is then always at a value of '1' (remember, irrespective of any actual ADU values. PI scales each and every image into the REAL domain, between 0 and 1 (inclusive).

Notice that the process I have described is purely 'additive' so, in theory, all pixel values should shift - by the same amount - 'up' and 'away' from zero.

Now, when this 'ScaledMasterFlat' is used to calibrate a raw image, the brightest areas of the flat will have little (or no) effect on the raw data, because you are 'dividing by 1' (or, pretty close to 1). In other words, the original raw image - now calibrated - won't get any 'brighter'. However, for those areas in the flat data where pixel values are close to zero (your dust donuts, vignetting, etc.), the division process will 'upscale' the raw data as required, making those specific areas 'brighter'.

You actually don't 'need'(or 'have') to do this though, the PI code neatly traps 'division by zero' errors and rescales the final output back into the 0.000... to 1.000... range automagically. However, I describe my methods because I use an OSC imager - if I apply my algorithm to the extracted CFA channels (individually, and then use these to reconstruct my ScaledMasterFlat) then the result of FlatCalibration routine produces images that have little, or no, added colour cast.

Remember, the illumination of your imager during Flat acquisition might not be using a 'perfect white' light source. On a Mono imager, with filters (including Ha, etc.) the problem doesn't really apply - beacuse you have to acquire a set of raw Flats for each and every filter that you use.

But, you have to take into consideration that a Flat image is used to correct for 'defects' in your optical chain. If you change that chain in any way, for example by changing to a different filter, then you need to acquire a new set of Flats - unless you are confident that Flats acquired as a 'batch', using all your filters (this batch probably being acquired either pre- or post-session) is 'statistically repeatable' as you perhaps change filters during an imaging session. This is why I gave up on mono imaging, preferring to use the OSC approach - we just don't get the weather-windows here in Scotland that allow for all the complexities and nuances of mono data acquisition.

Anyway, I realise that this is slightly OT, but hope that it helps somenone, somewhere, anyway (or even causes someone to think about 'my' methods and then call them into question - that's the only way to learn, "by your mistakes") 

[Duncan]

Interesting discussion.

Prompted by this, and having taken a closer look at my master dark and bias frames, I'm wondering whether it doesn't make more sense to just calibrate my flat frames with bias only. There's not that much difference, although it does look like there are identifiable hot pixels in the darks that aren't evident in the bias frames.

I'm not sure how adding a constant to the flat frames would help, though. If the computation (ignoring dark and bias subtraction) looks like:

L(x, y) / F(x, y)

adding a constant to the flat would look like:

L(x, y) / (F(x, y) + C)

Admittedly without spending a lot of time thinking about it, I don't think this would give a correct result; the additional constant is not just adding an additional constant or scaling factor, but results in additional terms dependent on both the light and flat values.

Agree that it wouldn't be good if the local mean floor value of the flat reached zero, such that with noise introduced you might get some zero or negative values, but I don't think that's a danger. The brightness level in darker zones (ie in corners or the darker part of dust donuts) is not *that* much darker than the bright values. Overall, what I see in flat frames is that the variation of the local mean brightness across the frame is not particularly large.

Anyway, like I said, interesting discussion... 

[Duncan]

Based on how I think the calibration routines are working, I've made some changes to the way that I do things. I'd really appreciate feedback on whether this sounds sensible.

[I usually have a set of 50 of each of bias and dark, and I have a set for each binning with dark exposure time the same as lights. I typically have 600s luminance lights and 300s RGB lights, with the RGB at 2x2 binning. So I have 50 1x1 bias, 50 2x2 bias, 50 600s 1x1 dark and 50 300s 2x2 dark. I have something between 20 and 40 flats for each channel, with corresponding binning settings.]


Previously, I was calibrating flat frames before integration.

With my new workflow, I am doing integration of uncalibrated bias, dark and flat frames. Combining set to average with no weighting. For each set, I do a run with no pixel exclusion and save the result, and then do an additional run with Winsorized sigma clipping. I've found limits of 4.0 sigma seems to work well for getting rid of cosmic ray artifacts, but I look at the STF'ed results comparing the excluded and non-excluded versions with the blink tool which seems to show pretty well whether I've successfully excluded cosmic rays and not touched anything else.

Having created the complete set of masters, I then calibrate the lights, checking the "calibrate" boxes for both dark and flats. I don't check the "calibrate" box for bias, which I don't think matters in this case because I have no overscan region.

In some cases I get the warning about "no correlation" for the flat frame. I don't think this matters, if I understand the process correctly. What I think happens in this case is that the bias is subtracted from the flats, but no dark subtraction, which shouldn't matter for short flat exposures.


The results look very flat to me; I'm not able to discern any residual vignetting in the calibrated lights.

Does this all sound reasonable?