Master flat for DSLR lights taken with in-camera long exposure reduction

[Armando]

Hi,

I'm briefly describing the issue...

1. Set of several 4' light frames taken by a (one shoot colour) DSLR @800ISO, dark calibrated by the camera itself (i.e. by long exposure noise reduction)
2. Set of 1/15 s flats @100ISO
3. No BIAS
4. No DARK

Issue: radial chromatic gradient, mainly noticeable near the corners of the flat-calibrated and integrated light frame; residual chromatic gradient after ABE processing.

In-camera noise reduction (i.e. in-camera auto-dark) alters the offset. So I think that calibrating flats by dark and/or bias would be still useless.

I'm thinking about linear fitting master flat to light background computed by ABE applied to a single light exposure (I think I'll solve the light<->flat offsets mismatch and the missing dark subtraction on the flats).

So I'm thinking about the following steps:
1. debayering master flat*;
2. debayering a single light*;
3. ABE processing on debayered light to get background;
4. upscaling ABE background result (to recover full resolution)**;
5. linear fitting debayered master flat to the upscaled ABE background;
6. debayering all lights;
7. calibrating all debayered lights by debayered (and linear fitted) flat.

*ABE works on debayered images
**ABE background seems to be scaled (as with 2x2 binning).

Should it work?

Thank You and Clear Skies!
Armando Beneduce

P.s. Flats were taken by a flat panel; so I think the chromatic gradient is also related to the spectrum of the light and maybe to different offsets applied by auto-dark on each channel so that the flat is not properly "aligned" to the unprocessed/raw lights. I think linear fitting the flat to background computed by ABE could properly "realign" the flat to the light.

[Armando]

I just realized I've named DBE as ABE: maybe as a PixInsight newbie I was so impressed by it to call it as "automatic background extractor"... 
Anyway I still think that fitting linearly the master flat to the background computed by PixInsight can improve calibration in my case.
I obviously used dithering; this is the reason why I'd prefer to correct the master flat thanks to DBE in place of applying DBE background to the integrated image...

What I'm not sure is related to debayering that should follow the flat division, what is not possible in my case since I mean to use DBE background...

Another reason why I'm still doubting about my idea is the lack of replies. 

I'll greatly appreciate your opinion.

Clear Skies!
Armando Beneduce

[Luigi]

This is not a 'high traffic' forum. Replies are sometimes slow in coming; sometimes they don't come at all

A couple of observations/suggestions:

Do NOT use in-camera NR. It's a waste of imaging time.
You said you were dithering your exposures. I am trying to forgo dark frames with my last couple of imaging runs where I dithered.

Regarding your flats: were they exposed to about 50% or so of your maximum? Take one of your flats, load it into the 'Statistics' process and see what the median/mean are.

My guess is that your flats are not properly exposed. I recently had the same issue with flats that 'looked' ok but I retook them and went by the numbers from the Statistics process and they worked much better.

[Armando]

Hi Luigi,

first of all thank you for your reply.

Sorry... Probably too hurry trying to learn PixInsight: the process used to correct my master flat is ABE. 

Next time I won't use long exposure noise reduction and probably I won't take any dark (on the field): PixInsight default dark calibration makes use of scaling (it seems to be preferrable also with temperature controlled sensors).
So I'm going to make a one-time and huge set of DARK and BIAS to store a master dark. Then I'll prefer to take only LIGHT, FLAT and BIAS frames. Maybe the stored bias_corrected_master_dark will be fine (thanks to the scaling) on bias_corrected lights. The same for flat too...

But now I need to process my lights, all with in camera noise reduction.
I think flats are fine. Anyway I'm not sure I understood your question related to the flat.
What did you mean by "your maximum"? 2^14? Or were you suggesting to set exposure time to have mean value at about 50% of max value, with max value meant as the actual max value (at the center of the frame)?

I think that mismatch between the pedestals of LIGHTS (affected by in camera NR) and FLAT (caused by missing BIAS subtraction) causes the issue: it's mainly noticeable at the corners, where the offset mismatch can be more comparable to the signal making the flat division less effective at the corners...

Thanks again!
Armando

[Luigi]

I'm still learning all of this myself, but ...

When I took my flats with my camera (modded D610, so 14-bit) at ISO 400, it maxed out at around 4000 (I suspect at ISO 100 it would be more near 16000, ~2^14). I exposed the sensor longer and longer until those numbers did not increase. Then I adjusted my exposure to get around 2000 median/mean, or 50%. I used the Statistics process to do this.

When I did this in a prior attempt I watched the histogram in my capture program (BackyardNIKON) but I learned that it is showing the histogram of the embedded JPEG, stretched. Those came out with a median/mean of about 300, which is underexposed even though the histogram in the capture program showed the peaks at 1/3 to 1/2 way to the right.

[Armando]

I just found my flats are underexposed (even if the on-camera histogram seems good: it's on the stretched image).  I need to tune exposure time by taking into account my flat panel light is stronger on the blue... Maybe just a correction of one stop...

Anyway I see that fitting the flat to the ABE background makes flat calibration working as expected.
So I've neither black clip nor uniform light on the flat at the corners...
In any case it's really a good advice to take care of a correct flat exposure time to get less noise after flat calibration and to make flat less BIAS dependent, thanks!

I played with Statistics process and I noticed that max value (with saturated sensor) is 16383 both at 800ISO and 100ISO if I make Statistics using 16 bit mode.
You selected 14 bit mode (by taking into account the ADC bit depth), didn't you?
I think we get 16bits (by dcraw) with the 2 most significative bits at 0 because of 14 bit raw format: we should select 14 bit mode only if 2 ('0') LSBs were added by dcraw...

Clear Skies!
Armando

[Armando]

I gave another look at the on-camera histogram.
Playing with exposure time I found that the x scale is logarithmic.
Both on D5300 and D600 the range is 3^4 (I see about 5/3 stops are required to shift the histogram to the left or to the right by 1 quarter of the x scale).
So the trick could be keeping the on-camera histogram "inside" the right quarter...

Clear Skies!
Armando

[jcinpv]

First, it is not recommended to let the camera do ANY processing on the images. If you taken appropriate Darks, Bias, and Flats, then PI can handle processing the images.
Second, Flats can be taken using an overcast sky during the day as long as the overcast is fairly even and you have defocused the image.
For a modified/cooled Canon DSLR, you may not need to bother with Bias or Darks for exposures under 30s. Flats are essential.
Third, the histogram peaks for Flats should be in the range of 1/3 to 1/2.

The radial chromatic gradient you are seeing may be the result of a bright object in the center, but more than likely due to vignetting. If it is vignetting, the Flats should take care of that. I have taken images of bright stars and find that any overexposing of the star causes blooming and radial chromatic gradients that is not removed with Flats. Careful post processing can reduce that effect. By adjusting the black level and/or contrast to darken the image you can probably eliminate that effect - at the cost of losing detail. ABE seems to enhance that effect while DBE seems to reduce it. Warren has suggested to follow DBE with ABE.

I'm surprised that I know this much.

Here is probably a very relevant post on this:
https://pixinsight.com/forum/index.php?topic=10254.msg66664#msg66664

Hi,

I'm briefly describing the issue...

1. Set of several 4' light frames taken by a (one shoot colour) DSLR @800ISO, dark calibrated by the camera itself (i.e. by long exposure noise reduction)
2. Set of 1/15 s flats @100ISO
3. No BIAS
4. No DARK

Issue: radial chromatic gradient, mainly noticeable near the corners of the flat-calibrated and integrated light frame; residual chromatic gradient after ABE processing.

In-camera noise reduction (i.e. in-camera auto-dark) alters the offset. So I think that calibrating flats by dark and/or bias would be still useless.

I'm thinking about linear fitting master flat to light background computed by ABE applied to a single light exposure (I think I'll solve the light<->flat offsets mismatch and the missing dark subtraction on the flats).

So I'm thinking about the following steps:
1. debayering master flat*;
2. debayering a single light*;
3. ABE processing on debayered light to get background;
4. upscaling ABE background result (to recover full resolution)**;
5. linear fitting debayered master flat to the upscaled ABE background;
6. debayering all lights;
7. calibrating all debayered lights by debayered (and linear fitted) flat.

*ABE works on debayered images
**ABE background seems to be scaled (as with 2x2 binning).

Should it work?

Thank You and Clear Skies!
Armando Beneduce

P.s. Flats were taken by a flat panel; so I think the chromatic gradient is also related to the spectrum of the light and maybe to different offsets applied by auto-dark on each channel so that the flat is not properly "aligned" to the unprocessed/raw lights. I think linear fitting the flat to background computed by ABE could properly "realign" the flat to the light.

[jcinpv]

Luigi, were you using the 16-bit option of Statistics? And is your maximum something like 5,000?
And does a stretched image of a flat look something like this?


I'm just asking so I can get a comparison.

I'm still learning all of this myself, but ...

When I took my flats with my camera (modded D610, so 14-bit) at ISO 400, it maxed out at around 4000 (I suspect at ISO 100 it would be more near 16000, ~2^14). I exposed the sensor longer and longer until those numbers did not increase. Then I adjusted my exposure to get around 2000 median/mean, or 50%. I used the Statistics process to do this.

When I did this in a prior attempt I watched the histogram in my capture program (BackyardNIKON) but I learned that it is showing the histogram of the embedded JPEG, stretched. Those came out with a median/mean of about 300, which is underexposed even though the histogram in the capture program showed the peaks at 1/3 to 1/2 way to the right.

[andreasmax]

to turn of the in-camera noise reduction completely at your nikon you need to turn

1) of all corresponding settings

and

2) you need to install a firmware hack. https://nikonhacker.com

hope that helps (if not already have installed it...)

cheers

max

[Luigi]

Luigi, were you using the 16-bit option of Statistics? And is your maximum something like 5,000?
And does a stretched image of a flat look something like this?


I'm just asking so I can get a comparison.

I set Statistics to 14-bit, so in my case the max was about 4000 at ISO 400.
My stretched master flat looks a bit brighter, but, as I said, it's all about the numbers. Plus, recall I was doing 50%. If you're at 5000 on a 16-bit scale you're more like 7% ...