Problems with noise in stack when using flats

[Emiel Kempen]

Hello,
Hopefully someone can give me some advise regarding my problem with getting a noisy stack when using flats during calibration.
When I use no flats during calibration, thus only bias and darks, the stack is nice smooth.

My camera is a modified Canon 40D and I shoot 50 bias, 40 dark and 40 flat frames at iso 400.

I expose my flats to mid histogram, but also tried too short and too long exposure time, but no difference, every time noisy stack.
I use the BatchPreProcessing script in PI 1.8 RC7.
My lights are exposed 180 sec at iso 400 through a 10" F/2.9 Newtonian (with ASA 2KORRR corrector / reducer), the sky was SQM-L 20.05.

On my website, one can find three images which hopefully makes things clearer:

Stack when using flats:               http://www.ekempen.com/met-flat.jpg  (1.2MB)

Stack when using NO flats:        http://www.ekempen.com/zonder-flat.jpg  (400kb)
 
The used masterflat:                  http://www.ekempen.com/flat.jpg  (400kb)

Hopefully there is an explanation for this.

Regards,
Emiel

[Carlos Milovic]

Your master flat looks very noisy. Try to expose for longer exposures (are you sure that you are in the mid hist in the linear range?).

[Andres.Pozo]

After flattening an image, the automatic STF can stretch it much more. That's why it seems to be noisier: although the noise is similar, after stretching it seems to be higher.

[pfile]

yes, i was just about to say what andres said. also, mid-histogram on back of camera is nowhere near mid-well depth. you need to make sure you are looking at the histogram in a tool that can represent the data linearly.

[Carlos Milovic]

As a rule of thumb, I take my flats in AV setting, using +2 stops (i.e. 4 times more exposure time)

[ruediger]

Just to make sure if there really is a noise difference between the two images (with/without flats):
- crop the center part of both images
- linear fit the images to each other
- run the NoiseEvaluation script to measure the noise

I doubt there will be much difference, because the two nonlinear jpgs were probably both stretched to a same median value, but the flattened image has a more uniform background, therefore it is stretched more (as Andres said already).

Rüdiger

[Emiel Kempen]

Thanks for the replies!
Carlos, I will make new flats this evening and use your rule of thumb.
Rüdiger, I did what you asked, here is the outcome of 2 NoiseEvaluation script runs, one on the with and one on the without LinearFitted images:
Processing script file: C:/Program Files/PixInsight-rc7/src/scripts/NoiseEvaluation.js

without
Calculating noise standard deviation...

* Channel #0
?R = 9.921e-006, N = 346811 (34.68%), J = 4

* Channel #1
?G = 1.746e-005, N = 394084 (39.41%), J = 4

* Channel #2
?B = 6.446e-005, N = 466694 (46.67%), J = 4


run --execute-mode=auto "C:/Program Files/PixInsight-rc7/src/scripts/NoiseEvaluation.js"

Processing script file: C:/Program Files/PixInsight-rc7/src/scripts/NoiseEvaluation.js

with
Calculating noise standard deviation...

* Channel #0
?R = 3.234e-005, N = 576147 (57.61%), J = 4

* Channel #1
?G = 3.313e-005, N = 464145 (46.41%), J = 4

* Channel #2
?B = 5.077e-005, N = 101318 (10.13%), J = 4

I attached the two resulting images.

Regards,
Emiel.

[ruediger]

Based on the noise evaluation results I think the problem lies in the (artificial?) light source you use for the flats. Red and Green are indeed noisier with flats, but there is no problem with the blue channel. How exactly did you take the flats and checked for proper exposure?

The next step would be to debayer the MasterFlat with "SuperPixel" mode and check the resulting RGB image with the ImageStatistics tool. I guess that the statistics (median/mean/standard deviation) for the red and green channel are very different compared to the blue channel statistics.

Rüdiger

[Emiel Kempen]

Hello Rüdiger,
I am using a flat-panel from Gerd Neumann (http://www.gerdneumann.net), which I bought 3 or 4 years ago. I put 4 sheets of white plastic on top of my my dewshield to dim the light and than the flat-panel on top of that.
I take a test exposure in Canon RAW format and a Digital Camera Raw File Conversion in ImagesPlus is done. The resulting fits file is opened in ImagesPlus and the Histogram is examined. After I have found an exposure time to give flats with the histogram peak in the middle, I take 30 - 40 flats. The flats I used have an exposure time of 3.2 sec at iso 400. These (Canon RAW) flats are then used in the BatchPreprocessing Script.

I have attached the statistics of the masterflat

Regards,
Emiel.

[ruediger]

I have attached the statistics of the masterflat

The flats are well exposed, but the standard deviation between red/green (the noisy channels) and blue are very different.
Can you repeat the statistics process for a preview of the center region, where the illumination is nearly uniform with no light fall-off?
It looks like there is much more remaining noise (nearly 10 times) in the red/green channel compared to the blue channel.

For comparison (I also use the Gerd Neumanns panel): My master flats made from about 20 single exposures have a stddev between 10 to 20 ADU in the center region.

So far I can't tell where the differences come from. Even the batch preprocessing script (which I don't use) could be a potential source for this weird error.

Anyone else has an idea?


Rüdiger

[pfile]

is mean of 0.08 really well exposed? i'd expect to see 0.125 or more for a centered histogram.

[ruediger]

is mean of 0.08 really well exposed? i'd expect to see 0.125 or more for a centered histogram.

=5242 ADU, for a DSLR with its reduced dynamic range at 400 ISO absolutely OK (compare e.g. http://www.astrosurf.com/buil/50d/test.htm).

[pfile]

full well at iso 400 for the 40d is 12,900 electrons with a gain of 0.84e-/ADU giving a full well ADU count of 15357. for approx 1/2 well depth then you want 7000-8000ADUs. the B channel is 0.01 of 65535 or only 655 ADU. that seems too low to me. the red channel at 0.08 is better.

i'm not sure where the 40d sensor departs linearity but you could probably go to more than 1/2 the well capacity.

[ruediger]

the B channel is 0.01 of 65535 or only 655 ADU. that seems too low to me. the red channel at 0.08 is better.

I see what you mean. A little more exposure is possible, but the values are not totally wrong (as when you get "fooled" by taken mid-histogramm from the camera display).
Blue is indeed under-exposed, but this is the channel that makes no problems (according to the noise evaluation).

The flatpanel I use (also from Gerd Neumann) behaves differently, exposing blue and green much more than red. For comparision, attached is a screenshot from a single flat exposure and the image statistics from the center region.

@Emiel: Can you provide similar data for a single exposure and your master flat?

Rüdiger

[Emiel Kempen]

Hello Pfile and Rüdiger,
I did some experimentation last evening.
I removed the plastic sheets from my flat panel and made new flats using white t-shirts, there is quite a difference. It looks like that the white plastic sheets are messing up the spectrum of the flatpanel?

I attached 4 images:
MasterFlat-with-plastic-sheets.jpg   MasterFlat made with 4 white plastic sheets between the flatpanel and the telescope.

SingleFlat-with-plastic-sheets.jpg    SingleFlat made with 4 white plastic sheets between the flatpanel and telescope.

MasterFlat-with-t-shirts.jpg              MasterFlat made with 5 layers of white T-shirts between the flatpanel and the telescope.

SingleFlat-with-t-shirts.jpg               SingleFlat made with 5 layers of white T-shirts between the flatpanel and the telescope.

Regards,
Emiel.