ImageCalibration Dark Optimization: How does this minimize noise?

[georg.viehoever]

The bubble help for MasterDark/Optimimze checkbox in ImageCalibration states that the MasterDark is scaled to minimize noise. In my opinion, if I have two noisy images, and I add/subtract them from each other, there is no way to reduce the noise of the images. Noise from both images will always be added. I cannot remove noise by adding/subtracting two noisy sources.

(Imagine two noisy tapes with the same music. What I can do is remove much of the music by subtracting the sound from both tapes, but I can nevertheless never remove the noise this way. If you can, Juan, you should file a patent  )

Am I getting something wrong here?
Georg

[ruediger]

It should read "fixed pattern noise" instead of noise. Think about the thermal noise that will build up in a dark frame over time. Due to unevenly heat distribution inside the camera, effects due to chip layout etc. a large scale dark noise pattern will form. The noise distribution of each pixel itself is of gaussian type and eliminated during master dark generation. Because the strength of the remaining dark noise pattern is a function of time, temperature, ISO and we probably never have a perfect match between the light frames and the master dark, "dark optimization" is needed. This is a computed factor that scales the dark noise pattern from the master dark against the dark noise pattern inside a single light frame. Without being able to verify this right now, because I have no access to sample files: I think the dark optimization is as simple as finding a scaling factor that after subtracting the corrected master dark leaves the light frame's mean value intact. (maybe this statement provokes Juan to kick in within the next 5 minutes to correct me and explain the algorithm in more detail  ).

Rüdiger

Edit: to be a little more serious: The PI homepage describes the dark scaling optimization as follows: "Our image calibration tool includes a powerful dark optimization/scaling algorithm based on iterative multiscale noise evaluation". Could this optimization also carried out if the master dark is perfect, i.e. noise free? And how does the dark scaling optimization compares against algorithms that not only have access to a master dark but to a dark library with lots of different single dark files to choose from, e.g. this one described here: http://www.kyb.tuebingen.mpg.de/fileadmin/user_upload/files/publications/ICCP09-GomezRodriguez_5491[0].pdf. If i understand the paper, their algorithm outperforms a simple master dark subtraction, even if the master dark consists of 25 perfectly matching single dark frames.

[Cleon_Wells]

And one more question on the MasterDark Optimization Scaling Factor; is there a point where the Scaling Factor is to large (>2.0) or to small (<0.5),  that can create  scaling artifacts?
Cleon

[ruediger]

I setup an interesting test project with only artificial dark and light frames:

- a noise free master dark, generated with "0.5 * ( 1 - Y() )" => vertical gradient
- a single dark file being a copy from the master dark with added gaussian noise
- a single light frame generated with "0.3 * X()" => horizontal gradient
- added some poisson noise to the light frame and then added the single dark file => "light_with_noise"

Now if i use ImageCalibration to calibrate the "light_with_noise" with the noise free master dark, the dark optimization factor is "1.697", which is too high and leaves a still visible downward gradient in the corrected light file. The upper left of the calibrated light is a strange totally noise free area filled with value "0.1325".

Simple subtracting the dark master (i.e. scaling factor 1.0) from the "light_with_noise" leads to a better result (upper right).

Hopefully my test setup is correct. I don't want to find strange scenarios that lead to errors or only getting the developers on their nerves, I'd rather to fully understand how things work and find ways to further improve quality.

Rüdiger

[pfile]

And one more question on the MasterDark Optimization Scaling Factor; is there a point where the Scaling Factor is to large (>2.0) or to small (<0.5),  that can create  scaling artifacts?
Cleon

i don't know if it's strictly associated with large/small scaling factors, but yeah on my DSLR images i have seen hot pixels being oversubtracted by the scaling algorithm at times. Vicente says this is normal, and they were working on some new algorithms to address this.

[Cleon_Wells]

Pfile, that’s good to know that they're working on the MasterDark Optimization  Hot Pixel issue.
The test that Ruediger performed is interesting; I wonder if he could introduce some hot and dead pixels in his test files?
Cleon

[pfile]

problem is... i think i got this answer more than a year ago 

[vicent_peris]

Hi,

The dark optimization works by looking for a minimum noise value in the resulting image after subtracting the scaled dark frame. This works because the dark is a fixed pattern noise. Thus we take the best of the signal and noise properties of the dark frame:

- The correct dark signal subtraction will be in a minimum in terms of noise because the dark pattern will disappear.
- The random distribution of this pattern allow us to analyze it as it were noise.

Said this, the test made by Rüdiger is not valid at all in one aspect: our noise analysis is performed only in the first wavelet layer. Thus, a smooth "dark" gradient is not being analyzed at all. On the other hand, this test is also valid to show that a very poor dark frame won't work with this method: a low quality dark will have a large amount of read noise, thus the scaling value will be seriously affected.


Regards,
Vicent.

[ruediger]

- The random distribution of this pattern allow us to analyze it as it were noise.

Thanks for the explanation, I think now I understand why my tests not give meaningful results and are not transferable to real life data.

But I don't understand at all why it's preferable to look only at the first wavelet layer and not on the higher order layers. What mechanism inside the camera generates this pattern with random distribution that does not vanish when averaging lots of dark frames, i.e. when trying to build really good master files? To the contrary I would believe it must be better to not look at layer 1 and start with higher layers? 

Rüdiger

[astropixel]

You guys have too much time on your hands

[georg.viehoever]

You guys have too much time on your hands

No, not really. But for me it is fun to understand what's going on, and if  lucky, find a better way to do it. I guess it is a little scientists strain in me....

[Juan Conejero]

for me it is fun to understand what's going on, and if  lucky, find a better way to do it.

You have basically described what the PixInsight project is all about with this sentence

I don't have the time right now (I'm going to have something similar to a 'weekend' today, wow!), but later I'll describe our dark optimization algorithm in detail. Actually it is rather simple; you just have to open your mind to a broader definition of the word 'noise'.

[vicent_peris]

The key point us understand the dark image as noise and signal at the same time.  Juan will explain this much better than me...  I'm good having ideas but not keeping them organized.


V.

[ruediger]

The key point us understand the dark image as noise and signal at the same time. 

Today I inspected a master dark with help of utility script "ExtractWaveletLayers". Very surprising for me! As if some bugs are eating up my sensor cells! Until now I thought, master dark is for large scale structures and dithering between exposures takes care of small scale noise structures like hotpixels etc.

Rüdiger

[georg.viehoever]

Using the data in http://pixinsight.com/forum/index.php?topic=4086.msg29086#msg29086, I created masterBias (from 10 shots at 1/8000s) and masterDark (from 10 shots at 120s) with the settings in screenshot 1. I then used these to calibrate 30 second darks, using the settings in screenshot 2. This resulted in dark scaling factors around 0.140-0.159, and not in those of 0.25 as expected.

Hmm