DSLR - Bias/fixed pattern template - master dark generation

[astropixel]

Hi. I would be grateful of some clarification of my understanding of the following, from

http://theory.uchicago.edu/~ejm/pix/20d/tests/noise/noise-p4.html

"Also, for cameras with a bias offset of black (e.g. 1024 on 14-bit Canon raw files), the offset should be removed from both the image and the flat field before doing the division (this will be done automatically if we subtract a pattern read noise
template from both images since the template will also be offset and the bias will cancel out). In equations, the manipulation to be done to remove PRNU is

corrected image = L*(image - offset)/(flatfield - offset)

where if desired one can substitute the pattern read noise template for the constant offset in order to remove pattern noise at the same time."

It seems to me that the master 'bias' created from very short exposure DSLR frames is in fact a fixed pattern template with random read noise zeroed in quadrature.

Given that dark frames of short duration (120 seconds) taken at low sensor temperatures (using a temperature regulated cold finger cooling modification) are dominated by the fixed pattern component of read noise, subtraction of a fixed pattern template (DSLR bias frame) should be adequate to produce a reliable master dark with fewer dark frames than for 'hot' DSLR sensors, where thermal noise is dominant and temperature varies considerably.

I notice that a bias/fixed pattern subtracted (10 frames) master dark at 11C looks good, except that dark scaling factors during calibration with Optimization selected are way too high - Optimization must be deselected - the processed image looks OK.

I can't see a reason to take lots of dark frames with a cooled CMOS sensor, particularly where sensor temperature is conservatively within +/-1C of set point temperature at any time, and conceivably, much better than that - say +/-0.5 maximum. The heat extraction capabilities of a cold finger mod are quite aggressive, and provides stable and consistent temperature regulation.

Is there a good reason to take more than 10 darks to achieve a reliable sampling of the thermal noise when using a regulated cooling system for a DSLR, if the fixed pattern noise can be effectively removed and thermal noise is consistent?

The difference is 30minutes of acquiring darks as opposed to several hours - if there is a better way, don't keep it a secret.

Image details...

11C - CMOS set point temperature
 
Top center - bias/fixed pattern calibrated (subtracted) dark frame

Bottom left - raw dark frame

Bottom right - raw bias/fixed pattern frame

[georg.viehoever]

If the effect of cooling is indeed so strong, then having real darks is redundant. The darks are there to compensate for thermal effects- and if there is no thermal effect, there is no need for darks. I understand that some of our CCD colleagues (Harry?) are indeed not using darks, and use the bias in place of darks in the calibration script.
Georg

[astropixel]

Thanks Goerg. No darks would be possible at very low temperatures, but the cooling is limited by dew point. The main advantage of cooling is that the CMOS temperature is more regulated. I have tried no darks with image sets at 3C and it works OK, but there is still dark current. Usually the cooling is set around 8 or 11C, sometimes as high as 14C.

The aim is to use as few darks as possible - just tried the preprocessing script with only 10 darks and it worked very well.

[vicent_peris]

Hi,

At Calar Alto I work with cameras cooled to -100 to -120 Celsius for visible cameras and about -200 Celsius for infrared ones, so I don't use master darks.

In your case, I would suggest to make a test. I would acquire a lot of dark and bias frames. Perhaps 100 - 200 frames for each one. Then I would make a master bias, and then a master dark without subtracting the master bias.

With these masters you can compare the dark signal to the bias one, and the dark signal to the read noise. Think that the dark signal is not only present in the hot pixels, but in all the sensor. At the end, the contribution in terms of noise of these hot pixels is very low compared to the dark signal of the whole sensor.


Regards,
Vicent.

[pfile]

-120C ! who designs these cameras? how do the designers get characterization of the silicon at that temperature?

[vicent_peris]

-120C ! who designs these cameras? how do the designers get characterization of the silicon at that temperature?

These are professional cameras. Usually they work with liquid nitrogen and are always custom made.

Characterization of the sensors is always convenient, specially because quantum efficiency changes when you go colder.


V.

[pfile]

well specifically what i mean is that if you look at TSMC's library for a given process node, they characterize the silicon between -20C and 125C. designing circuits that work properly at -120C is a challenge when you can't know how fast or slow they will run... there are tricks you can play to make sure your circuit will function properly but being so far outside the characterization envelope is a little scary.

[vicent_peris]

Well, this is a very specialized field and there are true experts designing this type of cameras... Unfortunately this is far from my knowledge. 


V.

[astropixel]

@ Vicent "With these masters you can compare the dark signal to the bias one, and the dark signal to the read noise. Think that the dark signal is not only present in the hot pixels, but in all the sensor. At the end, the contribution in terms of noise of
these hot pixels is very low compared to the dark signal of the whole sensor."

I agree, it is the whole sensor that I am thinking of. Hot pixels dissappear with dithering in any case. I will try your experiment over the weeknights. It will take many hours to acquire a hundred or so darks.

Regulating CMOS temperature will cause thermal noise to be more consistent frame-to-frame. As a result, light frames should have more predictable noise characteristics. In which case, it should be possible to use fewer dark frames for master dark generation.

[Cleon_Wells]

I notice that a bias/fixed pattern subtracted (10 frames) master dark at 11C looks good, except that dark scaling factors during calibration with Optimization selected are way too high - Optimization must be deselected - the processed image looks OK.

How high were your Dark Scaling Factors with Optimization checked ?
Cleon
PS nice to see you got your camera mod done

[astropixel]

Hi Cleon.Yes, at last...! reliable cooling.

Scaling ~159. The results were terrible with optimization. All good otherwise. However, the best result was with the Preprocessing script, which created new master frames from the raw files and seems to have worked with optimisation

Different calibration parameters, perhaps?

Rowland.

[Cleon_Wells]

Rowland, the high Scaling Factor is reminiscent of my thrash with integrating may Cal Darks using WS pixel rejection my Mac. I could take these same Cal Darks and integrate them on my Windows computer, using  WS pixel rejection and get a working MasterDark with low Scaling Factors.

http://pixinsight.com/forum/index.php?topic=3663.0

I would like to try the new BatchPreProcessing script with my Darks  but it won’t run, here’s a screen shot of my new problem.

Cleon

[georg.viehoever]

Cleon,
Your processing console should show something like this after startup:
...
C:/Program Files/PixInsight/bin/TIFF-pxm.dll
C:/Program Files/PixInsight/bin/Flux-pxm.dll
C:/Program Files/PixInsight/bin/CosmeticCorrection-pxm.dll
* Scripting resources updated.
42 of 42 module(s) installed.

Processing script file: C:/Program Files/PixInsight/bin/startup.scp

PixInsight Core 01.07.06.0793 Starbuck (x86_64)
Copyright (C) 2003-2012 Pleiades Astrophoto
----------------------------------------------------------------------
Welcome to PixInsight. Started 2012-Apr-30 18:14:54 J2456048.26035 UTC

* Parallel processing enabled: Using 2 logical processors.
* Thread CPU affinity control enabled.
* PSM AutoSave enabled. Auto-save period: 30 seconds.
...
If it does not, your system missed some updates. The calibrate script is searching for the CosmeticCorrection module, not the script. If choosing "Resources/Updates/Check for Updates"+PI restart does not help, try to reset PI by holding down the Ctrl-key during startup. If this fails, try http://pixinsight.com/forum/index.php?topic=4119.msg28691#msg28691.

Georg

[Cleon_Wells]

Georg, I couldn't figure out how to reset PI on the Mac so I read a Google search that said to delete the Updates.xri file in the app/contents folder, worked like a charm, had 40 modules now I've got 42 modules.    Thanks
Now I'll have to try working with those Dark subs that gave me the high Scaling Factor with the WS pixel rejection settings during integration for the MasterDark.
Cleon

[astropixel]

I have done a lot of experimenting with BatchPreprocessing using several sets of dark frames (10 subs only), at various temperatures, without any calibration errors.

However, the script baulked at 5 dark frames, with a no compatibility error.

As far as temperature regulated DSLR dark frames are concerned, I'm very satisfied with the quality and consistency of the process and integrated (stacking artifact cropped) image with as few as 10 dark and flat frames.

I used 50 bias frames for a fixed pattern template.