Preprocessing Canon DSLR frames - a different approach

A

astropixel

Guest
I have taken a completely different approach to this DSLR image business. Re-reading Craig Stark's paper, I conclude that the modification of DSLR RAW data in-camera warrants a different method of calibration.

Based on nothing more than a hunch - very unscientific - I ran a few preprocessing exercises using different combinations of frames for calibration. The method that produced scaling factors near 1.0, with consistent noise estimates across all channels and weights near 1, was considered valid. Presuming that these results are desirable.

Except for the use of Multiplicative and Equalize Flux for flat frame integration, all other frames were integrated with the same parameters as the light frames. Flats were calibrated with a bias frame, but the dark was not. Light frames were batch processed with a master dark (no bias subtraction) and master flat only.

Frame set, as follows; -5C ISO800, 66 bias, 12 dark, 18 flats and 18 lights

Master frame Mean - bias 255, dark 255, flat with bias subtraction 800 (1056 without bias subtraction)

Scaling factors 1.0, Noise estimates ~7, Weights ~0.99 - 0.98.

These are the most consistent results to date. Certainly cold finger, accurate, regulated cooling has an influence on the residual dark current value. It's not high and personally, I think the bias is better left in the dark in this case - for best results.

Superbias may however, produce different results.

EDIT: http://pixinsight.com/forum/index.php?topic=7079.msg47768#msg47768
 

Attachments

idiaz

Well-known member
Rowland said:
Light frames were batch processed with a master dark and master flat only.
If by this you mean that lights were calibrated with an uncalibrated master dark, and a calibrated master flat, and without a master bias, then I am not surprised by your results: you avoided completely data truncation in dark current calibration. This problem is more evident in cooled cameras, where the darks current is lower and thus, more prone to clipping when bias-subtracted.

The only catch is that, as it is a lot easier to come up with a very accurate master bias (frames in the hundreds, and possibly superbias script), you are kind of settling for a lower quality bias calibration, given by the amount of dark frames that you have (frames in the tens). My guess is that if you build an accurate master bias, and you don't calibrate your master dark (until you calibrate lights, that is), your results will be even better.

Ignacio
 

idiaz

Well-known member
Adding to the above, and borrowing from the other post on DSLR dark investigation, long exposures seem to suffer some form of black point adjustment inside canon cameras, even if all camera options are disabled. For instance, in my 1000D, I have found that the adjustment kicks in for exposures longer than or equal to 10 seconds, evidenced by the saturation levels achieved.

Hence, when subtracting a master bias, to a long-exposure dark frame that has already suffer some constant level subtraction, data gets clipped.

Ignacio
 
A

astropixel

Guest
Just as you describe, Ignacio, except that the bias and dark frames were integrated using the same settings as light frames. This seemed reasonable given the in camera manipulation - for no other reason than I have no basis on which to trust the actual status of the RAW data.

I will work on better quality calibration frames - while my methodology may not be particularly scientific, it produced the best results to date.

Thanks for your feedback. Cooling introduces another side to image calibration.
 

Phil Leigh

Well-known member
Feb 27, 2012
220
0
A question; if the camera is adequately cooled (sensor at ~ -25c) why bother at all with darks? The residual dark current is very low and the few remaining hot pixels are easily removed by dithering.

My empirical testing (i.e. looking at the images) suggests that dark subtraction makes no visible difference to the end result...?
 
A

astropixel

Guest
I agree Phil, I don't think there is a need for darks at those sorts of temperatures. My DIY gear wont do that and at -5C seems to be on the cusp of darks or no darks.

I am currently experimenting with dark frames of 0C 210sec. Preliminary findings indicate that even at 0C bias subtraction of darks truncates data. I am inclined to think that darks might be unnecessary. Bias subtraction of lights and flats may be all that is required. At the moment I'm exploring the data to get a better idea of what I'm working with.
 

AstroScience

Well-known member
Jan 25, 2013
169
0
Rowland, I'm in the same boat as you with cooled DSLR.

My last data that I have acquired was 16 frames 900s each, after 15 minutes of exposure I got frames at 0c.
I also have proper uncalibrated Master Dark built from 45 frames matching the temps.

Next I made 2 sets, one when I did calibrated with master dark and the second when I calibrated only with master flat and master bias.

I can't tell difference between both final integrations of both sets. When integrated, I used no Pixel Rejection and many more hot pixels were obvious on the stack with no dark calibration, but they all were nicely rejected later. So I skip using darks at all.
 
A

astropixel

Guest
Hi Sergio. I need a lot more bias frames and I mean to try out your's, Phil's and Ignacio's suggestions and see how it looks with my data. The mistake I made was assuming that the usual bias subtraction of darks would be OK - not so at these temperatures.
 

AstroScience

Well-known member
Jan 25, 2013
169
0
I have also noticed, that if I subtract Bias from dark frames, be they at 25c or -5C , cause same truncation (K=0).

Ignasio suggest to create master dark, then calibrate it with Bias, during lights calibration, but that makes me wonder. When I calibrate my master dark with master Bias, as if during light calibration, that still causes truncation...

Maybe we should pre-calibrate dark frames with pedestal output then create master dark? I'm still getting my head around it...

 

pfile

PTeam Member
Nov 23, 2009
5,603
102
you can do that - set an output pedestal when calibrating the dark subs with the 'problem' bias files. the PEDESTAL keyword will get written to the calibrated darks and ImageIntegration will pick it up and put the PEDESTAL keyword (with the value of course) on the integrated master dark.

then when you use that master dark , the pedestal will automatically be subtracted by ImageCalibration.

rob
 

AstroScience

Well-known member
Jan 25, 2013
169
0
Thanks, Rob
I did exactly that, calibrated 45 darks with master Bias (200 frames) while adding pedestal of 200 . Integrated them to master Dark and saved. Made another master Dark from the same 45 darks, only that time didn't calibrated them with master Bias.
Then I took Light frame and calibrated with both master darks.
I got almost identical results....



 

idiaz

Well-known member
I have tested this in detail. While bias-subtracting an uncalibrated master dark during light calibration may still clip some pixels, if you use more than, say, 15 darks to build your uncalibrated master, then the clipping is very minor. I had measured differences in noise only in the 3rd or 4th digit, when comparing with a pedestal approach. When conducting these test, I looked for the smallest pedestal manually, using pixelmath, to avoid truncation (looking at min value in image statistics).

But be aware that this is not what the option "pedestal" does in ImageCalibration. In there, a pedestal is subtracted from all images, as a first step, which is not what we need.

To do this correctly, you need to add a pedestal to the uncalibrated master dark using pixelmath, and use this to calibrate lights.

Ignacio

 

AstroScience

Well-known member
Jan 25, 2013
169
0
Ignasio,
I used pedestal option that is in Output Files section: Output Pedestal (DN) and mean value of the single frames were raised by this number, checking the statistics.
Is this wrong way to do it?
 

idiaz

Well-known member
That looks fine, Sergio, only if you calibrate your dark frames this way and then build a master dark. I prefer to do it manually to have more control (and understand exactly what is going on in terms of order of operations). My choice is to integrate an uncalibrated master dark, and then, if you like, add a pedestal so that (master_dark_uncal + pedestal) - master_bias > 0 for all pixels. Then use (master_dark_uncal + pedestal) as new uncal master dark during lights calibration, ensuring that you tick the calibrate option.

Ignacio
 
A

astropixel

Guest
Thinking a bit more clearly about this (uncal_master_dark + 0.0002) - superbias, keeps dark frames values >0. Lowest k = 0.0001 and hot pixel values as much as k=0.0454. Dark frame mean ~13.

The lowest bias k value 0.0037, consequently adding 0.0002 avoids zero value pixels, and adds very little to the master dark frame, given that uncalibrated it is predominantly bias value.

Is this a valid method?
 

Attachments

bitli

PTeam Member
Feb 1, 2009
520
3
Switzerland
I have measured darks and bias of a few DLSR (non cooled).  These are my finding:

Most DSLR do some internal processing of the image even in raw mode. The expectation we have on well behaved bias and darks do not apply. The darks are 'relocated' at about the same level than the bias.

As suggested by the first post, using a factor near 1 (that is darks at the same exposure than the lights) will work the best. We do not need to optimize the darks (optimize here just mean applying a uniform scaling factor, there is no magic). In this case there is no point in calibrating the darks and lights as the bias cancels out in ImageIntegration even if intermediate results are negative. Naturally if you make a separate bias subtraction and then truncate the darks, then you get a difference, but this is no good (why bother with complex rejection algorithms if your are going to cut everything which is below 0.5 sigma of the mean anyhow?).

The flats should be calibrated, in theory with a flat dark, because we want to make the master flat values directly proportional to the illumination. However because the flats have short exposure and have (normally) much higher values than darks or lights, we use the bias as a proxy of a light dark and this works fine enough. On any DSLR I have seen there is no 'amplifier glow' or any significant systematic and repeatable error on the bias (I mean significant compared to the level of the flats). The simplest way to have a good bias is to take 1 (one) very short exposure and takes its median with PixelMath (I hear the shouting here - but try). Spend your time on making good flats (and check their histogram in linear mode, not on the DSLR screen). You will get a much greater enhancement of SNR by making sure your flats are near or above the 50% ADU level than by accumulating bias.

I think that some of the complications than most people used to add (me included) come from one or more of these reasons:
  • We focus too much on the numbers calculated by PI and not enough on checking the intermediate results, the logs, the histograms. If you truncate your darks so that they are 0 almost everywhere, they will not add any noise!  But they are also pretty useless. When you use optimisation, look at the factor logged by PI.  If you use 0.05% of a dark of the same duration of your light, you should ask yourself if this is meaningful.
  • If your DLSR has statistically a median of the dark slightly above the median of the bias, even very small, then by taking hundreds of them you will narrow the histogram enough so that there will be little truncation. But what will be the use of this small calibration?
  • We apply methods that were developed for regulated cooled CCD providing raw images to DSLR that are no regulated and already do some kind of pre-processing internally.

Note that very good bias are still needed if you want to calibrate darks for optimization on cooled CCD. I only state that this is not useful (and in most case harmful) for most DSLR.

If time permit I will write a separate article and provide some more quantitative results, but I urged people having (non regulated) DSLR to try the simple method of using non calibrated darks and lights (with the same exposure time, iso, and approximate temperature), make good flats, calibrate the flats with a single value (the median of a bias is fine), and do not use optimize (due to lack of reliable calibration).

If you have no gradient due to amplifier or battery temperature in your dark (or if you have a DSLR that already truncated most of the dark and bias to 0, as done by some brands), you can avoid the calibrating with the dark and rely on CosmeticCorrection (possibly with a dark as support) to remove the hot pixels.

A possible fine tuning would be to apply a linear transformation to the flat (changing the bias value and/or multiplying it by a value very near 1.0) to fine tune the removal of the vignetting.  Because the DSLR has no well defined absolute offset value, using the bias is just an approximation. Try taking 2 flats of a surface with the same illumination, using a time ration of 2, like 1 and 2 seconds, and then make a linear fit between them, you should get a factor near 2 and the value of the offset ... repeat the experience with different shots. This will give you an idea of the range of possible offset values.

-- bitli
 
A

astropixel

Guest
I agree. As in my reply to your response to my earlier thread. I came to a similar conclusion, which you explain more clearly and with a better understanding of the problem. This is very good. A sticky tutorial would be excellent - someone?
 

Phil Leigh

Well-known member
Feb 27, 2012
220
0
I think this needs 2 stickies - for cooled vs normal DSLR's

I think I agree with your findings so far.

My personal findings with a cooled DSLR (-25c) are that the BPP script calibration process works fine with just Bias, flats and Lights. Dithering during capture and Winsorized sigma or linear fit rejection in Integration seem to deal with the residual warm pixels (that darks would normally deal with).
 
A

astropixel

Guest
I ran a few more pixel math exercises on frames from my cooled Canon 1000D @ -5C ISO1600. I think the results indicate your point Phil. In-fact, the dark component was troublesome, in that at -5C it is virtually all bias.

Normalization means of course, that k=0.0000 can actually be negative. Superbias subtraction required less addition to the light frame to set all pixels >0; that is, if my methodology is valid.

I followed up with a set of ISO800 frames @ -5C of the same object. These required addition of 0.0008 for superbias subtraction. Both sets were calibrated in BPP and registered and integrated with the standalone modules. The result was smoother. Winsorized clipping produced a better result than Linear clipping in this case.

STF on calibtrated dark frames showed a little variation (overlaid transparency).

Bias and dark subtraction produced different results; as follows

Caveat- >0 means dark pixel sampling found no values <0 ... I stretched the STF background so that pixels of 0.000 and blacker were visible.

Note: master_dark 40 frames (includes bias) -  with_bias and no_bias for clarity. Superbias 200 frames - average value k=0.0039 -

light - bias = truncated (+ 0.0026 = >0)

light - superbias = truncated (+ 0.0012 = >0)

light - dark = truncated (+ 0.0246 = >0)

light - master_dark_with_bias = truncated (+>255 ~>0); gave up on this one

(light - superbias) - master_dark_no_bias = truncated (+>0.0040 = >0);

dark - bias = truncated (+ 0.0032 = >0)

dark - superbias = truncated (+0.0020 = >0)

master_dark_with_bias - superbias = truncated (+0.0006 = >0)
Superbias subtraction seems to be more effective, whereas a single bias is not far behind, raising pixel values >0.

Given that DSLR RAW data is modified in camera, it seems necessary to add a nominal value to the lights to avoid negative numbers. I notice the value changes with ISO all other things being equal.

The addition of 0.0012 to all ISO1600 frames and 0.0008 to iso800 frames calibrating with superbias and flat produced a brighter, smoother image - 12bit camera.

I used ImageContainer and Pixel Math instance - $T+0.0012 and Replace Image.
 

Phil Leigh

Well-known member
Feb 27, 2012
220
0
Hmmm... if I subtract a master bias from a dark frame I do get some zero value pixels. If I subtract a master bias from a light frame I don't get any zero value pixels - there are some pixels with very low values but no zeroes.