Author Topic: Preprocessing Canon DSLR frames - a different approach  (Read 15197 times)

astropixel

  • Guest
I have taken a completely different approach to this DSLR image business. Re-reading Craig Stark's paper, I conclude that the modification of DSLR RAW data in-camera warrants a different method of calibration.

Based on nothing more than a hunch - very unscientific - I ran a few preprocessing exercises using different combinations of frames for calibration. The method that produced scaling factors near 1.0, with consistent noise estimates across all channels and weights near 1, was considered valid. Presuming that these results are desirable.

Except for the use of Multiplicative and Equalize Flux for flat frame integration, all other frames were integrated with the same parameters as the light frames. Flats were calibrated with a bias frame, but the dark was not. Light frames were batch processed with a master dark (no bias subtraction) and master flat only.

Frame set, as follows; -5C ISO800, 66 bias, 12 dark, 18 flats and 18 lights

Master frame Mean - bias 255, dark 255, flat with bias subtraction 800 (1056 without bias subtraction)

Scaling factors 1.0, Noise estimates ~7, Weights ~0.99 - 0.98.

These are the most consistent results to date. Certainly cold finger, accurate, regulated cooling has an influence on the residual dark current value. It's not high and personally, I think the bias is better left in the dark in this case - for best results.

Superbias may however, produce different results.

EDIT: http://pixinsight.com/forum/index.php?topic=7079.msg47768#msg47768
« Last Edit: 2014 May 12 05:00:44 by Rowland »

Offline Ignacio

  • PixInsight Old Hand
  • ****
  • Posts: 375
    • View Profile
    • PampaSkies
Re: Preprocessing Canon DSLR frames - a different approach
« Reply #1 on: 2014 April 23 13:50:40 »
Light frames were batch processed with a master dark and master flat only.

If by this you mean that lights were calibrated with an uncalibrated master dark, and a calibrated master flat, and without a master bias, then I am not surprised by your results: you avoided completely data truncation in dark current calibration. This problem is more evident in cooled cameras, where the darks current is lower and thus, more prone to clipping when bias-subtracted.

The only catch is that, as it is a lot easier to come up with a very accurate master bias (frames in the hundreds, and possibly superbias script), you are kind of settling for a lower quality bias calibration, given by the amount of dark frames that you have (frames in the tens). My guess is that if you build an accurate master bias, and you don't calibrate your master dark (until you calibrate lights, that is), your results will be even better.

Ignacio

Offline Ignacio

  • PixInsight Old Hand
  • ****
  • Posts: 375
    • View Profile
    • PampaSkies
Re: Preprocessing Canon DSLR frames - a different approach
« Reply #2 on: 2014 April 23 15:12:00 »
Adding to the above, and borrowing from the other post on DSLR dark investigation, long exposures seem to suffer some form of black point adjustment inside canon cameras, even if all camera options are disabled. For instance, in my 1000D, I have found that the adjustment kicks in for exposures longer than or equal to 10 seconds, evidenced by the saturation levels achieved.

Hence, when subtracting a master bias, to a long-exposure dark frame that has already suffer some constant level subtraction, data gets clipped.

Ignacio

astropixel

  • Guest
Re: Preprocessing Canon DSLR frames - a different approach
« Reply #3 on: 2014 April 23 16:28:33 »
Just as you describe, Ignacio, except that the bias and dark frames were integrated using the same settings as light frames. This seemed reasonable given the in camera manipulation - for no other reason than I have no basis on which to trust the actual status of the RAW data.

I will work on better quality calibration frames - while my methodology may not be particularly scientific, it produced the best results to date.

Thanks for your feedback. Cooling introduces another side to image calibration.
« Last Edit: 2014 April 23 19:48:29 by Rowland »

Offline Phil Leigh

  • PixInsight Addict
  • ***
  • Posts: 220
    • View Profile
Re: Preprocessing Canon DSLR frames - a different approach
« Reply #4 on: 2014 April 24 01:20:19 »
A question; if the camera is adequately cooled (sensor at ~ -25c) why bother at all with darks? The residual dark current is very low and the few remaining hot pixels are easily removed by dithering.

My empirical testing (i.e. looking at the images) suggests that dark subtraction makes no visible difference to the end result...?

astropixel

  • Guest
Re: Preprocessing Canon DSLR frames - a different approach
« Reply #5 on: 2014 April 24 03:11:23 »
I agree Phil, I don't think there is a need for darks at those sorts of temperatures. My DIY gear wont do that and at -5C seems to be on the cusp of darks or no darks.

I am currently experimenting with dark frames of 0C 210sec. Preliminary findings indicate that even at 0C bias subtraction of darks truncates data. I am inclined to think that darks might be unnecessary. Bias subtraction of lights and flats may be all that is required. At the moment I'm exploring the data to get a better idea of what I'm working with.
« Last Edit: 2014 April 24 03:39:13 by Rowland »

Offline AstroScience

  • PixInsight Addict
  • ***
  • Posts: 169
    • View Profile
Re: Preprocessing Canon DSLR frames - a different approach
« Reply #6 on: 2014 April 24 09:21:20 »
Rowland, I'm in the same boat as you with cooled DSLR.

My last data that I have acquired was 16 frames 900s each, after 15 minutes of exposure I got frames at 0c.
I also have proper uncalibrated Master Dark built from 45 frames matching the temps.

Next I made 2 sets, one when I did calibrated with master dark and the second when I calibrated only with master flat and master bias.

I can't tell difference between both final integrations of both sets. When integrated, I used no Pixel Rejection and many more hot pixels were obvious on the stack with no dark calibration, but they all were nicely rejected later. So I skip using darks at all.

astropixel

  • Guest
Re: Preprocessing Canon DSLR frames - a different approach
« Reply #7 on: 2014 April 24 13:04:46 »
Hi Sergio. I need a lot more bias frames and I mean to try out your's, Phil's and Ignacio's suggestions and see how it looks with my data. The mistake I made was assuming that the usual bias subtraction of darks would be OK - not so at these temperatures.
« Last Edit: 2014 April 24 20:53:52 by Rowland »

Offline AstroScience

  • PixInsight Addict
  • ***
  • Posts: 169
    • View Profile
Re: Preprocessing Canon DSLR frames - a different approach
« Reply #8 on: 2014 April 24 21:08:00 »
I have also noticed, that if I subtract Bias from dark frames, be they at 25c or -5C , cause same truncation (K=0).

Ignasio suggest to create master dark, then calibrate it with Bias, during lights calibration, but that makes me wonder. When I calibrate my master dark with master Bias, as if during light calibration, that still causes truncation...

Maybe we should pre-calibrate dark frames with pedestal output then create master dark? I'm still getting my head around it...


Offline pfile

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 4213
    • View Profile
Re: Preprocessing Canon DSLR frames - a different approach
« Reply #9 on: 2014 April 24 21:58:28 »
you can do that - set an output pedestal when calibrating the dark subs with the 'problem' bias files. the PEDESTAL keyword will get written to the calibrated darks and ImageIntegration will pick it up and put the PEDESTAL keyword (with the value of course) on the integrated master dark.

then when you use that master dark , the pedestal will automatically be subtracted by ImageCalibration.

rob

Offline AstroScience

  • PixInsight Addict
  • ***
  • Posts: 169
    • View Profile
Re: Preprocessing Canon DSLR frames - a different approach
« Reply #10 on: 2014 April 25 05:23:15 »
Thanks, Rob
I did exactly that, calibrated 45 darks with master Bias (200 frames) while adding pedestal of 200 . Integrated them to master Dark and saved. Made another master Dark from the same 45 darks, only that time didn't calibrated them with master Bias.
Then I took Light frame and calibrated with both master darks.
I got almost identical results....




Offline Ignacio

  • PixInsight Old Hand
  • ****
  • Posts: 375
    • View Profile
    • PampaSkies
Re: Preprocessing Canon DSLR frames - a different approach
« Reply #11 on: 2014 April 25 06:10:54 »
I have tested this in detail. While bias-subtracting an uncalibrated master dark during light calibration may still clip some pixels, if you use more than, say, 15 darks to build your uncalibrated master, then the clipping is very minor. I had measured differences in noise only in the 3rd or 4th digit, when comparing with a pedestal approach. When conducting these test, I looked for the smallest pedestal manually, using pixelmath, to avoid truncation (looking at min value in image statistics).

But be aware that this is not what the option "pedestal" does in ImageCalibration. In there, a pedestal is subtracted from all images, as a first step, which is not what we need.

To do this correctly, you need to add a pedestal to the uncalibrated master dark using pixelmath, and use this to calibrate lights.

Ignacio


Offline AstroScience

  • PixInsight Addict
  • ***
  • Posts: 169
    • View Profile
Re: Preprocessing Canon DSLR frames - a different approach
« Reply #12 on: 2014 April 25 06:51:25 »
Ignasio,
I used pedestal option that is in Output Files section: Output Pedestal (DN) and mean value of the single frames were raised by this number, checking the statistics.
Is this wrong way to do it?

Offline Ignacio

  • PixInsight Old Hand
  • ****
  • Posts: 375
    • View Profile
    • PampaSkies
Re: Preprocessing Canon DSLR frames - a different approach
« Reply #13 on: 2014 April 25 07:02:24 »
That looks fine, Sergio, only if you calibrate your dark frames this way and then build a master dark. I prefer to do it manually to have more control (and understand exactly what is going on in terms of order of operations). My choice is to integrate an uncalibrated master dark, and then, if you like, add a pedestal so that (master_dark_uncal + pedestal) - master_bias > 0 for all pixels. Then use (master_dark_uncal + pedestal) as new uncal master dark during lights calibration, ensuring that you tick the calibrate option.

Ignacio

astropixel

  • Guest
Re: Preprocessing Canon DSLR frames - a different approach
« Reply #14 on: 2014 April 26 06:34:30 »
Thinking a bit more clearly about this (uncal_master_dark + 0.0002) - superbias, keeps dark frames values >0. Lowest k = 0.0001 and hot pixel values as much as k=0.0454. Dark frame mean ~13.

The lowest bias k value 0.0037, consequently adding 0.0002 avoids zero value pixels, and adds very little to the master dark frame, given that uncalibrated it is predominantly bias value.

Is this a valid method?