Author Topic: Dark frame scaling fooled  (Read 10971 times)

Offline Ignacio

  • PixInsight Old Hand
  • ****
  • Posts: 375
    • PampaSkies
Re: Dark frame scaling fooled
« Reply #15 on: 2013 October 10 08:46:54 »
I always thought that dark scaling in PI was purely empirical, i.e., to minimize resulting noise. So no linearity assumption is built in, that I know of.

Ignacio

Offline georg.viehoever

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2132
Re: Dark frame scaling fooled
« Reply #16 on: 2013 October 10 11:03:54 »
Ignatio, it is also an empirical finding that for my DSLR not using dark scaling is better...
Georg (6 inch Newton, unmodified Canon EOS40D+80D, unguided EQ5 mount)

Offline Ignacio

  • PixInsight Old Hand
  • ****
  • Posts: 375
    • PampaSkies
Re: Dark frame scaling fooled
« Reply #17 on: 2013 October 10 11:20:15 »
Ignatio, it is also an empirical finding that for my DSLR not using dark scaling is better...

Georg,
Are you saying that if your master dark is mismatched to your lights (by duration and/or temperature), you do better by dark-subtracting without scaling? Or by not dark-subtracting at all?

Ignacio

Offline georg.viehoever

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2132
Re: Dark frame scaling fooled
« Reply #18 on: 2013 October 10 11:53:01 »
No, I just make sure my darks are properly patched to the lights.
Georg
Georg (6 inch Newton, unmodified Canon EOS40D+80D, unguided EQ5 mount)

Offline AstroScience

  • PixInsight Addict
  • ***
  • Posts: 169
Re: Dark frame scaling fooled
« Reply #19 on: 2013 October 10 11:58:14 »
Georg, like, degree to degree? You have cooled DSLR?

Offline georg.viehoever

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2132
Re: Dark frame scaling fooled
« Reply #20 on: 2013 October 10 13:11:20 »
No, not degree to degree - a colled DLSR so far seemed to be overkill. Just same exposure on the same night.
Georg (6 inch Newton, unmodified Canon EOS40D+80D, unguided EQ5 mount)

Offline pfile

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 4729
Re: Dark frame scaling fooled
« Reply #21 on: 2013 October 10 13:53:54 »
I always thought that dark scaling in PI was purely empirical, i.e., to minimize resulting noise. So no linearity assumption is built in, that I know of.

Ignacio

well the dark scaling is done linearly, by multiplying the dark frame by some number. what i am saying is that if canon has played tricks with the dark signal in the light frame (and also the dark frame) then the dark is not truly a dark. we don't know what they have done to it. i guess you could probably give ImageCalibration some bogus "dark" frame and it would then scale it until the noise in the result was minimized. but that does not make the result correct...

i have scaled DSLR darks in the past and mainly the problem is that hot pixels are either under- or over- corrected. generally these are handled fine by dithering and rejection during integration, though. i have not done any empirical experiments to see if the results are superior with scaled DSLR darks; i have just been put off of it due to the lack of knowledge about what canon is really doing behind the scenes to the dark current.

An interesting thing... If you setup the camera to do in-camera dark subtraction the turn the power off immediately after taking a picture (so before the dark frame gets subtracted) the resulting image is much noisier than normal - this means to me that the "dark" processing Canon is doing can be defeated in firmware.

Maybe a comparison of the two images would reveal something?

now that is really interesting, i did not know about that. that must mean that it writes an unmodified image to the flash card's filesystem and then later reads it back in, subtracts the dark, and then deletes (or overwrites) the original file? i would not have expected that but then again a CR2 is pretty huge so they may not have enough memory on board the camera to hold 3 frames to do the subtraction without writing the image out anywhere.

rob



Offline Ignacio

  • PixInsight Old Hand
  • ****
  • Posts: 375
    • PampaSkies
Re: Dark frame scaling fooled
« Reply #22 on: 2013 October 11 04:43:59 »
well the dark scaling is done linearly, by multiplying the dark frame by some number. what i am saying is that if canon has played tricks with the dark signal in the light frame (and also the dark frame) then the dark is not truly a dark. we don't know what they have done to it. i guess you could probably give ImageCalibration some bogus "dark" frame and it would then scale it until the noise in the result was minimized. but that does not make the result correct...

i have scaled DSLR darks in the past and mainly the problem is that hot pixels are either under- or over- corrected. generally these are handled fine by dithering and rejection during integration, though. i have not done any empirical experiments to see if the results are superior with scaled DSLR darks; i have just been put off of it due to the lack of knowledge about what canon is really doing behind the scenes to the dark current.

Yes, scaling is multiplication by a constant. What I meant by linearity is the duration and temperature dependence of dark current. In this sense, PI could take the dark current form the master dark and use its duration and temp as a basis to extrapolate/interpolate linearly and come up with a scale factor. Instead, it just empirically minimizes the resulting noise.

If you throw some random dark, there will likely be no correlation with the light noise, so k will end up being zero. So I guess, even with DSLR that somehow messes up the raw files, there is some value in dark scaling. And, of course, the best thing is to just match time and temp, and avoid scaling altogether. In my experience, temp scaling within +/- 2°C works very well. Never tried time scaling.

Regarding hot pixels, them being highly non-linear, they cannot be calibrated whenever there is a linear scaling factor involved, independent of the camera being a dslr or an astro ccd. I understand that PI is working on a solution to this.

Ignacio

Offline Phil Leigh

  • PixInsight Addict
  • ***
  • Posts: 220
Re: Dark frame scaling fooled
« Reply #23 on: 2013 October 11 04:59:59 »
Hot pixels are easily dealt with by dithering and cosmetic correction.
I should point out that the "turning off the camera before the internal dark frame is taken" trick doesn't work on modern cameras because they have enough RAM to hold the light and dark frame in situ and do the calibration.

This is annoying because I'd like to see what Canon is doing. The other annoying thing is that whilst clearly Canon firmware can take a dark frame without firing the shutter (like a CCD) it's not possible to do this yourself.

I wish someone would hack the firmware and figure out what is going on...

Offline papaf

  • PixInsight Addict
  • ***
  • Posts: 182
Re: Dark frame scaling fooled
« Reply #24 on: 2013 October 17 02:05:10 »
I made a test with a friend's data. Attached you find the result. Left is without dark scaling, right with scaling. The dark is as closely matched as possible, ie taken the same night.
Now, it seems quite clear to me that, noise wise, the scaled dark worked much better. But the thermal reddish thing in the corner is obviously still there.
I'm quite confused as what to tell him...