Dark frame scaling fooled

[papaf]

Hi,
I'm working on a set of images from a friend, taken with a Canon 350D camera. I have a set of biases, darks and of course lights.
The matter I'm trying to solve is that I get better results if I don't rescale the dark. If I do, some of the typical thermal noise in a corner remains in the final image.
It must be noted that the darks are quite good, ie they're the same ISO, exposure and temperature is quite similar.
Yet, I didn't expect the dark to be scaled nearly in half...
What could be the source of this big mismatch introduced by the scaling?

Thanks!

[pfile]

it's just how PI works - the noise in the image calibrated with the scaled dark is less than the noise in the image with the unscaled dark - PI scales the dark until the noise is minimized. this may result in hot or cold pixels left partially corrected.

for DSLR it might be better to never scale your darks, since canon plays tricks behind the scenes with dark signal suppression and so the dark signal does not necessarily scale linearly with exposure times.

rob

[papaf]

Interesting, I never thought at it this way.
Thanks for the info!

[MikeOates]

And to take Pfiles reply further, because DSLR darks don't scale well, you may find taking dark-flats an advantage, i.e. take a set of darks frames the same exposure as the flats. The batch preprocessing script will use those when calibrating the flats.

Mike

[papaf]

Thanks to you too, Mike. Duly noted!

[pfile]

And to take Pfiles reply further, because DSLR darks don't scale well, you may find taking dark-flats an advantage, i.e. take a set of darks frames the same exposure as the flats. The batch preprocessing script will use those when calibrating the flats.

Mike

good point. i'm assuming that's the only way to get BPP to use special calibration files on the flats? i usually only calibrate my flats with bias frames since the flat exposures are pretty short. but BPP probably can't be forced to do this?

rob

[Phil Leigh]

Rob - if you just add flats, bias and lights in the BPP it will use the bias to calibrate flats and lights correctly
This is what I do all the time since my (cooled) camera does not require darks at all. Indeed, adding darks just makes things worse for me because of the bizarre internal processing of Canon exposures over 10 seconds...
regards
Phil

[pfile]

thanks, but what happens if you also add darks? last i remember the script will try to scale the darks for the flats... so if you have a 600s dark it's going to be scaling all the way down to 2s or so which seems dubious even for a "real" astro camera and certainly for a canon DSLR. of course if it does that it will also subtract the bias from the flat subs, so maybe everything is okay if the scaled dark turns out to be almost 0.

rob

[Phil Leigh]

thanks, but what happens if you also add darks? last i remember the script will try to scale the darks for the flats... so if you have a 600s dark it's going to be scaling all the way down to 2s or so which seems dubious even for a "real" astro camera and certainly for a canon DSLR. of course if it does that it will also subtract the bias from the flat subs, so maybe everything is okay if the scaled dark turns out to be almost 0.

rob

Yes Rob, it will rescale the 600s dark down to 2s. I don't think this will work very well because there is a VERY non-linear transition in Canon DSLR's between exposures < 10 seconds and those > 10s. because of their interesting (and undocumented) internal processing. This processing is not "dark subtraction" ... it's something else. Looks like they may be messing with the dark pedestal?

What I find with my sensor running at ~-30c is that I have no hot/warm pixels and the dark current is so low that bias subtraction + flat division essentially leaves almost nothing (in both flats and lights) but shot noise.

I use a 200-frame master bias taken at -30c

If I try and use darks (either scaled or duration matched) the noise in the images increases significantly in both flats and lights. I've spent a long time analysing the noise figures for different combos of calibration!

If I was using a non-cooled Canon DSLR I would avoid dark scaling and use flat darks. I think other DSLR's (Nikon, Sony etc) would behave more like CCD's with regard to dark scaling...maybe?

I wish Canon would explain what they are doing on > 10s exposures - and it would be even better if it could be turned off!

[georg.viehoever]

...Yes Rob, it will rescale the 600s dark down to 2s. I don't think this will work very well because there is a VERY non-linear transition in Canon DSLR's between exposures < 10 seconds and those > 10s. because of their interesting (and undocumented) internal processing. This processing is not "dark subtraction" ... it's something else. Looks like they may be messing with the dark pedestal?...

Is there anyone out there who knows the details of what Canon is doing to their RAWs? Sure there must be someone who did detailed investigations or who decoded the firmware. Maybe I just did not find the right webpage (yet).

Georg

[Phil Leigh]

...Yes Rob, it will rescale the 600s dark down to 2s. I don't think this will work very well because there is a VERY non-linear transition in Canon DSLR's between exposures < 10 seconds and those > 10s. because of their interesting (and undocumented) internal processing. This processing is not "dark subtraction" ... it's something else. Looks like they may be messing with the dark pedestal?...

Is there anyone out there who knows the details of what Canon is doing to their RAWs? Sure there must be someone who did detailed investigations or who decoded the firmware. Maybe I just did not find the right webpage (yet).

Georg

Georg - please see here: http://pixinsight.com/forum/index.php?action=post;quote=31052;topic=4432.0;last_msg=31116

[papaf]

From that post, it seems Juan still suggest the use of dark scaling, even with DSLRs. But my empirical experience is not reflecting this...
Is the multi point dark adaptation in place already?
Also: are difference to be expected with newer Canons? My experience has been with a 350D set of images, but I've got access to 550D images too. Since the newest models don't show the typical issues of the older model, could it be dark scaling can work better?

[georg.viehoever]

From that post, it seems Juan still suggest the use of dark scaling, even with DSLRs. But my empirical experience is not reflecting this...
...

I agree. I always avoid dark scaling with my Canon EOS40D, i.e. I use darks that have the same exposure time as the lights. Any significant exposure difference causes severe artifacts.

I was hoping someone would know the details of what Canon does, so we can "re-linearize" images before sending them into PI side pre-processing.

Georg

[Phil Leigh]

An interesting thing... If you setup the camera to do in-camera dark subtraction the turn the power off immediately after taking a picture (so before the dark frame gets subtracted) the resulting image is much noisier than normal - this means to me that the "dark" processing Canon is doing can be defeated in firmware.

Maybe a comparison of the two images would reveal something?

[georg.viehoever]

Just to be sure: You know that you should disable "in-camera dark subtraction" for astrophotography, right?
Georg