Problems to calibrate DSLR images

[georg.viehoever]

Hi,

I have been following Astropixel's excellent tutorial http://pixinsight.com/forum/index.php?topic=2276.0 for calibrating images of my Canon EOS 40D. So far the results are not really convincing.
The screenshot below shows (all with strong STF to highlight the remaining vignetting)
- top left: calibration in PI  . Obviously something is wrong with the red channel
- top right: original RAW frame
- bottom: calibration results from DeepSkyStacker

I am not really sure what is going wrong here. One idea is that the dark calibration of the flats is not really working. I took the darks as 20 second frames, the flats as 1/64 sec frames. The scaling factors of the flat calibration process do not really reflect this diifference of a factor of 12800. Console output:

Code: [Select]

Calibrating target frame 24 of 28
Loading target frame:
E:/Benutzer/Oeffentlich/Pictures/2010_08_12_Amerika_Astro/2010_09_19_calibration/flat24mm5.CR2
Timestamp: Sun Sep 19 15:03:30 2010
Camera: Canon EOS 40D
Owner: GeorgSibylle Viehoever
ISO speed: 800
Shutter: 1/64.0 sec
Aperture: f/4.0
Focal length: 24.0 mm
Embedded ICC profile: no
Number of raw images: 1
Thumb size:  1936 x 1288
Full size:   3944 x 2622
Image size:  3908 x 2602
Output size: 3908 x 2602
Raw colors: 3
Filter pattern: RGGBRGGBRGGBRGGB
Daylight multipliers: 2.444244 0.930336 1.163797
Camera multipliers: 2289.000000 1024.000000 1496.000000 1024.000000

Invoking: dcraw -D -k 0 -t 0 -o 0 -4
Decoding Canon EOS 40D file (3908x2602 pixels, ISO=800, Exposure=1/64s): 100%
Loading raw image: 100%

Writing output file: E:/Benutzer/Oeffentlich/Pictures/2010_08_12_Amerika_Astro/2010_09_19_calibration/calibrated_flats/flat24mm4_c.fit
Dark scaling factors:
k0 = 0.020
k1 = 0.031
k2 = 0.105
Writing FITS image: 32-bit floating point, 3 channel(s), 3908x2602 pixels: 100%

Any comments or suggestions?

Georg

[Carlos Milovic]

Are you using the Hubble palete?

I would not use darks for the flats, as they have short exposure times. Bias should be good enough.

[astropixel]

Georg. I have seen similar things, and up until now it has been a flat problem. Interested to know what you find.

[georg.viehoever]

Hi,
I tried to do the same without using darks during the flat calibration. The result is not really different. I also tried to use different options (with or without bias/dark/flat calibration/optimazation) when calibrating the light frames. Either no improvement, or even worse results. The integrated bias/dark/flat frames seem to be reasonable, see debayered and strongly STFed versions it the screenshot below.

I am getting the feeling that this may be related to the use of "Create RAW Bayer" image in the DSLR_RAW format settings. When reading RAW/CR2 frames in this format, for each pixel either R or G or B are set, the other values being zero. As a consequence, this is also true for the bias/dark/flat/light frames. I get the feeling that the calibration process is thorughly irritated by this.

I tried to reproduce what Image Calibration does, using ($T-bias-dark)/flat as a fomula. As long as I used the images created with "Create RAW Bayer", the results were always unusuble, no matter if I used Rescale Result or with int or float as result. I guess the reason is the numerous divisions by 0 caused by the flat frame. When going to monochrome versions generated with max($T[0],$T[1],$T[2]]), things improved a lot.

Any comments anyone?

Georg

[Carlos Milovic]

Are you using a color image to store the bayer data? If so, I would avoid that, and always use a grayscale one. Check the DCRAW options.

[georg.viehoever]

Hi Carlos,

Are you using a color image to store the bayer data? If so, I would avoid that, and always use a grayscale one. Check the DCRAW options.

Storing them in color images is indeed what Astropixel's tutorial http://pixinsight.com/forum/index.php?topic=2276.0 suggests. See my question in http://pixinsight.com/forum/index.php?topic=2362.0 about this.

Georg

[Carlos Milovic]

Well, I suggest not doing that XD <vbg>

[georg.viehoever]

Carlos,

unfortunately, Juan (and others) suggested otherwise, see for example http://pixinsight.com/forum/index.php?topic=2118.msg13737#msg13737

All,

I finally managed to get useful results from PI. The key insight was that I need to disable "Calibrate" on the Master Flat section (see screenshot). Unfortunately, this option is enabled by default, in AstroPixels's Tutorial http://pixinsight.com/forum/index.php?topic=2276.0 this section is not shown, and I also did not find any reference in this forum that suggested to do so (BTW: Where is the posting that explains what exactly Image Calibration does?). Is there anyone that can confirm that this was my original problem?

The results are neither better nor worse than those obtained with DeepSkyStacker (Screenshot left: DSS, right PI). The relative effort for those 2 results was 10 minutes vs 10 hours... well, I am just a little bit frustrated  ... I want PI documentation, now  ...


As far as the question "DSLR RAW CFA or not CFA" is concerned: this appears not to matter for Image Calibration, but it does matter if you try to emulate the process via PixelMath. If you would like to comment on this question, please use thread http://pixinsight.com/forum/index.php?topic=2362.0.

Georg
 -

[Carlos Milovic]

Yes, it doesn't matter to ImageCalibration. The code understands when the image comes from a CFA pattern, and avoids nasty zero divisions.

Anyway, my point on RGB or K (gray) bayered images is why do we waste disk space/memory with empty data? And, after all, as far as I know, there is no reason to prefer RGB.

[astropixel]

Georg. calibrate is enabled as far as I recall, and this is reflected in the console output during image calibration.

I'm not sure that I understand the problem, it's a little bit over my head, but I've had no problems at all with cr2 images. I did however, go to the trouble of creating a completely new set of bias, darks and flats for my optical train, following Vicent's instructions. This was in response to a problem where flat calibration was rejected and the results poor indeed. These problems did not surface again.

Happy to make changes to the dslr_raw procedure and include any notes that are needed to elaborate on your findings, or create your own

I've had no problems whatsoever processing cr2 images for calibration purposes, for astro images. Terrestrial HDR images were a bit of a challenge, because there are subtle differences, but the dslr_raw format settings produced acceptable results.

Let's get to the bottom of this.

[georg.viehoever]

Hi Astropixel,

my results are strictly empirica with this single set of images only, producing nice calibrated images with no obvious vignetting. And the tooltips for Masterflat/Calibrate in ImageCalibration seem to suggest that an already calibrated flat does not need to be calibrated again. But I would hope for confirmation of this finding from one of those who should know...

Another possible source of problems could also be the fact that the MSR noise estimate usually fails for my images - it often falls back to k-sigma. I had the hope that MSR was more robust with 1.61, but this appears not to be the case. The noise estimate is used to determine some of the weights during calibration and integration.

Georg

[astropixel]

I've noticed the MSR noise evaluation issue also, but it was less of a problem once I sorted out the master calibration issues, and I took this to mean that that's where the problems lay.

I have obtained reasonable results without the use of a flat in some cases. In-fact, some image sets were adversely affected when using a master flat. I am not sure of the correlation, but I suspect different iso values might be the problem, where flats are concerned. While Pixinsight was quite happy to process master bias and dark frames taken at different iso settings - not so with flats.

I dont know whether this is helpful, but I kept coming back to flat calibration as the problem. That's why I completely replaced my master library and in some cases didn't use a flat at all. I did not use a flat with the calibrate box unchecked. I'll try it over this coming weekend and let you know.

[pauljv]

I used to get the problem of MRS noise evaluation failing, since using the VNG debayer that I am currently writing, this tends to be successful.  If you'd like to give it a try I'll e-mail you a copy.  It's currently compiled in Windows 32 and 64 bit versions and will debayer RGGB images only, also no batch script.
Regards Paul

[georg.viehoever]

... I am not sure of the correlation, but I suspect different iso values might be the problem, where flats are concerned. While Pixinsight was quite happy to process master bias and flat frames taken at different iso settings - not so with flats.
...

Hi,

as I understand the calibration process, it is important to take light, flat, bias, darks all at the same ISO values (different exposure times are permitted). Since the ISO setting influences the gain of the amplifier in the A/D converter, different ISOs give different bias values. I am not sure if it is possible to compensate for this by estimating ISOs from the noise level, but if MSR is unreliable, so would be any algorithm using MSR to estimate ISO as well.

Georg

[georg.viehoever]

I used to get the problem of MRS noise evaluation failing, since using the VNG debayer that I am currently writing, this tends to be successful.  If you'd like to give it a try I'll e-mail you a copy.  It's currently compiled in Windows 32 and 64 bit versions and will debayer RGGB images only, also no batch script.
Regards Paul

Thanks Paul. All this calibration stuff essentially happens on the RAW pixels, without any real debayering. So VNG would probably make no difference in the calibration scenario.

Georg