Problems to calibrate DSLR images

[pauljv]

I thought the noise evaluation took place after the debayering. i.e. when integrating calibrated, debayered images.  I thought that the noise evaluation failed for me because of the poor quality of my images, but it now seems successful with any images.

Regards Paul

[georg.viehoever]

I thought the noise evaluation took place after the debayering. i.e. when integrating calibrated, debayered images.  I thought that the noise evaluation failed for me because of the poor quality of my images, but it now seems successful with any images.

Regards Paul

Hi Paul,

the creation of master dark/bias/flat frames also uses ImageCalibration and ImageIntegration, thus using noise evaluation as well. All this happens before debayer. Calibration of the light frames also happens before debayering. In AstroPixels current workflow, registration+integration of light frames happens after debayer, so that may be the place where your tools may help.

Georg

[pauljv]

Georg

According to Vicent's tutorial when integrating the flat frames "Evaluate Noise" is left unchecked.  It also says "Just remember to disable the Calibrate checkbox in the Master Flat section, as we already have calibrated our master flat frame.". http://pixinsight.com/tutorials/master-frames/en.html

Probably was your original problem.

Regards Paul

[georg.viehoever]

Hi Paul,

Vicent's tutorial also says:

In the Master Dark section, be sure to enable the Optimize checkbox. This will rescale the master-bias-subtracted master dark to fit the thermal noise of each flat frame. Don't worry if your master dark has 1000 seconds of exposure and your flat frame is only 10 seconds: IC will multiply the thermal noise of the master dark by 0.01

So some noise evaluation is done during calibration of flat frames. I did follow Vicent's description by not activating "Evaluate Noise" during integration, so this unfortunately is not the cause of my problems.

Thanks for your hints!
Georg

[astropixel]

OK. So, I've followed Vicent's tutorial to the letter, and have the master flat calibrate box checked during image calibration, and it all works fine.

I should add that the workflow is that described by the developers, and it works everytime for me, including for daylight/terrestrial images, with the exclusion of master dark and flat frames which are not essential.

When we get to the bottom of this I'm happy to change the workflow. For now, it is as described by the developers and should remain that way until finding are conclusive.

[georg.viehoever]

Astropixel,
thanks for looking into this.

Vincent, Juan, ...: Can you guys comment on the question of switching on "Calibrate" for the Master Flat during calibration of the light frames (http://pixinsight.com/forum/index.php?topic=2371.msg15857#msg15857) ? There is also still the question of "CFA or not CFA"  http://pixinsight.com/forum/index.php?topic=2362.0 .

Georg

[Juan Conejero]

Hi georg,

The key insight was that I need to disable "Calibrate" on the Master Flat section (see screenshot). Unfortunately, this option is enabled by default

If you have generated a valid master flat, then you already have calibrated it, so you must disable the corresponding Calibrate option when you calibrate your lights. The same is true for the master dark and the master bias (the latter in case there are overscan regions): they should be already calibrated before calibrating lights.

And yes, you're right in that these options should be disabled by default in ImageCalibration. They will be in the next version.

[astropixel]

I spoke too soon - here are the results with the Master Bias, Dark and Flat frame Calibrate boxes checked and unchecked.

Image on the left (with STF applied) - Calibrate selected.

Image on the right (with STF applied) - Calibrate not checked. The level of posterization is quite marked.

When integrating the images following processing with calibrate unchecked - noise evaluation would not work in MRS or k-sigma.

While MRS was troublesome with the calibrate check boxes checked k-sigma worked.

Eliminating one variable, I have carefully produced master bias, dark and flat frames, according to Vicent's tutorial and these are applied to both images.

I should add that I use an Astronomik CLS-CCD filter. The resulting hue of the RAW image is almost identical to that of the image on the left, and so I presume that the dslr_raw settings are correct. Background correction, colour calibration and SCNR in G, fix that.

Any thoughts?

[georg.viehoever]

...
Image on the left (with STF applied) - Calibrate selected.

Image on the right (with STF applied) - Calibrate not checked. The level of posterization is quite marked.
...

Hmm, so now we have to contradicting experimental results (I get better results with "calibrate flat" unchecked during light frame calibration, you get better results with activated calibration), plus Juan's statement that it should be unchecked, plus the result that MRS noise evaluation often fails (which I also can confirm for my Canon EOS40D images. It often does work with aligned images, but not always. It almost never works before alignment).

A confusing mess...

Georg 

[astropixel]

Georg. I'm inclined to start over. I think a reductionist approach to find the offending component. I'll start with no flat, and the master bias and dark unchecked. Watch this space

EDIT: OK - Same result. I know that if I use nothing but a dark with calibrate checked that the result will be correct, because I have spent hours trying various combinations, but until now always used the default calibration settings. I took particular care to follow Vicent's tutorial exactly, so I'm not concerned about the integrity of master files.

Noise evaluation caused image integration to fail, however, Average value failed on MRS, but evnetually succeeded on k-sigma?

All I can suggest, is that we compare settings and try a small batch of 3 files as a test run and examine the results. So that we are not comparing apples with oranges, so to speak.

As a matter of interest are you using in-camera noise reduction?
 

[Jack Harvey]

Interesting,  For the first time I had Noise Eval cause integration fail.  Wonder is this is a bug?  Juan?

[astropixel]

Experimental runs - using exactly the same set of images and calibration files since posting the tutorial.

With correctly calibrated Bias, Dark and Flat frames, per Vicent's tutorial.

Top image DSLR_RAW - Bayer CFA option - ImageCalibration - No Bias Dark and Flat Calibration. (background neutralisation and colour calibration applied to better illustrate the problem)

Middle image - DSLR_RAW CFA option - Image Calibration - Bias Dark and Flat Calibration.
(background neutralisation and colour calibration applied to better illustrate the problem)

This image - DSLR_RAW Bayer (no CFA) - with and without Bias Dark and Flat Calibration (guess) - http://pixinsight.com/forum/index.php?action=dlattach;topic=2371.0;attach=1172

Georg, the top images look suspiciously similar to that first result of yours.

Note: in the case of the top image both the background (to give it a name) and M8 are badly affected.
In the bottom image, the background is still affected, but M8 has recovered from the previous image.

This is why I am not using the CFA option, despite the success of others.

Personally, I think that we are comparing apples and oranges. Possibly, a more controlled approach might resolve some of the differences?

EDIT: I've added another image (auto STF) - this time, ImageCalibration with the Master Flat Calibration OFF. However, this was only achievable with a Master dark (as a minimum) and Calibration of the dark ON. If I turn the dark Calibration option OFF I get the terrible results posted above.

The only difference that I see in the image calibrated with master flat calibration on and off, is that with calibration off the STF image is lighter and the nebulae less dense.

[pauljv]

I can understand the reason to leave calibration unchecked for flats if they have already been calibrated.

I personally do not calibrate my dark frames before calibrating the lights, so leave calibrate checked for dark frames.  If calibrate were left unchecked I would presume that if any scaling of the darks took place the bias signal still present in the dark frame would also be scaled.  My understanding is if calibrate is left checked the bias signal is removed before any scaling takes place.

I have no idea how I would go about calibrating a bias frame so I leave this checked but have no idea what it does.

Regards Paul

[georg.viehoever]

Hi Rowland,

...
Georg, the top images look suspiciously similar to that first result of yours.
....

Yes, I agree, those suspiciously look like my first experiments. The strong gradients (even going into saturation) only disappeared for me when I deacvitated "Masterdark/Calibrate" during the calibration of the light frames. BTW, flats are absolutely essential for me...but that may depend on the equipment.

I have not had the time to do additional experiments, but here is just and idea what might be going wrong when activating "Masterdark/Calibrate": as said elsewhere, PI sometimes has difficulties in estimating the noise in images correctly. It is using those noise estimates during optimizing frames. So I wonder if this is driving the flat optimization into the wrong direction, resulting in tje saturated red channel I showed in my first post http://pixinsight.com/forum/index.php?topic=2371.0. Actually, when a channel is saturated, the noise is zero, so PI may consider this as optimal  ?

Anyway, this is just a a theory, no experimental results yet...

Georg