Author Topic: Image Calibration - Optimization threshold of Master Darks  (Read 2611 times)

Offline dpaul

  • PixInsight Addict
  • ***
  • Posts: 124
Can someone help with this general question-

I am ok with calibrating light frames using Superbias and a Master dark (without extracting bias) then using both in ''Image Calibration''. Sometimes however it warns me that the optimisation threshhold may be too high when using the default of 3. This seems to occur with luminance frames which have more saturated sky background - moving the threshold down to 2.0 gets rid of the warning message.

So here's my question - the higher the threshold, the darker the background of the calibrated frames becomes. At 10 fer example the result seems much better. So I'd like to know why this is not good practice?

Thanks

David




Offline bulrichl

  • PixInsight Guru
  • ****
  • Posts: 524
Re: Image Calibration - Optimization threshold of Master Darks
« Reply #1 on: 2018 February 25 02:03:55 »
Hi David,

see this thread for an explanation of what the input value "Optimization threshold" effects:

https://pixinsight.com/forum/index.php?topic=11922.0

Quote
So here's my question - the higher the threshold, the darker the background of the calibrated frames becomes. At 10 fer example the result seems much better. So I'd like to know why this is not good practice?
I don't understand the correlation higher optimization threshold -> darker background, but more important: I don't understand your judgement, that a darker background in the calibrated frames is much better as well. Can you please explain?

Bernd

Offline dpaul

  • PixInsight Addict
  • ***
  • Posts: 124
Re: Image Calibration - Optimization threshold of Master Darks
« Reply #2 on: 2018 February 25 04:09:57 »
Hi Bernd,

Thanks for the quick reply - yes I did see you interesting previous post before I even sent mine - no final conclusion there but I could the logic of challenging why the optimisation value of 3.0 was necessarily the best one.

My comments were not making a judgement, more an observation!

I did a test night on a single light frame (Luminance) that was calibrated at varying optimisation levels, the optimisation window was fixed at 1024 and everything was default. Here are the pixel counts (% in brackets) using statistics (binning was 2x2):

Uncalibrated raw frame ---- 4066260 (99.89667%)
Optimisation level 0 - ---4070466 (100%)
Optimisation level 0.1----4070466 (100%)
Optimisation level 0.5---ditto
1.0---ditto
2.0---ditto
3.0---ditto
5.0---ditto
7.5---ditto
10.0---ditto

For interest, the uncalibrated frame detailed statistics were:
mean 22166
med 21952
min 14928
max 65520

The 0.0 optimisation level calibrated frame details were:
mean 20900
med 20641
min 13637
max 64239

The 3.0 optimisation level calibrated frame details were:
mean 19905
med 19644
min 12740
max 63310

The 10.0 otimisation level calibrated frame details were:
mean 13909
med 13637
min 7050
max 57730

The higher the optimisation level the darker the appearance of the unstretched frames, the histogram peak effectiveley moves to the left. I have no idea whether it makes any difference at all but when combining the LRGB results later, there seems to be a better balance between the lum and RG and B frames.

This may be coincidental but a quick trial of processing M65 with the optimisation at 10.0 seemed to give better color calibration (using color calibration).

Certainly the warning of ''optimisation threshold may be set too high'' when I'm using the default of 3.0 shouldn't be an issue - if I slide down to 2.0 then is that really benefitting me?

Again, these are all observations, I'm open to logical conclusions that suggest what's best.

Thanks

David





 

Offline bulrichl

  • PixInsight Guru
  • ****
  • Posts: 524
Re: Image Calibration - Optimization threshold of Master Darks
« Reply #3 on: 2018 February 26 11:14:39 »
Hi David,

Quote
My comments were not making a judgement, more an observation!

Sure, it was you who wrote in your first post:
Quote
So here's my question - the higher the threshold, the darker the background of the calibrated frames becomes. At 10 fer example the result seems much better.

'Better' is sort of a judgement, isn't it?

OK, let's stay at your first post:
Quote
I am ok with calibrating light frames using Superbias and a Master dark (without extracting bias) then using both in ''Image Calibration''. Sometimes however it warns me that the optimisation threshhold may be too high when using the default of 3. This seems to occur with luminance frames which have more saturated sky background - moving the threshold down to 2.0 gets rid of the warning message.

The warning has nothing to do with the light frames that you want to calibrate. It only depends on the MasterDark you are applying and on the inputted optimization threshold.

As described in the thread that I cited, optimization threshold has the following meaning:

Quote
PixInsight computes median and standard deviation of the MasterDark. From the input value "Optimization threshold" (in sigma units) the corresponding value of Td0 is evaluated (a floating point number, range 0 - 1) so that (Median + Td0) defines the lower bound for the set of dark frame pixels that are used for the dark frame optimization. Td0 is outputted to the console as well as the number of pixels and the the number of pixels in %. So for dark frame optimization PixInsight uses only the pixels with intensity between (Median + Td0) and 1.

So if you increase optimization threshold more and more, the pixel count (= number or fraction of pixels of the MasterDark that is used for Dark frame optimization)  is getting lower and lower. This value is outputted to the process console only, and it has nothing to do with the pixel count that is indicated by ImageStatistics for the calibrated image. The statistical values (of the calibrated lights) that you gave in your second post are not meaningful in this respect. You need to observe (as a function of 'optimization threshold) the values 'count [%]' and 'k0' that are outputted to the process console.

The warning will appear, when this pixel count (process console!) has been fallen short of an PI-internal limit. So PI warns you that the fraction of pixels that is used to calculate k0 (the dark optimization factor) is far too low! Therefore one should decrease the optimization threshold if the warning occurs - if you don't, you also might determine k0 by throwing the dice. My guess is that in your case increasing optimization threshold results in decreasing mean or median of the calibrated lights because k0 is calculated more and more imprecisely. Bottom line: it doesn't make sense to increase optimization threshold when PI outputs the warning - you have to decrease it. Calibrations results with an optimization threshold that lead to the warning are supposed to be incorrect. I fyou want to repeat your experiments (and take a note of count [%] and k0), begin with an optimization threshold = 3 and decrease it successively.

In my experiments with the Canon EOS 600D (= Rebel T3i) the warning never appeared, independent from the inputted optimization threshold. So obviously your and my camera behave differently, the difference being in the MasterDark. What camera are you using?

Bernd

Offline dpaul

  • PixInsight Addict
  • ***
  • Posts: 124
Re: Image Calibration - Optimization threshold of Master Darks
« Reply #4 on: 2018 February 26 15:20:40 »
Hi Bernd,

The Camera is Atik Horizon (mono) with cooled CMOS sensor.

When using the default optimisation of 3, the warning usually occurs with the luminance frames and not the RGB ones. Not all the time, say 50% of occasions.
So it would seem to be occuring when the frames are more light saturated (appear a lot brigher unstretched than the RGB). Maybe I could also slightly back-off on the exposure length if necessary. I live in a reasonably dark area, visible mag about 5.5 to the naked eye but there is some light pollution. I'm not using any other filters except for the Baader LRGB set.

I'll take note of the warnings when they occur and drop the optimisation a little.

Thanks for the input!

Regards

David





Offline bulrichl

  • PixInsight Guru
  • ****
  • Posts: 524
Re: Image Calibration - Optimization threshold of Master Darks
« Reply #5 on: 2018 February 27 01:20:12 »
Hi David,

with my cooled CMOS camera (ASI294MC Pro) I did not notice an improvement whith dark frame optimization enabled - rather the contrary, because the "amplifier glow"  would not be completely calibrated out (see https://pixinsight.com/forum/index.php?topic=11968.0 ). At the bottom of the post you'll find the settings that I use for image calibration with this camera - just a suggestion.

Bernd

Offline dpaul

  • PixInsight Addict
  • ***
  • Posts: 124
Re: Image Calibration - Optimization threshold of Master Darks
« Reply #6 on: 2018 February 27 12:29:19 »
Hi Bernd,

Interesting -

I'll try both ways in due course.

Thanks

David