ImageCalibration fails to subtract hot pixels

[jeffweiss9]

    This is a recurring problem that must be tractable somehow but I still need help from Vicent or someone to get past it.
    Following Vicent's Calibration Tutorial with additional details picked up from responses to previous posts by myself and Marek, I still find that clicking the optimize box in the Master Dark section of the ImageCalibration tool, as he explained is required so that the bias frame is subtracted from the dark frame (otherwise you get, or can get, the 'dreaded grey' calibrated lights), causes most, if not all, of the bright hot pixels to leak through into the calibrated lights making those calibrated frames completely unusable.  If I do a PixelMath subtraction of the Master Dark from the outputted "calibrated light" frame, it subtracts all those bright hot pixels that came through ImageCalibration although PixelMath then adds in an equally large number of dark cold pixels.  If I keep the "optimize" option UNCHECKED, than in these same images that does not occur and it appears that all (or almost all) the hot and cold pixels are eliminated from the outputted calibrated lights.   So I conclude that checking the 'optimize option" of the ImageCalibration tool is somehow disabling the dark frame subtraction of most hot pixels that can occur otherwise.
    Unfortunately, just unchecking the "optimize" is not the solution of the problem because, although it then does work fine for some (i.e. some filters on some nights) of my data and hot pixels are subtracted , for most of my data it does result in the "dreaded gray" calibrated lights that are also unusable.  
    So on my latest image I again am completely unsuccessful in getting PI to do calibration and it seems to be very data dependent which is driving me nuts.  Because I didn't take flat darks, I can't revert to AIP4WIN as I've done when encountering previous problems and I must find some way to make PI calibration work for me to use this data.
    Thanks very much for help on this nasty problem that I believe has to be a problem with the optimize option in ImageCalibration plus the fact that you can't robustly uncheck that option and always avoid the 'dreaded grey' catastrophe which is somehow data dependent.  After spending many hours on this, I'm just stumped.  My darks are at the same temperature and exposure as my lights and Vicent said that under those conditions, checking optimize in ImageCalibrate was only "recommended" in a different post.  Since I don't see any quality advantage in the images with it checked but just have a slew of extra hot pixels, another possibility would be if someone can just tell me what I need to do to avoid getting the dreaded grey calibrated frames (they don't appear to be recoverable), then at least I would be back in business.
-Jeff
  More details:
Note 1: There was a previous discussion about the Master Bias frames needing to be 16 bit unsigned integers vice 32 bit.  As far as I can tell, the behavior above is independent of whether I save them as 16bit.  I have some with BITPIX=16 that failed (dreaded gray with optimize checked) and some with BITPIX=-32 that "worked" (no hot pixels but requires optimize unchecked to avoid them).  So that just does not correlate with whether the calibrated lights are successful without hot pixels or not (100% calibrated with whether the optimize box is checked in ImageCalibration), or whether I get the 'dreaded grey' calibrated lights when the optimize box is unchecked in ImageCalibration.  Other than the bias frames, I always save them as the PI default choice of "32 bit IEEE 764 floating point."
Note 2: I did have one complete image (my previous ic443) sail through with L, Ha, R, G,B all calibrating correctly without either hot pixels or "dreaded grey" problems with my calibrated lights.  Unfortunately my notes don't tell me what the state of the optimize box was when I processed that image, although the saved process icon indicated it was unchecked.  From my recent testing on my new image (above), I have to conclude it must have been unchecked in image calibration and I was just lucky that that worked without dreaded grey calibration images for all 5 filters taken that night.   That is, in my experiments I ALWAYS see the hot pixels coming through when optimize is checked, but with it unchecked I only get the dreaded grey images ~half the time.  
Note 3: I note from the post below "Calibrating. At Wits' end" that I'm not the only person going nuts trying to avoid hot pixels in PI-calibrated light frames. I also can say that integrating with median rather than average does NOT fix this.  Maybe my tracking error is too small, but Dark Subtraction should remove these hot pixels from the calibrated light frames.  If that were the 'fix', it would imply that DITHERING was absolutely required for PI Calibration to work.  Perhaps that is the problem because none of my data is dithered.  

I have also sent a series of messages to Vicent's hotmail account containing links to a sample raw, master dark, master bias, master flat, calibrated light with Optimization Checked and calibrated light with Optimization unchecked demonstrating these problems.

[jeffweiss9]

...doing some further research here on ImageCalibration with optimize checked leading to hot pixels in the calibrated lights, I find this explanation for the #1 problem of the two problems described above that was given by Juan in the thread:

http://pixinsight.com/forum/index.php?topic=2471.0

He said there:

"It is true that dark frame optimization can lead to undercorrection of some bright hot pixels. However, the goal of dark frame optimization is not removing hot pixels better, but to minimize the noise induced by dark frame subtraction in the calibrated images. So even if some hot pixels survive with dark frame optimization enabled, the result will be less noisy. Hot pixels should be easy to remove with Carlos' DefectMap tool or Nikolay's CosmeticCorrection script, but less noise is always preferable in my opinion.

Why dark frame optimization can degrade hot pixel removal? Mainly due to nonlinear sensor behavior, and this means that each CCD is a different world. Our current dark frame optimization routine applies a linear scaling function to the dark frame. In an incoming version of ImageCalibration, we have implemented a multipoint dark frame optimization algorithm (basically a spline-based optimization curve) that hopefully will provide much better results. Multipoint dark frame optimization will compute separate scaling factors for different sections of the dynamic range. In this way hot pixels will be optimized in a different way from the rest of dark frame pixels."

   So that is some further information that the problem #1 is recognized by Juan et al and a hoped-for fix is in progress.  Since in my case it's a lot more than "some" hot pixels surviving but enough to result in a totally unusable image (before or after integration).   I guess I can try to use the DefectMap tool or CosmeticCorrection script to see if they can help avoid catastrophic levels of hot pixels coming through the current calibration scheme. Do they operate during calibration or only on the integrated image which would be a harder way to find them?

   I hope that while working on this next version of ImageCalibration that might actually work for me, an effort will also be made to prevent whatever it is that is causing the intermittant 'dreaded grey' calibration frames when optimize is unchecked.  That could be due to the same nonlinear sensor behavior, I suppose, but I didn't realize that the SBIG ST2000XM was so nonlinear.  Whatever the cause, it makes the software very nonrobust and data dependent.

   I look forward to some further responses from Juan or Vicent on this subject which, it seems to me, is rather a first order impediment for PI to fully function.  I remain a complete PI enthusiast and anxiously await some solutions/suggestions that might work for me in the short run or confirmation that I just need to be patient and wait for the next release.

   Thanks very much.
-Jeff

[jeffweiss9]

     For the record, I note that Nikolay's CosmeticCorrection script in utilities was able to take care of almost all of the hot pixels that were leaking through ImageCalibration after running that process with "optimize" checked in the Master Dark section to get best noise reduction.   So I do have a quite viable solution for calibrating with PI, at least until the next version of IC comes out with its nonlinear spline sensor response that is supposed to treat the hot pixels better.
     Thanks very much, Nikolay.
-Jeff

[jeffweiss9]

At this point I'm wondering if I have offended anyone or, more likely, everyone since I'm sure that there are plenty of people on the PixInsight Team and the community, in general, who could contribute some information on this hot pixel calibration problem that I've been reporting.  Yet I find I'm talking only to myself.  A bit wierd. 
-Jeff

[Juan Conejero]

Hi Jeff,

Of course you have not offended anyone, not at all. It's just that we are so busy that we can't deal with some complex problems posed on the forum. Vicent is right now flying to New York to give his workshop and lecture at NEAIC, which he has been preparing during the last weeks. I am working like crazy because PI 1.7 must hit the road before the end of the next week, etc.

Let's apply some systematic analysis to this problem. Facts that you have observed:

(a) With dark frame optimization enabled, you generally get good calibration results.

(b) With dark frame optimization disabled, master dark subtraction tends to destroy your lights.

So we have in essence two possible diagnostics:

(1) The ImageCalibration tool does not work correctly.

(2) Dark frame optimization (DFO) is fixing some critical problem with your master dark frames. This assertion is consistent with: (a) with DFO -> good results / (b) without DFO -> bad results.

Without further elements of judgement, let's assume, for now, that the probability of (1) being true is very small compared to the probability of (2) being true. The main reason for this assumption is that we use ImageCalibration on a regular basis with images of all kinds, and the obtained results are consistently excellent.

So if we concentrate on (2) —for now—, then one plausible cause that explains this problem is that your master dark frames are not bias-subtracted. When you enable dark frame optimization, the DFO routine converges to a solution where the optimal dark scaling factor is close to zero, as expected in this case, and hence it is preventing subtraction of the master dark from your lights.

Some questions:

- I assume you have generated all your master calibration frames with PixInsight. Is this true?

- How are you applying your master bias? Are your master dark frames bias-subtracted?

- When you calibrate with DFO enabled, what dark scaling factors are you getting on the console?

[jeffweiss9]

Hi, Juan-
 Thanks for responding and I don't want my impatience to get in the way of the next release or Vicent's talk at NEAF.

Sounds like a reasonable operating assumption. Here's what I can tell you:

- I assume you have generated all your master calibration frames with PixInsight. Is this true?
       Yes.

- How are you applying your master bias? Are your master dark frames bias-subtracted?
       I attach here the process icons that I use for all the steps. Yes, I believe the master darks are bias subtracted.  At one point, as a test, I used PixelMath to subtract the Master Bias from the Master Dark and used that as the Master Dark for one of the filters producing 'grey' images with optimize unchecked.  It did not work so I presumed that the bias had in fact already been subtracted and that was not the cause of the grey calibrated lights.

- When you calibrate with DFO enabled, what dark scaling factors are you getting on the console?
       I would get .5 to .65 (avg .6), 0.2 to 0.6 (avg 0.5), 0.08 to .35 (avg 0.25) to give three that I looked at. (The middle one gave grey calibrated frames when run with DFO disabled, while the other two didn't have a problem with that.

Again, I've got a work-around in place so I can be more patient now for when you can get back to this.
-Jeff

[Alejandro Tombolini]

Hola,

En este punto quizás pueda aportar algo más para analizar y ver que estoy haciendo mal.

En un set de imágenes algunas quedaron bien calibradas y otras no.

Las tomas son del mismo día, poca variación de temperatura, misma orientación del telescopio (no flip)
y se calibraron todas juntas con el mismo MasterBias y MasterDark. Canon Xsi

Imágenes nº 25 a 30 mal calibradas (solo el canal rojo) (+25ºC)
Imágenes nº 53 a 58 mal calibradas (solo el canal verde) (+21ºC)

El resto nº 13 a 24, 31 a 52 y 59 quedaron perfectamente calibradas.


Adjunto las impresiones de pantalla de dos imágenes consecutivas para cada caso.
(No considerar los pixeles rojo oscuro que aparecen por la compresión).


La imagen 52 quedó bien calibrada y la 53 mal solo en el canal verde.


La imagen 24 quedó bien calibrada y la 25 mal solo en el canal rojo.


Adjunto los datos de las fotos 52 y 53 (canal verde)

Dark scaling factors 52:
k0 = 1.350
k1 = 1.348
k2 = 1.343

Dark scaling factors 53:
k0 = 1.441
k1 = 0.464 ¿?
k2 = 1.454


Saludos.
Alejandro.




Hi,

At this point perhaps I could contribute with something more to analyze and see what I am doing wrong.

In a set of images some were well calibrated and other not.

The captures are of the same day, few change of temperature, the same orientation of the telescope (no flip)
and all together were calibrated with the same MasterBias and MasterDark. Canon Xsi

Images nº 25 to 30 bad calibration (only the red channel)(+25ºC)
Images nº 53 to 58 bad calibration (only the green channel) (+21ºC)

The rest nº 13 to 24, 31 to 52 and 59 they were perfectly calibrated.

See the print screen of two consecutive images for each case.
(Do not consider dark red pixels that appear due to compression)

Image 52 well calibrated and 53 bad only in green channel.

Image 24 well calibrated and 25 bad only in red channel.

See information of pictures 52 and 53 (green channel)

Dark scaling factors 52:
k0 = 1.350
k1 = 1.348
k2 = 1.343

Dark scaling factors 53:
k0 = 1.441
k1 = 0.464 ¿?
k2 = 1.454

Regards,
Alejandro.

[Cleon_Wells]

Alejandro, here's a screen shot of 1 raw sub after Calibration and DeBayer utility. Look at my process console and note how low my cal numbers are compared to yours. I don't know if there is a rule of thumb for the average cal numbers.
My MasterBias is iso100@1/4000-MasterDark is iso400@50sec-MasterFlat is iso400@1/800 and the Dark and Flat are not calibrated.
Have you looked at the differences between your sub 52 and 53 with out calibrating them?
Cleon

[Alejandro Tombolini]

Hi Cleon

Have you looked at the differences between your sub 52 and 53 with out calibrating them?
Cleon

I looked at the images 52 and 53 and they look like exactly the same picture, there are no visual differences.

I believe that it is relative to the temperature of the light.

I compared two images of other objects at the same ISO and the same time but different temperatures and  the Dark scaling factor changes considerably. 
Also it is clear looking at the images that one has much more noise than the other one.

Writing output file: F:/Mis Imágenes/2011_01_03 LT 02_38_32 ISO800 IMG_0400+25_0056_c.fit
Dark scaling factors:
k0 = 2.082
k1 = 2.081
k2 = 2.067

Writing output file: F:/Mis Imágenes/2010_07_17 LT 19_57_50 ISO800 IMG_0400+00_0006_c.fit
Dark scaling factors:
k0 = 0.229
k1 = 0.227
k2 = 0.221

The highlighted temperature is in ºC.

What I see like a constant is that the values are always similar between them, and in the previous case in which the green channel is not well calibrated, the value of k is very different.



Hola Cleon,

Miré las fotos 52 y 53 y parecen exactamente la misma foto, no hay diferencias a simple vista. 

Creo que es un tema relativo a la temperatura del los light.
 
Comparé dos fotos de otros objetos a la misma ISO y mismo tiempo de exposición pero a diferentes temperaturas y el Dark scaling factor varía considerablemente.
También es evidente mirando las fotos que una tiene mucho mas ruido que la otra. 

Writing output file: F:/Mis Imágenes/2011_01_03 LT 02_38_32 ISO800 IMG_0400+25_0056_c.fit
Dark scaling factors:
k0 = 2.082
k1 = 2.081
k2 = 2.067

Writing output file: F:/Mis Imágenes/2010_07_17 LT 19_57_50 ISO800 IMG_0400+00_0006_c.fit
Dark scaling factors:
k0 = 0.229
k1 = 0.227
k2 = 0.221

La temperatura resaltada está en ºC.

Lo que veo como un patrón es que los valores son siempre similares entre ellos, y en el caso anterior que el calibrado no queda bien en en canal verde, el valor del k correspondiente queda muy distinto a los otros.

Saludos.
Alejandro.

[Cleon_Wells]

Alijandro, what would happen if you create a new MasterDarkRGBXsec800C12. This would split the Temp Diff?

[netwolf]

Juan,

So if we dont have Bias frames and only have Darks, is it then best to disable Optimize?

Regards
Fahim