Problem with CosmeticCorrection hot pixel removal

[sharkmelley]

Hi all,

I'm totally new to PixInsight - though quite experienced with DSLR astro-image processing.  I'm currently playing with the trial version of PI having gone through Harry Page's tutorials.

I've noticed that curiously the Image Calibration process is missing the ability to perform hot pixel removal - maybe I'm missing something? I want to deal with hot pixels before debaying otherwise they become tiny coloured crosses after debaying. 

Anyway I discovered CosmeticCorrection v1.6.1 in the Utilities menu and decided to use this.   However, the whole image that I'm calibrating ends up being convolved with a kind of square operator!  It's as if the CosmeticCorrection process decides the whole image consists of hot (or dead) pixels.

The details of the CosmeticCorrection I'm performing is as follows:
The images I'm correcting are single channel TIFs in i16 Gray format (batch converted from Canon CR2 raw files)
The MasterDark is a single channel TIF in i16 Gray format - hot pixels are value 1, everywhere else is precisely 0 (at least that's how the cursor reports the values when I load the TIF in PI (without using a Screen Transfer Function).
Dead pixel threshold is set to 0.0 which gives a dead pixel count of 0 in the dialog
Hot pixel threshold is set to 1.0 which gives a hot pixel count of 157 in the dialog
"Amount" is set to 1.0 and CFA is ticked.

My guess is that the whole image is being treated as hot or dead pixels.

I suppose I could get around this by doing some kind of PixelMath instead but it wouldn't be that easy.

Any help appreciated!

Mark

[Carlos Milovic]

Or, you may use the DefectMap process. You just have to invert the image signalling your hot pixels (zero is a bad pixel, white is a good pixel). Juan should upload very soon a new version that deals with CFA, but you may try the result as it is right now.

[NKV]

Hi Mark!
I can't reproduce your problem with on my images.
Upload your MasterDark and one Light image anywhere. I will test it.

Also there are new version of CosmeticCorrection script there and CosmeticCorrection module there.

Best regards,
Nikolay.

[sharkmelley]

Hi Nikolay,

Thanks.

I'll upload them tonight.

Mark

[sharkmelley]

Hi Nikolay,

I tried it again - this time with FITS files - I experienced the same problem.  I did also try inverting the hot pixel map to produce a defect map but got the same result.

Here are the sample files:
http://www.markshelley.co.uk/webdisk/light.fit
http://www.markshelley.co.uk/webdisk/hotpixels.fit

The result is here:
http://www.markshelley.co.uk/webdisk/corrected.fit

The parameters I used to generate the "corrected" file were as follows:
Dead Pixels:  Count=0  Threshold=0.0
Hot Pixels:     Count=1  Threshold=1.0
Amount=1.0
CFA: ticked

Mark

[Carlos Milovic]

I just tried the DefectMap process, and it yields correct results. See it under the ImageCalibration category.

[sharkmelley]

The DefectMap process works for me as well - in the sense that it does remove the hot pixel.  But it does not do the right thing for a raw DSLR (or one shot colour) image.  For a DSLR image, it is important that a Red hot pixel is corrected by interpolating between other Red pixels.  The same argument for Green and Blue.  Otherwise, artifacts will be produced when debayering.

That's why the CosmeticCorrection process looked attractive - because it is designed to apply the correction by interpolating between pixels of the appropriate colour.  But for some reason I seem to have problems running it.

Mark

[Carlos Milovic]

Yes, I know... Wait a couple of days for the new ImageCalibration module, with the CFA option in it. I've compiled it for Fedora 16 x64 exclusively.

[pfile]

just a question - shouldn't the hot pixels show up in your dark frame and thus be calibrated away?

or are those technically stuck pixels, and hot pixels are just too sensitive such that they don't show up on a dark but immediately saturate when exposed to visible photons?

[Carlos Milovic]

Yes, they show up in darks, but are non linear. This means, pixel intensities in the darks and the lights are not correlated, specially if you had to rescale the darks.

[NKV]

Hi Nikolay,

I tried it again - this time with FITS files - I experienced the same problem.  I did also try inverting the hot pixel map to produce a defect map but got the same result.

Here are the sample files:
http://www.markshelley.co.uk/webdisk/light.fit
http://www.markshelley.co.uk/webdisk/hotpixels.fit

The result is here:
http://www.markshelley.co.uk/webdisk/corrected.fit

The parameters I used to generate the "corrected" file were as follows:
Dead Pixels:  Count=0  Threshold=0.0
Hot Pixels:     Count=1  Threshold=1.0
Amount=1.0
CFA: ticked

Mark

Hi Mark,
I tested you fits on old 1.6.1 version and on new 1.7.3 in Linux and window PI 1.7.5.0779
I don't see the problem. Sorry...
Hot pixel removed absolutely correctly.

Mark, can you try script version 1.7.3 ?
Fast way to try: Ctrl+Alt+E, Ctrl+N, paste code from forum. Press F9 to run.
No need MasterDark, just select Light and tick CFA. The script must detect the hot pixel automatically.

[sharkmelley]

Hi Nikolay,

I tried 1.7.3 with the same version of PI as you:  PI 1.7.5.0779   Unfortunately I got exactly the same problem with or without Auto Detect.

I guess the main difference is that I am running on Windows XP  which I know is not officially supported.

I may try to debug it - I'm an experienced programmer but never used Java Script before.   I haven't found the debugging tools (if there are any) so I'll have to write variables to the console or a log file.

Mark

[sharkmelley]

Hi Nikolay,

I've made a bit of progress on this - I don't know how to single step through the script so I peppered the script with console.writeln!

The problem only happens when I run the script as a process from the desktop (I dragged it there using the blue triangle).

Under these conditions, HotThreshold has a sensible value before the statement
     this.controlEnable();
but is zero directly afterwards.

So something is resetting HotThreshold - I think it is something to do with:
   HotCountControl.onValueUpdated(HotCount);

Trouble is, I don't understand enough about what is going on to get any further.

Hope this helps!

Mark

[Nocturnal]

I haven't really dug into javascript coding with PI either for that same reason (no debugger). PCL coding is a lot harder than javascript but at least you can run your code in the debugger which helps a great deal. Luckily Juan has planned javascript debug support for a future PI version.

[NKV]

The problem only happens when I run the script as a process from the desktop (I dragged it there using the blue triangle).

I catched the bug. Thank you and sorry for troubles.
Try Bug fixed version.
Best regards,
Nikolay.