dark frames from raw look different PI and LightRoom

[troypiggo]

I have been playing around experimenting with biases, darks, flats and calibration.  There is something I discovered by accident that I can't explain and wondering if you guys can help me out.

Taking the same raw file from my 30D DSLR, it looks like this if I use LightRoom to export to a full-sized jpg, no editing:
http://dl.dropbox.com/u/53563365/30D/dark%20uncal%20unstretched%20LR.jpg

Same raw file, opened, debayered, and then saved as a full-sized jpg in PixInsight, no editing:
http://dl.dropbox.com/u/53563365/30D/dark%20uncal%20unstretched%20PI.jpg

See the PI one is much darker?  If I stretch it, it looks like this:
http://dl.dropbox.com/u/53563365/30D/Screen%20Shot%202012-03-20%20at%207.59.25%20PM.jpg

But if I use the original raw file, debayer it, then stretch it, this is how it looks:
http://dl.dropbox.com/u/53563365/30D/Screen%20Shot%202012-03-20%20at%208.00.34%20PM.jpg
Got some more colour in there, and looks a bit closer to the LR one.

Can anyone explain what's going on?  They're not FITS files, they were captured in-camera CR2 files.  So not sure if pedestals are there?

[NKV]

8bit problem.
Never use jpg before stretch.

[troypiggo]

Ok. I see.  I saved the file as both a 32 bit TIFF and an 8 bit TIFF, then opened them and stretched them in PI.  Can see the difference there.

I still don't see the difference between the untouched, unstretched PI image and LR image.   

[georg.viehoever]

Are you sure that LR does not do any internal rescaling/stretching? Many photo applications do.
Georg

[troypiggo]

Not sure, mate.  I specifically reset any development adjustments, so if there's any stretching or data manipulation, it's out of the user's hands and done internally as far as I can tell.

[ruediger]

Can anyone explain what's going on?

Probably because of:
1. Lightroom automatically removes hot pixel and there are lots in a dark frame
2. You used PixInsight's VNG debayering even though there are no cross color plane dependencies in dark frames. How does it look with bilinear debayering?
Rüdiger

[troypiggo]

Can anyone explain what's going on?

Probably because of:
1. Lightroom automatically removes hot pixel and there are lots in a dark frame
2. You used PixInsight's VNG debayering even though there are no cross color plane dependencies in dark frames. How does it look with bilinear debayering?
Rüdiger

At 100% you can notice subtle differences between the VNG and bilinear, but at zoomed to fit you can't really tell any difference.  Both methods are much closer to each other than they are to the LR image.

Side question: For debayering DSLR raws in PI, is VNG the recommended algorithm?

LR does try to do some noise reduction, but there's so much there because the shot is a 10min sub at ISO1600, that plenty of hot pixels do remain.

[ruediger]

Side question: For debayering DSLR raws in PI, is VNG the recommended algorithm?

I'd say: "it depends". For low SNR frames you can still use bilinear debayering with good results. VNG tends to introduce isolated red pixels in star cores or at star edges (see screenshot), that might not fully disappear during stacking. I only use VNG if it helps to overcome the feared "no convergence in MRS noise evaluation" in ImageIntegration. But VNG is no guarantee that you will never see this message.

Rüdiger

[Nocturnal]

I'm glad my old bilinear debayer still has its uses

[zvrastil]

Hi Ruediger,

these red pixels look more like a bug in VNG implementation to me. Would it be possible to get some image that exhibits this problem? Cropped tiff is ok as long as its top-left pixel is of the same color as original image (and thus cropped image has same bayer pattern).

thanks, Zbynek

[ruediger]

these red pixels look more like a bug in VNG implementation to me.

Probably not, because I remember that I saw the same patterns in other demosaicking algorithms I implemented last year for testing purposes. On a quick test, I only found the shown patterns on 135mm focal length shots, but not on 400mm shots. Now what might be the influence of focal length on demosaicing quality? 
The 135mm lens is a) used stopped down, thus producing diffraction spikes and b) very nervous to focus and therefore easily produces red halos around bright stars. If the demosaicing algorithms relies on assumptions like smooth color transitions that are well aligned on all color planes, maybe it will fail.
Give me some time to collect more information and find good examples to maybe enhance demosaicing quality (e.g. by combining results of different algorithms).
Rüdiger

[georg.viehoever]

I am not sure if this is the same problem you are discussing:
I usually find "red pixels" in my wide fields because there is not enough dithering there to get rid of hot pixels. Long focal length images get "automatically" dithered by my messy mount, short focal length don't. This is why the pixel rejection algorithms gave a much harder time to get rid of those hot pixels (see screenshot). The hot pixels themselves are also there for long focal lengths...but pixel rejection can successfully remove them during stacking.

I am hoping to get a DefectMap/CosmeticCorrection tool that can handle OSC/bayered images from the PI team sooner or later...

Georg

[ruediger]

I am not sure if this is the same problem you are discussing:
I usually find "red pixels" in my wide fields because there is not enough dithering there to get rid of hot pixels.

But hot pixels must show up also in the bilinear debayering result, whereas in the shown example there is no sign of them.
Rüdiger

Edit: In my screenshot I debayered a single calibrated lightrame with two different algorithms. The red pixels must be "invented" red values on the G/B-locations of the bayer matrix. I'll check again later when I've access to the source image

[georg.viehoever]

...But hot pixels must show up also in the bilinear debayering result, whereas in the shown example there is no sign of them.

I have the suspicion that the VNG algorithm may be more sensitive to hot pixels, or to sharp brightness changes as in the vicinity of stars. Bilinear seems to be more "robust". However, I did not look into this in detail.

Georg

[topboxman]

Nebulosity does a good job of removing bad pixels called Bad Pixel Mapping. Nebulosity's new version 3.0 now has several methods of deBayering including VNG and bilinear. It also has other methods like PPG (patterned pixel grouping) and AHD (Adaptive Homogeneity). I do not yet have Nebulosity version 3.0 but I have version 2.5.3 and does a good job of deBayering.

Peter