Hot Pixels not rejected in DrizzleIntegration

[mstriebeck]

I am trying to use the DrizzleIntegration script and always end up with non-rejected hot pixels.

I calibrate, register (using the BatchPreprocessing Script) and stack (using Linear Fit in ImageIntegration) my 40 frames. The resulting image doesn't have any remaining hot pixels in it.

When I then use DrizzleIntegration with the drizzle files from the ImageIntegration process, the resulting image has some hot pixels in it (I use DrizzleIntegration with the default values).

I tried using DrizzleIntegration with "Enable pixel rejection" unchecked. Then I have way more hot pixels. So, it does reject - just not enough.

Here are the two images:

ImageIntegration result: https://drive.google.com/file/d/0B0KcEqw-v71VY2g0QXFzeVB2X0U/view?usp=sharing (no hot pixels)
DrizzleIntegration result: https://drive.google.com/file/d/0B0KcEqw-v71VQ1k5cDV3Y2Y1OUE/view?usp=sharing (has hot pixels)

Any ideas what I might be doing wrong?

Thanks
      Mark

[vicent_peris]

Hi Mark,

When you use drizzle integration you usually increase resolution, so you average less pixels into a single one.

Let's supose that you have an image set of 12 subframes. Now, imagine that you have a faint hot pixel in one of the images, having an intensity of 120 ADUs, and that hot pixel is not filtered out by the rejection algorithm. Because you are averaging 12 images, the integrated pixel will increase only 10 ADUs due to the hot pixel, and it may result faint enough to become unnoticed.

If you apply X2 drizzle, you are only averaging (roughly speaking, since it's a completely different algorithm) 4 pixels; then, the residual hot pixel will have 30 ADUs when you generate the master light through drizzle. It may then become very noticeable.

This happens partially because we need a non-linear stretching to be able to look at the image. Something similar happens with my dark scaling algorithm: It usually has less than 1? error in the scaling factor but, due to the non-linear stretching of STF, a residual of 200 ADUs can look similar in brightness to a 20,000 ADU hot pixel.

I'm out of home writing from my phone, but I have an idea in mind that could work for you. When you run ImageIntegration to apply the outlier rejection to the drizzle files, select the maximum value instead of an average. This will enhance the visibility of any residual hot pixel and will let you better tune the rejection.

Next year I plan to start teaching drizzle in my workshops, even in the beginner level, since it's really the way to go in astrophotography.

Best regards,
Vicent.

[mstriebeck]

Thanks a lot Vicent for the elaborate explanation - this makes a lot of sense. I was trying this morning the opposite (setting the rejection value lower) - with the same results. So, I'll try now to max it.

Just to be clear: You mean the Sigma-high value, right? The range-high value is already almost maxed (0.98)

Will give it a try and let you know.

     Mark

[vicent_peris]

Hi Mark,

You have several options to combine an image set: the minimum, maximum, average or median value of each pixel set. I'm sorry but I'm writing from my phone and don't remember the exact names of the parameters in the tool. The default value is the average, which gives the highest signal to noise ratio. Instead of the average, if you calculate the maximum value for each pixel set, the residual hot pixels won't be averaged, so it will be easier to check the rejection parameters.

You should be more aggressive rejecting hot pixels. This means that you should lower the Sigma high value.

It's very difficult to teach this through the forum. This delicate process should be taught face to face because the key is to learn how to look at the rejection maps. Juan was brilliant creating the rejection maps, they are one of the most important tools in PixInsight right now.

Best regards,
Vicent.

[mstriebeck]

Hi Vicent,

Thanks for the clarification. I thought you meant to set the rejection parameters all the way to their max value - sorry, my mistake. I did what you said: used my normal parameters but used "maximum" as the Combination for the images. Unfortunately, the result is exactly the same: the hot pixels are still in DrizzleIntegrated image.

     Mark

[vicent_peris]

Hi Mark,

Well, I would need to check what's happening with your data set...


V.

[mstriebeck]

Hi Vicent,

OK, here are my Lights: https://drive.google.com/drive/folders/0B0KcEqw-v71VM0R4QWlpLWxqSE0?usp=sharing
here are the weighted lights: https://drive.google.com/drive/folders/0B0KcEqw-v71VczFZTHB5bFUteTQ?usp=sharing (keyword SSWEIGHT)
and here my Calibration Frames: https://drive.google.com/drive/folders/0B0KcEqw-v71VN2xxTkNOWjNFcHc?usp=sharing

Or let me know if it's easier to make zip files out of these.

Thanks a lot for looking

     Mark

[rdryfoos]

Hi Mark,

When you use drizzle integration you usually increase resolution, so you average less pixels into a single one.

Let's supose that you have an image set of 12 subframes. Now, imagine that you have a faint hot pixel in one of the images, having an intensity of 120 ADUs, and that hot pixel is not filtered out by the rejection algorithm. Because you are averaging 12 images, the integrated pixel will increase only 10 ADUs due to the hot pixel, and it may result faint enough to become unnoticed.

If you apply X2 drizzle, you are only averaging (roughly speaking, since it's a completely different algorithm) 4 pixels; then, the residual hot pixel will have 30 ADUs when you generate the master light through drizzle. It may then become very noticeable.

This happens partially because we need a non-linear stretching to be able to look at the image. Something similar happens with my dark scaling algorithm: It usually has less than 1? error in the scaling factor but, due to the non-linear stretching of STF, a residual of 200 ADUs can look similar in brightness to a 20,000 ADU hot pixel.

I'm out of home writing from my phone, but I have an idea in mind that could work for you. When you run ImageIntegration to apply the outlier rejection to the drizzle files, select the maximum value instead of an average. This will enhance the visibility of any residual hot pixel and will let you better tune the rejection.

Next year I plan to start teaching drizzle in my workshops, even in the beginner level, since it's really the way to go in astrophotography.

Best regards,
Vicent.

  Vicent, is drizzle better even if you can't use Mure Denoise?  I find MD to be amazing.  But it does not work with drizzle (according to the literature).  is there a way around this?

Thanks,

Rodd

[vicent_peris]

Hi Mark,

Thank you for the data set. There is a confirmed bug that affects the outlier rejection but it's already solved, waiting to be released in the upcoming days. Anyway, I'll take a look at your data set.


Thanks again,
Vicent.

[vicent_peris]

Hi Mark,

I don't get any hot pixel with your data set running DrizzleIntegration (I'm still using the version before the last update). So let's review the steps:

- I run BPP with your lights and your master calibration files. Dark optimization is active and I choose Linear Fit Clipping for the automatic master light generation. I also check the "Generate drizzle files" option.

- Now, I go to ImageIntegration. Here I load the lights and then I load their associated drizzle files. Everything is set to default and I only have to activate the pixel rejection. I set it up with the Linear Fit Clipping algorithm and fine tune the Sigma Low and High until I get the desired result. I used 4 for Sigma High and 3 for Sigma Low. Remember, this step is necessary to reject the pixels that should not be integrated with DrizzleIntegration. This is a unique feature of our drizzle tool.

- Now I run DrizzleIntegration with default parameters. The image is completely clean from artifacts. I attach a small crop.


Best regards,
Vicent.

[mstriebeck]

Thanks a lot for all the work here Vicent,

I just tried this again exactly as you described and I still have hot pixels. So, maybe it is the latest update...

A few questions:

• Is there a way to revert DrizzleIntegration to the previous version?
• From your description it sounds as if you do ImageIntegration twice (first in BPP and then in ImageIntegration). Did I just misread this? (you said that you set Linear Fit Clipping in BPP)
• How did you come up with the Sigma Low and High values (btw, I'm sure you meant Linear Fit Low and Linear Fit High, correct?) Do they have to be set such that hot/cold pixels and other outliers aren't visible in the ImageIntegrated image or do I have to run both, ImageIntegration and DrizzleIntegration to set them)?

Again, thanks for all the help here!!!

     Mark

[vicent_peris]

Hi Mark,

There is a bug in DrizzleIntegration when you have images rotated by 180º, as in this data set. So the bad rejection is simply that bug.

Is there a way to revert DrizzleIntegration to the previous version?

Please be patient, a new version of the tool is going to be released very soon.

From your description it sounds as if you do ImageIntegration twice (first in BPP and then in ImageIntegration). Did I just misread this? (you said that you set Linear Fit Clipping in BPP)

You're right. The ImageIntegration in the BPP script gives only a "reference" result. It is always strongly recommended to run manually the image integration process; poor settings in this process could lead to heavy artifacts in your integrated image.

On the other side, if you're doing drizzle, you shoul *always* run manually ImageIntegration with the aligned images and the drizzle files to update those with the pixels rejection data.

How did you come up with the Sigma Low and High values (btw, I'm sure you meant Linear Fit Low and Linear Fit High, correct?) Do they have to be set such that hot/cold pixels and other outliers aren't visible in the ImageIntegrated image or do I have to run both, ImageIntegration and DrizzleIntegration to set them)?

Yes, Linear Fit Low/Hight. It's very difficult to teach this through the forum because every single data set is very different when it comes to pixel rejection. This article may be of some help though.


Best regards,
Vicent.


Again, thanks for all the help here!!!

     Mark
[/quote]

[Juan Conejero]

Hi Mark, Vicent,

Please confirm that this problem has been fixed with the update to the ImageIntegration module that I've just released for all supported platforms. Sorry for the trouble.

[mstriebeck]

YES!

I just ran the same process as before - and NO hot pixels!

Thanks!

     Mark

[vicent_peris]

Hi Mark,

We just made an important improvement in this tool. Give a try to the different drops available. In your data set, the FWHM goes from 3.6 to 3.4 pixels by using the Gaussian drop.


V.