Need help eliminating hot pixels - ASI2600MC

pfile

PTeam Member
Nov 23, 2009
5,603
102
i guess one thing you could do is zoom in on a hot pixel in a sub where you performed CC and one where you didn't and see what the values are like using the readout cursor.

to eliminate any ambiguities you might calibrate without the flat; not sure if both subs would get exactly the same flat scaling value. if they are different, then the hot pixel values would be different even if they were the same in each unflattened sub.
 

joelshort

Well-known member
Dec 21, 2012
317
4
buckeyestargazer.net
i guess one thing you could do is zoom in on a hot pixel in a sub where you performed CC and one where you didn't and see what the values are like using the readout cursor.

to eliminate any ambiguities you might calibrate without the flat; not sure if both subs would get exactly the same flat scaling value. if they are different, then the hot pixel values would be different even if they were the same in each unflattened sub.
I tried to do that (compare CC with non-CC subs) but I couldn't tell any visual difference between the two, but I also didn't use the readout cursor either so it's possible there was some small difference. Also tried calibrating without the flat and it didn't seem to make any difference.

Since I had success not using CC I finished the project and got rid of the raw files so at this point I can't go back and check things.
 

bulrichl

PTeam Member
Nov 2, 2016
817
56
La Palma, Canary Islands
I agree that last sentence is nonsense, as it was meant to be. Obviously I know that CC can't be adding in hot pixels.

But you are also making an assumption that is not true, as I stated in the original post and as I have tested multiple times. I AM using pixel rejection during integration (Linear fit to be precise).

My workflow is
Calibrate Lights with master Bias/Dark/Flat
CosmeticCorrection
Debayer
StarAlignment
ImageIntegration (LinearFit rejection + large scale pixel rejection)

If I do the above the resulting image has those hot pixel streaks. If I do the above but simple eliminate CC I do not get the hot pixel streaks in the integrated image. I don't have an explanation for that. It makes no sense.

In any event I have moved on, since I found a solution to the problem.
OK. If I understand it correctly that means that CosmeticCorrection with option 'CFA' does something strange. Unfortunately I cannot investigate this issue further without having the whole dataset at my disposal. I'm glad that it works for you by simply omitting CC in the above workflow.

Bernd
 

bulrichl

PTeam Member
Nov 2, 2016
817
56
La Palma, Canary Islands
That's not a problem, fortunately I have a fast internet access. I will download it and tomorrow take a look at the data. Thank you very much.

Bernd
P.S.: OK, downloaded.
 
Last edited:

bulrichl

PTeam Member
Nov 2, 2016
817
56
La Palma, Canary Islands
Here are my findings:

1) No CosmeticCorrection
When the calibrated light frames are debayered and then registered with default parameters, the alignment is on the hot pixels. This can be fixed by setting 'Hot pixel removal' from 1 (the default) to 2 in StarAlignment. In ImageIntegration I tried several rejection algorithms in order to remove the hot pixels. In the order of decreasing effectiveness:

Winsorized Sigma Clipping (Sigma_low=4.0, Sigma_high=2.5; slightly more effective than Sigma Clipping, but the fraction of rejected values was higher as well)
Sigma Clipping (Sigma_low=4.0, Sigma_high=2.5)
ESD (default parameters)
Linear Fit (lf_low=5.0, lf_high=4.0)

So winsorized Sigma or Sigma Clipping seemed suited best. However, even with the shown settings, slight remnants of hot pixels remained detectable in the integration despite a fraction of rejected values in the range 1 - 1.5 %.


2) CosmeticCorrection applied on CFA data
As stated in this thread, applying CosmeticCorrection (Auto detect, Hot sigma) on the calibrated light frames in CFA format (option CFA enabled) yielded strange results. I gradually lowered Hot Sigma until there was erosion of the stars detectable. At a Hot sigma of 1.6 this was just not yet the case. This rendered possible that StarAlignment worked OK with 'Hot pixel removal' set to the default value of 1. However, applying rejection algorithm ESD with default parameters showed walking noise clearly.


3) SplitCFA, then CosmeticCorrection applied on the monochrome images
The workflow SplitCFA, application of CosmeticCorrection to the monochrome images, register, integration to R, G and B monochrome images, drizzle integration of the separate channels and combination of the R,G and B channels with ChannelCombination led to the best results. Using rejection algorithm ESD with default parameters resulted in a very small fraction of rejected values (around 0.10 - 0.12 % for both rejected low and rejected high values), and the hot pixels were removed completely.


So my conclusion is that the appropriate application of CosmeticCorrection is the most efficient way (in terms of preserving the highest SNR) to avoid the streaks in the integration. Unfortunately, currently applying CosmeticCorrection on the calibrated light frames in CFA format did not work satisfactory with these data. The approach using of splitting the CFA data to monochrome images worked fine, but is rather tedious. With these data, I was not able to reproduce your finding that omitting CosmeticCorrection leads to an integration devoid of walking noise. To my opinion this is only possible when pixel rejection is applied rather aggressively, and that will lower the SNR. The linear fit algorithm with parameters lfit_low=4.0, lfit_high=2.0 (your settings in post #36) lead to the following total fractions (low + high) of rejected values: R: 5.5 %, G: 5.7 %, B: 5.3 %.

Bernd
 
  • Like
Reactions: pfile

joelshort

Well-known member
Dec 21, 2012
317
4
buckeyestargazer.net
So my conclusion is that the appropriate application of CosmeticCorrection is the most efficient way (in terms of preserving the highest SNR) to avoid the streaks in the integration.
Thanks Bernd, for taking the time to look at this.
But I'm struggling to understand your comments. I need the "for dummies" version. What is the "appropriate application of CosmeticCorrection"? I couldn't find any way to utilize CC in a way that didn't result in the hot pixel walking noise.
 

bulrichl

PTeam Member
Nov 2, 2016
817
56
La Palma, Canary Islands
I am sorry, currently I suggest the (tedious) detour over SplitCFA, CC of the individual channels and then use Rob's proposal. I hope that Juan finds the time to take a look at the issue when CC is applied to CFA data. Of course you can go the way lowering the threshold for pixel rejection in ImageIntegration. However, you will loose about 5 % of your data (after all, in this case more than 3 out of 63 subframes).

Bernd
 
  • Like
Reactions: joelshort

blueridgeDSIA

Active member
Apr 6, 2017
30
1
If you were to take an image cosmetically corrected after splitting the channels so no hot pixels remain, and then register the troublesome data to that image, would image integration reject hot pixels then?

Im also wondering if you have simply run multiple iterations of integration in a small roi box over an area in an integrated image with hot pixels? If using linear fit clipping you could just kerp turning up rejection high and re integrating until the hot pixels disappear. Small roi boxes stack really fast so when you find the right setting turn off roi snd stack the whole image.
 

sn2006gy

Member
Jun 10, 2020
22
2
The process described above (SplitCFA; CosmeticCorrect each image ; MergeCFA) - isn't that what CosmeticCorrect does if you set the CFA check box?
That's what the tooltip says.

BTW, just noticed that cosmetic correction doesn't have any documentation. What exactly does this magic tool do?