Bad black columns

[alphascorpi]

Hi, I have shot some images of NGC253 with T32 iTelescope.
I have checked the LRGB Images in Pixinsight and found that there are so lot bad columns that a cosmetic correction with bad bixel map is not the way.
I have not count the lines but I think there are about 100 of them.

Is there another solution?
The image is calibrated (itel), Red Filter and AutoStretch with Pixinsight

Thanks for any help, Ed

[aworonow]

I have used T32, and had reasonable processing results. I have not used cosmetic correction (although I would recommend starting there, even if there are 100 bad columns). Ultimately, CC + dithering generally solves the issue and produces stacks that are quire nice. Of course you need enough subs so that whatever rejection criteria you use, it can discard the bad pixel columns...try different rejection criteria at different degrees of aggressiveness.
However, if your attached image shows your target, than you the scale of the pixels (or binned pixels?) is not sufficient to yield a quality image, I would suspect.

[oldwexi]

For this issue CosmeticCorrection is not the right solution!
As it averages the bad column with the 2 neighbouring columns. This
will be visible in the final stack when you need to stretch a little stronger finally.

I had the same experience on T8 with these vertical "a very little darker" (not black) lines.
The difference in brightness to the neighbouring columns is around 0.01 to 0.001.

So my solution, which recovers(!) the original data is:
Create for each darker column with PixelMath a one pixel vertical line.
Use this line as mask and increase the brightness in one raw image carefully.
If you have covered all the lines in the one image you can process and correct all images
with imageContainer and Process Container.

Cosmetic correction should only be used when the column is near 0 = black or near 1 = white.

Solving the issue at the root -  try to get new darks and bias for this telecope. This could eliminate
painful corrections and create images without these artefacts.

Gerald

[Geoff]

Gerald, I have some questions about your solution, which I have used with some success for a single really bad column.
1. If the subs differ in brightness (thin cloud, different nights or whatever) then the curves adjustments will differ from sub to sub, so using the containers won’t give consistent results.
2. If there is more than one problem column the curves adjustments needed to correct each column will be different, so I assume you would correct one column using the image and process containers, then repeat the whole process again for the next column (and the next, and the next,...)

Do you have a way to shorten this process? If not, then I guess that in the pursuit of excellence we just have to follow through.

[alphascorpi]

Hi Aworonow
Thanks for your help.
Enough subs? The subs are:
L (1x1) 8x120s / RGB each (2x2) 8x120s.
Perhaps this is too little for this case.
Dithering is done, but I think with less than 20 images dither dont work.
I will try with different rejection criteria.

[alphascorpi]

Hi Gerald
Thanks for your help.
I will try your method, but with 100 or more columns is this not so "amusing" 
"Create for each darker columns....": Can PM this automatically, like a script?
I have the latest bias and darks of T32 and have shot 100 new bias.

[dave_galera]

Ed, No sure if this may help
https://pixinsight.com/forum/index.php?topic=13595

[oldwexi]

Hi Gerald
Thanks for your help.
I will try your method, but with 100 or more columns is this not so "amusing" 
"Create for each darker columns....": Can PM this automatically, like a script?
I have the latest bias and darks of T32 and have shot 100 new bias.

Hi alphascorpi!
You are right with so many defect columns its a lot of work.

But, CosmeticCorrection needs also to find the column x position and to enter it into the dialog box.
Means, needs one Move with the ReadoutCursor to get the x value and entering the value into CC.
Using the PixelMath line creation you have to change also only the column value.
Execute it and save the image. after having created all line images i save the whole as project.
So when i get the next images to "decolumn" i load the project and have all the columns ready to adapt.

So,  there is not much difference in work but a lot in the result as CC destroys the real data in the columns
and PM plus mini stretch gets the real data correct!
All these mini stretches i save in a process container and run it against the image container of the new images.

Yes, i adapt the stretch parameter for each night separately. But thats not a great deal.

Usually this works.  I only use good data on iTelescope. You dont have to use bad data (haze, cloudy etc.)
you can reject them with iTelescope and get refund. So, with good data most of the time
the work done once helps also with the other ones.

My last final image consists of 108 raw images "corrected" the way i describe.
The image before, the SMC, consists out of 352 single raw images treated the same way.

Gerald

P.S.: for these column troubles dither does not help (i dither every image), when you need to stretch the final
        image very much you will see these troubles if you use CC or nothing.

[dave_galera]

Hi Ed,

The script that Vincent developed writes a CosmeticCorrection-compatible defect table to disk. You can then use this table in CosmeticCorrection, but it better if you use this table to correct the defects with LinearPatternSubtraction.

LinearDefectDetection script:

Script to detect defective columns or rows in a reference image. It can detect entire or partial defective columns or rows. The best approach to detect these line defects is to use an integrated image of non-aligned images. This way, we increase the signal to noise ratio of the lines, which is crucial to be able to detect them above the noise floor. This wouldn't be possible in a single subframe because most of the defects are well below the noise of the image and they only appear as diffuse lines in the integrated image. So, first you need to generate this kind of master image. The script will be always applied to the active image. It takes its time to finish since the algorithms are complex, but the time is well worth and it's a one-time process.

Hope this is helpful.

[alphascorpi]

Hi Dave
Thanks for the tip with this script.
I think this is not a script in PI 1.8.6.
Can you help me with a link to get "LinearDefectDetection"?

[dave_galera]

Hi Ed,
The script is not included in any version of PI it has to be downloaded as required.

The download link is noted in the post https://pixinsight.com/forum/index.php?topic=13595, however, I have attached the file PatternSubtraction.zip which has all the scripts in it