Fixing Bad Pixel Columns

[kerrywaz1]

Hello,

I have a 16803 CCD camera (Apogee F16M) that has some bad pixel columns. I tried calibrating these out with bias frames, but some are still there. Can someone please share a technique in PixInsight for getting rid of these artifacts?

Thanks,
Kerry

[pfile]

you can try CosmeticCorrection to repair bad/weak columns.

rob

[oldwexi]

Hi Kerry!
CosmeticCorrection helps,  but it is not perfect because it averages the bad column with the neighbour column.
This is necessary if the colum defect is really black or complete white, but it will  leave an
averaged  visible column defect which maybe will be reduced by stacking dithered images.

Most of the time the defect columns are only a tiny bit brighter or darker.
Here it is better not to use CC.
You create first a vertical line mask (or more) with Pixelmeth.
And than yous test on one masked image with curves the improvement which makes the line artefact complete
invisible because the real data are a little brightened or darkened!
Than i create an image container with alle images showing the artefact and
repair the column(s) in one step using the process container with the masked Curves processes.

If this sounds to complex, so provide me 3 calibrated images with these artefacts and i will produce
a projectfile as a show case. This will include the mask generation, the curves process(es) process container
and the image container usage.

Gerald

[Geoff]

That’s very clever Gerald!

[pfile]

another thing to try, as long as you have dithered, is to just write '1's over the bad columns using pixelmath and then the pixel rejection during imageintegration will take care of rejecting the columns. however, if some of the subs have equal X dithers then this may not work so well - you may end up with bright columns in the integrated result.

rob

[kerrywaz1]

Hi Gerald,

Your method sounds interesting. I sent you a private message with Dropbox links to three cal'd images with the bad columns.

Thanks,
Kerry

[oldwexi]

Hi Kerry!
Project for column correction without CC is already in my dropbox.
I sent the link to you just now.

Gerald

[ngc1535]

Interesting:

1. What method does CC employ to substitute pixels in columns (median, mean? of neighbors?)? I have always wondered....
2. OldWexi: I have simply used DefectMap and choosing a method (such as mean or median). Does your way have a significant improvement on say mean or median of DefectMap?

I guess a side-by-side comparison would be really great. If both methods make an "invisible" result- I am going to use the one with less steps.

-adam

[oldwexi]

Hi Adam!
CC and/or Defectmap is painting/inventing data in the bad columns by using the neighbouring pixels.
This is ok if the column defects are white or black only or with only one brightness.
But a line mask together with Curves increases or decreases the originnal data to a level which makes the column defect disappear but with "correct" data.

If Kerry allows me to show his images und the project i can make it visble to all.
As the used images belong to Kerry i cant go public with the project...

Gerald

[ngc1535]

Yes, I understand what you are doing and I agree it will work just fine- but I wonder if you need to make a curve adjustment for each column. I am guessing you would. Biases usually have large scale gradients (sometimes in the horizontal direction) so that an adjustment you make for one column would not be valid for another column.

As Rob mentions, with a truly random dither and enough frames- no data needs to be invented at all through subsequent rejection. The only quibble for me there is I think writing zeros instead of ones is better since large outliers tend to be on the high end.

And just a step further... you might want to comment on whether you would make your adjustment before or after creating a "superbias."

thanks,
-adam

[kolec]

Hi

I dont know how I can  create a vertical line mask (or more) with Pixelmath ?

kolec

[oldwexi]

One of the Pixelmath expression for creating a white line
in column 427 is:
1-d2seg(427,0,427,2048)
427,0  is the xy position of the line start
427,2048  is the xy position of the end of the line.
(In this case my image has a heigt of 2048)

If you want to create a shorter line for example
1-d2seg(427,350,427,1200)
This creates a vertical line from
xy 427,350  to yx 427,1200

xy 0,0 is in PI ususally the upper left corner.
But you can find the start- and endpositions of your future line with the ReadoutCursor

Gerald

[kolec]

Great thank you, Gerald!

kolec

[martin farmer]

Hello All,

I too would like to see the results of Gerald's workflow - if Gerald & Kerry would allow us to see.

Regards

Martin

[oldwexi]

Hi Martin!
I was thinking already about a little video which shows
how i use this technics. But it wont be ready before next weekend.

Generally
There are different cases where you use different solutions.
In my case i do get from the remote iTelescope T8 this data where the calibration,
because of old bias files, is not 100% correct.
Yes, I do dither but as i exposure very faint molecular clouds which i have to stretch strongly,
and despite dither the vertical stripes come out during extreme processing.

So the only way in my case is:
Check the column defects,
For each column i create a line with pixelMath,
Mask the bad image with one line mask and increase the intensity of this part
of the bad image with Curves or PixelMath  until the line is not visible anymore. (using a preview for finding the right value)
This i do for every bad column.
Out of the historyExplorer i copy/create a macro with all previous steps for all lines.
Than i create an ImageContainer and execute
the macro on the container on all images of a filter serie.

This is where i plan the video

There are other situations:
If i have a destroyed unrevoverable column i use the columns mask and replace
the column with the same but better data from another good/better image.
(needs to register the good image to the bad imag...)

Other situation:
If there are only a few images with bad columns, when dithering
the image integration process helps to let disappear these artefacts.
if not the  CC process can help but is the worst solution from data quality.

Gerald