Author Topic: Fixing Bad Pixel Columns  (Read 1551 times)

Offline kerrywaz1

  • Member
  • *
  • Posts: 68
    • View Profile
    • Spirit Bear Observatory
Fixing Bad Pixel Columns
« on: 2018 September 13 15:53:44 »
Hello,

I have a 16803 CCD camera (Apogee F16M) that has some bad pixel columns. I tried calibrating these out with bias frames, but some are still there. Can someone please share a technique in PixInsight for getting rid of these artifacts?

Thanks,
Kerry
Deep Sky Instruments RC14C
FLI PL16803 w/MOAG + StarlightXpress Ultrastar
Pyxis 3"
Astro Physics AP1200
Tak FSQ-106EDXII
Apogee Alta F16M
Hap Griffin modified Canon EOS 5D Mark II
PixInsight, MaxIm DL, PhotoShop CS5

Offline pfile

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 4133
    • View Profile
Re: Fixing Bad Pixel Columns
« Reply #1 on: 2018 September 13 17:45:10 »
you can try CosmeticCorrection to repair bad/weak columns.

rob

Offline oldwexi

  • PixInsight Guru
  • ****
  • Posts: 605
    • View Profile
    • Astronomy Pages G.W.
Re: Fixing Bad Pixel Columns
« Reply #2 on: 2018 September 14 11:18:34 »
Hi Kerry!
CosmeticCorrection helps,  but it is not perfect because it averages the bad column with the neighbour column.
This is necessary if the colum defect is really black or complete white, but it will  leave an
averaged  visible column defect which maybe will be reduced by stacking dithered images.

Most of the time the defect columns are only a tiny bit brighter or darker.
Here it is better not to use CC.
You create first a vertical line mask (or more) with Pixelmeth.
And than yous test on one masked image with curves the improvement which makes the line artefact complete
invisible because the real data are a little brightened or darkened!
Than i create an image container with alle images showing the artefact and
repair the column(s) in one step using the process container with the masked Curves processes.

If this sounds to complex, so provide me 3 calibrated images with these artefacts and i will produce
a projectfile as a show case. This will include the mask generation, the curves process(es) process container
and the image container usage.

Gerald

Offline Geoff

  • PixInsight Padawan
  • ****
  • Posts: 860
    • View Profile
Re: Fixing Bad Pixel Columns
« Reply #3 on: 2018 September 14 20:44:00 »
That’s very clever Gerald!
Don't panic! (Douglas Adams)
Astrobin page at http://www.astrobin.com/users/Geoff/
Webpage (under construction) http://geoffsastro.smugmug.com/

Offline pfile

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 4133
    • View Profile
Re: Fixing Bad Pixel Columns
« Reply #4 on: 2018 September 14 21:22:44 »
another thing to try, as long as you have dithered, is to just write '1's over the bad columns using pixelmath and then the pixel rejection during imageintegration will take care of rejecting the columns. however, if some of the subs have equal X dithers then this may not work so well - you may end up with bright columns in the integrated result.

rob

Offline kerrywaz1

  • Member
  • *
  • Posts: 68
    • View Profile
    • Spirit Bear Observatory
Re: Fixing Bad Pixel Columns
« Reply #5 on: 2018 September 16 09:47:49 »
Hi Gerald,

Your method sounds interesting. I sent you a private message with Dropbox links to three cal'd images with the bad columns.

Thanks,
Kerry
Deep Sky Instruments RC14C
FLI PL16803 w/MOAG + StarlightXpress Ultrastar
Pyxis 3"
Astro Physics AP1200
Tak FSQ-106EDXII
Apogee Alta F16M
Hap Griffin modified Canon EOS 5D Mark II
PixInsight, MaxIm DL, PhotoShop CS5

Offline oldwexi

  • PixInsight Guru
  • ****
  • Posts: 605
    • View Profile
    • Astronomy Pages G.W.
Re: Fixing Bad Pixel Columns
« Reply #6 on: 2018 September 16 14:50:11 »
Hi Kerry!
Project for column correction without CC is already in my dropbox.
I sent the link to you just now.

Gerald

Offline ngc1535

  • PixInsight Addict
  • ***
  • Posts: 248
    • View Profile
Re: Fixing Bad Pixel Columns
« Reply #7 on: 2018 September 18 22:15:03 »
Interesting:

1. What method does CC employ to substitute pixels in columns (median, mean? of neighbors?)? I have always wondered....
2. OldWexi: I have simply used DefectMap and choosing a method (such as mean or median). Does your way have a significant improvement on say mean or median of DefectMap?

I guess a side-by-side comparison would be really great. If both methods make an "invisible" result- I am going to use the one with less steps. :)

-adam

Offline oldwexi

  • PixInsight Guru
  • ****
  • Posts: 605
    • View Profile
    • Astronomy Pages G.W.
Re: Fixing Bad Pixel Columns
« Reply #8 on: 2018 September 19 10:59:42 »
Hi Adam!
CC and/or Defectmap is painting/inventing data in the bad columns by using the neighbouring pixels.
This is ok if the column defects are white or black only or with only one brightness.
But a line mask together with Curves increases or decreases the originnal data to a level which makes the column defect disappear but with "correct" data.

If Kerry allows me to show his images und the project i can make it visble to all.
As the used images belong to Kerry i cant go public with the project...

Gerald

Offline ngc1535

  • PixInsight Addict
  • ***
  • Posts: 248
    • View Profile
Re: Fixing Bad Pixel Columns
« Reply #9 on: 2018 September 19 11:24:14 »
Yes, I understand what you are doing and I agree it will work just fine- but I wonder if you need to make a curve adjustment for each column. I am guessing you would. Biases usually have large scale gradients (sometimes in the horizontal direction) so that an adjustment you make for one column would not be valid for another column.

As Rob mentions, with a truly random dither and enough frames- no data needs to be invented at all through subsequent rejection. The only quibble for me there is I think writing zeros instead of ones is better since large outliers tend to be on the high end.

And just a step further... you might want to comment on whether you would make your adjustment before or after creating a "superbias."

thanks,
-adam

Offline kolec

  • PixInsight Enthusiast
  • **
  • Posts: 97
    • View Profile
Re: Fixing Bad Pixel Columns
« Reply #10 on: 2019 March 09 07:22:31 »
Hi

I dont know how I can  create a vertical line mask (or more) with Pixelmath ?

kolec

Offline oldwexi

  • PixInsight Guru
  • ****
  • Posts: 605
    • View Profile
    • Astronomy Pages G.W.
Re: Fixing Bad Pixel Columns
« Reply #11 on: 2019 March 09 09:44:28 »
One of the Pixelmath expression for creating a white line
in column 427 is:
1-d2seg(427,0,427,2048)
427,0  is the xy position of the line start
427,2048  is the xy position of the end of the line.
(In this case my image has a heigt of 2048)

If you want to create a shorter line for example
1-d2seg(427,350,427,1200)
This creates a vertical line from
xy 427,350  to yx 427,1200

xy 0,0 is in PI ususally the upper left corner.
But you can find the start- and endpositions of your future line with the ReadoutCursor

Gerald

Offline kolec

  • PixInsight Enthusiast
  • **
  • Posts: 97
    • View Profile
Re: Fixing Bad Pixel Columns
« Reply #12 on: 2019 March 10 04:47:27 »

Great thank you, Gerald!

kolec



Offline martin farmer

  • Newcomer
  • Posts: 32
    • View Profile
Re: Fixing Bad Pixel Columns
« Reply #13 on: 2019 March 10 12:33:57 »
Hello All,

I too would like to see the results of Gerald's workflow - if Gerald & Kerry would allow us to see.

Regards

Martin

Offline oldwexi

  • PixInsight Guru
  • ****
  • Posts: 605
    • View Profile
    • Astronomy Pages G.W.
Re: Fixing Bad Pixel Columns
« Reply #14 on: 2019 March 10 13:10:58 »
Hi Martin!
I was thinking already about a little video which shows
how i use this technics. But it wont be ready before next weekend.

Generally
There are different cases where you use different solutions.
In my case i do get from the remote iTelescope T8 this data where the calibration,
because of old bias files, is not 100% correct.
Yes, I do dither but as i exposure very faint molecular clouds which i have to stretch strongly,
and despite dither the vertical stripes come out during extreme processing.

So the only way in my case is:
Check the column defects,
For each column i create a line with pixelMath,
Mask the bad image with one line mask and increase the intensity of this part
of the bad image with Curves or PixelMath  until the line is not visible anymore. (using a preview for finding the right value)
This i do for every bad column.
Out of the historyExplorer i copy/create a macro with all previous steps for all lines.
Than i create an ImageContainer and execute
the macro on the container on all images of a filter serie.

This is where i plan the video

There are other situations:
If i have a destroyed unrevoverable column i use the columns mask and replace
the column with the same but better data from another good/better image.
(needs to register the good image to the bad imag...)

Other situation:
If there are only a few images with bad columns, when dithering
the image integration process helps to let disappear these artefacts.
if not the  CC process can help but is the worst solution from data quality.

Gerald