Author Topic: Pattern subtraction scripts  (Read 2249 times)

Offline vicent_peris

  • PTeam Member
  • PixInsight Padawan
  • ****
  • Posts: 987
    • View Profile
    • http://www.astrofoto.es/
Pattern subtraction scripts
« on: 2019 May 06 04:03:40 »
Hi all,

I'm working on a new image, but the CCD camera is getting very old, so it has a sensor plenty of defects. To remove these defects I had to write some scripts. I'll be writing an online article about the scripts in early June. The scripts don't have a GUI at this moment (and I have no plan to add it, I simply don't have the time). So anybody is welcome to code it. :-)

The attached image shows an integration of all the H-alpha exposure without alignment. This image increases the signal to noise ratio of all the defects in the image, which are the following:

- The top and bottom dark bands are some sort of gradients introduced (I guess) by the amplifier. These gradients vary in intensity from frame to frame.

- The diagonal dark structure at right bottom is an RBI residual coming from the flats on the afternoon. This planetary nebula has an extremely low surface brightness, and shooting with RBI supression would increase the background noise, so always do a regular readout. The problem is that you can have an RBI residual of varying intensity in certain frames.

- The CCD has a lot of entire and partial defective columns. More of 10% of the columns are now defective, which means that you cannot use CosmeticCorrection because you would be painting a lot of data...

These are the scripts I've developed:

PatternSubtraction script:

Script to subtract a fixed pattern from an image. This script can correct the active image or a list of images in an specified input directory.

To correct the RBI residual signal, I made a master dark with RBI-cancellation readout. Then, you remove the small-scale components with MMT. This image will work as the pattern reference. The script can be applied to the active image or a list of images in an input directory. In any case, you need to open the processed pattern reference image and specify its identifier in the script setup. The script works by progressively subtracting the pattern from the image until it finds a minimum in the noise measurement. I'm also attaching the RBI pattern reference image.

LinearDefectDetection script:

Script to detect defective columns or rows in a reference image. It can detect entire or partial defective columns or rows. The best approach to detect these line defects is to use an integrated image of non-aligned images. This way, we increase the signal to noise ratio of the lines, which is crucial to be able to detect them above the noise floor. This wouldn't be possible in a single subframe because most of the defects are well below the noise of the image and they only appear as diffuse lines in the integrated image. So, first you need to generate this kind of master image. The script will be always applied to the active image. It takes its time to finish since the algorithms are complex, but the time is well worth and it's a one-time process.

PartialLineDetection script:

A stand-alone version of the partial line detection function. It's always applied to the active image, which should be an integrated image of the subframes without alignment, in the same way as in the LinearDefectDetection script. It writes a CosmeticCorrection-compatible defect table as well.

LinearPatternSubtraction script:

Script to correct residual column or row patterns in an image. It can read the previously generated defect table and correct the defects on the active image or in an image list. It is possible to correct all the entire columns or rows in an image and, at the same time, correct the specified defects in the defect table.



The script writes a CosmeticCorrection-compatible defect table to disk. You can then use this table in CosmeticCorrection, but it better (IMHO) if you use this table to correct the defects with LinearPatternSubtraction.

All these scripts are configured by modifying the properties of the Config function. If you want to give it a quick try, open the LinearPatternSubtraction script in the script editor. Set the correctEntireImage property to true, don't specify a defect list in the partialDefectsFilePath property, and modify the backgroundReferenceLeft, backgroundReferenceRight, backgroundReferenceTop and backgroundReferenceBottom properties to select the right area in your image. Then, hit F9 to apply the script to the active image. It will correct and vertical banding in your image.

The scripts I'm sharing solve all the problems above. Feel free to contribute. I guess my code won't be very smart since this is my first time programming in an object-oriented language (my only experience up to now was shell scripting). But the code is very well commented and documented. PLEASE READ THE INCLUDED DOCUMENTATION!

Feel free to ask anything about the scripts and how to set them up.


Thanks,
Vicent.
« Last Edit: 2019 May 08 11:50:29 by vicent_peris »

Offline oldwexi

  • PixInsight Guru
  • ****
  • Posts: 622
    • View Profile
    • Astronomy Pages G.W.
Re: Pattern subtraction scripts
« Reply #1 on: 2019 May 06 05:23:35 »
Hi Vicent!
That sounds great!
Question: Will the LinearDefectDetection script be usable with ImageContainer?
So i can remove one column defect on all images of a filter session.

Gerald

Offline hvb356

  • Member
  • *
  • Posts: 63
    • View Profile
Re: Pattern subtraction scripts
« Reply #2 on: 2019 May 06 06:05:04 »
Hola Vicent,

welcome to the club.

Hartmut
TOA-150, ML8300, CFW2-7 w. Astronomik HaLRGB, PDF, EM-400, guiding FS-60 + ST402, ASCOM bsd. image acquisition

Offline dgruber

  • Newcomer
  • Posts: 11
  • Dr. Franz Gruber
    • View Profile
Re: Pattern subtraction scripts
« Reply #3 on: 2019 May 06 09:09:49 »
Dear Vicent,

Thank you for your scripting.
Please allways inform about new work from you , our facebookgroup Pixinsight Österreich (you are member)  and  PixInformatik.

I forwarded your work allready.

Franz
Fiii

Offline vicent_peris

  • PTeam Member
  • PixInsight Padawan
  • ****
  • Posts: 987
    • View Profile
    • http://www.astrofoto.es/
Re: Pattern subtraction scripts
« Reply #4 on: 2019 May 08 04:34:32 »
Hi,

I made multiple improvements to the code, just download again the new version in the first message.


V.

Offline vicent_peris

  • PTeam Member
  • PixInsight Padawan
  • ****
  • Posts: 987
    • View Profile
    • http://www.astrofoto.es/
Re: Pattern subtraction scripts
« Reply #5 on: 2019 May 08 04:38:59 »
Hi Vicent!
That sounds great!
Question: Will the LinearDefectDetection script be usable with ImageContainer?
So i can remove one column defect on all images of a filter session.

Gerald

Hi Gerald,

LinearDefectDetection does not work in batch mode. I designed that script with the idea in mind of having a high quality reference image. I use an integrated image without alignment. This improves a lot the defect detection.

The problem is that those defects come out in the final master because most of them were below the noise floor in the single subframes. So, asuming you have the same defects in all the image set, you can use that integrated image as reference to detect the defects; then, you apply LinearPatternSubtraction in batch mode with the previously generated defect table.


Best regards,
Vicent.

Offline vicent_peris

  • PTeam Member
  • PixInsight Padawan
  • ****
  • Posts: 987
    • View Profile
    • http://www.astrofoto.es/
Re: Pattern subtraction scripts
« Reply #6 on: 2019 May 08 04:40:11 »
Dear Vicent,

Thank you for your scripting.
Please allways inform about new work from you , our facebookgroup Pixinsight Österreich (you are member)  and  PixInformatik.

I forwarded your work allready.

Franz

Franz, thank you very much for forwarding the message from one board to the another. Next time I'll post in both.

Best regards,
Vicent.

Offline gvanhau

  • PixInsight Old Hand
  • ****
  • Posts: 344
    • View Profile
Re: Pattern subtraction scripts
« Reply #7 on: 2019 May 13 13:46:06 »
Hello Vicent.

Have you checked it is realy te CCD aging?
I had a simmilar issue on  my CCD Camera (increasing amount of hot pixels), but after several tests, I decided to open de Camera to see if there was any visible damaged component. What I found out was that the Thermal paste between the cooling element a the CDD had lost contact.
After replacing the thermal paste the camera worked again like new.

Regards
Geert
Geert Vanhauwaert

Offline gianpri

  • Member
  • *
  • Posts: 56
    • View Profile
Re: Pattern subtraction scripts
« Reply #8 on: 2019 May 16 04:21:51 »
Hi Vincent,
unfortunately I can't install your scripts that I decompressed in the PatternSubtraction folder. I get this message:


Online Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 6873
    • View Profile
    • http://pixinsight.com/
Re: Pattern subtraction scripts
« Reply #9 on: 2019 May 16 05:05:01 »
These scripts cannot be featured because they don't include #feature preprocessor directives, so the automatic script featuring process cannot identify them.

You have to load the scripts in Script Editor and modify a few parameters manually to adapt them to your working environment, then you can run them (F9 on Linux and Windows, Ctrl+R on macOS). The scripts provide good information about these parameters and how to customize them.
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline gianpri

  • Member
  • *
  • Posts: 56
    • View Profile
Re: Pattern subtraction scripts
« Reply #10 on: 2019 May 16 12:37:19 »
Thank you Juan.

Offline aworonow

  • PixInsight Addict
  • ***
  • Posts: 247
    • View Profile
    • Faint Light Photography
Small bug in
« Reply #11 on: 2019 September 19 07:21:43 »
In LinearPatternSubtraction, around line 670 (I've added annotations, so the line number maybe off a bit), the following statement occurs:

if ( ! CONFIG.targetIsActiveWindow )

The variable is undefined. I think it should be

if ( ! CONFIG.targetIsActiveImage )

Also, has the write-up on this script been posted somewhere? I cannot find it.

Alex

Offline Geoff

  • PixInsight Padawan
  • ****
  • Posts: 885
    • View Profile
Re: Pattern subtraction scripts
« Reply #12 on: 2019 October 01 17:15:09 »
The linear pattern subtraction script works fine on an active image, but when I try to run it on an entire folder I get a message that there are no files to be processed. The files are there, they are fit files read in from Maxim and I have specified the extension in the targetImageExtension parameter. The error occurs in line 688.

Any suggestions?

Geoff
Don't panic! (Douglas Adams)
Astrobin page at http://www.astrobin.com/users/Geoff/
Webpage (under construction) http://geoffsastro.smugmug.com/

Offline dgruber

  • Newcomer
  • Posts: 11
  • Dr. Franz Gruber
    • View Profile
Re: Pattern subtraction scripts
« Reply #13 on: 2019 October 13 03:49:12 »
Dear Vicent !

Your scripts are very useful. There is a big demand, because i-telescope frames are suffering from bias-problems  (vertical lines) ,  and there are many PixInsight users, which are doing a lot of frames with i-telescope.  (Me too).  Please combine your scripts with image-container,  so it would be more comfortable working with about 150 frames in one session.  As my friend Gerald Wechselberger and me want to presentate your work at the Attersee-PixInsight Meeting we would like to have a more comfortable way,  installing the scripts.  In short : We would like a final version. Many thanks ,  Franz
Fiii

Offline dave_galera

  • PixInsight Addict
  • ***
  • Posts: 153
    • View Profile
    • QDigital Astro
Re: Pattern subtraction scripts
« Reply #14 on: 2019 October 13 06:18:36 »
Hi Vincent,

I know you are short of time as you stated in your original post.

Can I help you in making the scripts installable from a repository and possibly work with image-container? I also use iTelescope and are suffering from the same problems as Franz noted above.

Would be please to help as a thank you for your course I did with you in Spain.
Dave