PixInsight Forum

Software Development => New Scripts and Modules => Topic started by: vicent_peris on 2019 May 06 04:03:40

Title: Pattern subtraction scripts
Post by: vicent_peris on 2019 May 06 04:03:40
Hi all,

I'm working on a new image, but the CCD camera is getting very old, so it has a sensor plenty of defects. To remove these defects I had to write some scripts. I'll be writing an online article about the scripts in early June. The scripts don't have a GUI at this moment (and I have no plan to add it, I simply don't have the time). So anybody is welcome to code it. :-)

The attached image shows an integration of all the H-alpha exposure without alignment. This image increases the signal to noise ratio of all the defects in the image, which are the following:

- The top and bottom dark bands are some sort of gradients introduced (I guess) by the amplifier. These gradients vary in intensity from frame to frame.

- The diagonal dark structure at right bottom is an RBI residual coming from the flats on the afternoon. This planetary nebula has an extremely low surface brightness, and shooting with RBI supression would increase the background noise, so always do a regular readout. The problem is that you can have an RBI residual of varying intensity in certain frames.

- The CCD has a lot of entire and partial defective columns. More of 10% of the columns are now defective, which means that you cannot use CosmeticCorrection because you would be painting a lot of data...

These are the scripts I've developed:

PatternSubtraction script:

Script to subtract a fixed pattern from an image. This script can correct the active image or a list of images in an specified input directory.

To correct the RBI residual signal, I made a master dark with RBI-cancellation readout. Then, you remove the small-scale components with MMT. This image will work as the pattern reference. The script can be applied to the active image or a list of images in an input directory. In any case, you need to open the processed pattern reference image and specify its identifier in the script setup. The script works by progressively subtracting the pattern from the image until it finds a minimum in the noise measurement. I'm also attaching the RBI pattern reference image.

LinearDefectDetection script:

Script to detect defective columns or rows in a reference image. It can detect entire or partial defective columns or rows. The best approach to detect these line defects is to use an integrated image of non-aligned images. This way, we increase the signal to noise ratio of the lines, which is crucial to be able to detect them above the noise floor. This wouldn't be possible in a single subframe because most of the defects are well below the noise of the image and they only appear as diffuse lines in the integrated image. So, first you need to generate this kind of master image. The script will be always applied to the active image. It takes its time to finish since the algorithms are complex, but the time is well worth and it's a one-time process.

PartialLineDetection script:

A stand-alone version of the partial line detection function. It's always applied to the active image, which should be an integrated image of the subframes without alignment, in the same way as in the LinearDefectDetection script. It writes a CosmeticCorrection-compatible defect table as well.

LinearPatternSubtraction script:

Script to correct residual column or row patterns in an image. It can read the previously generated defect table and correct the defects on the active image or in an image list. It is possible to correct all the entire columns or rows in an image and, at the same time, correct the specified defects in the defect table.



The script writes a CosmeticCorrection-compatible defect table to disk. You can then use this table in CosmeticCorrection, but it better (IMHO) if you use this table to correct the defects with LinearPatternSubtraction.

All these scripts are configured by modifying the properties of the Config function. If you want to give it a quick try, open the LinearPatternSubtraction script in the script editor. Set the correctEntireImage property to true, don't specify a defect list in the partialDefectsFilePath property, and modify the backgroundReferenceLeft, backgroundReferenceRight, backgroundReferenceTop and backgroundReferenceBottom properties to select the right area in your image. Then, hit F9 to apply the script to the active image. It will correct and vertical banding in your image.

The scripts I'm sharing solve all the problems above. Feel free to contribute. I guess my code won't be very smart since this is my first time programming in an object-oriented language (my only experience up to now was shell scripting). But the code is very well commented and documented. PLEASE READ THE INCLUDED DOCUMENTATION!

Feel free to ask anything about the scripts and how to set them up.


Thanks,
Vicent.
Title: Re: Pattern subtraction scripts
Post by: oldwexi on 2019 May 06 05:23:35
Hi Vicent!
That sounds great!
Question: Will the LinearDefectDetection script be usable with ImageContainer?
So i can remove one column defect on all images of a filter session.

Gerald
Title: Re: Pattern subtraction scripts
Post by: hvb356 on 2019 May 06 06:05:04
Hola Vicent,

welcome to the club.

Hartmut
Title: Re: Pattern subtraction scripts
Post by: dgruber on 2019 May 06 09:09:49
Dear Vicent,

Thank you for your scripting.
Please allways inform about new work from you , our facebookgroup Pixinsight Österreich (you are member)  and  PixInformatik.

I forwarded your work allready.

Franz
Title: Re: Pattern subtraction scripts
Post by: vicent_peris on 2019 May 08 04:34:32
Hi,

I made multiple improvements to the code, just download again the new version in the first message.


V.
Title: Re: Pattern subtraction scripts
Post by: vicent_peris on 2019 May 08 04:38:59
Hi Vicent!
That sounds great!
Question: Will the LinearDefectDetection script be usable with ImageContainer?
So i can remove one column defect on all images of a filter session.

Gerald

Hi Gerald,

LinearDefectDetection does not work in batch mode. I designed that script with the idea in mind of having a high quality reference image. I use an integrated image without alignment. This improves a lot the defect detection.

The problem is that those defects come out in the final master because most of them were below the noise floor in the single subframes. So, asuming you have the same defects in all the image set, you can use that integrated image as reference to detect the defects; then, you apply LinearPatternSubtraction in batch mode with the previously generated defect table.


Best regards,
Vicent.
Title: Re: Pattern subtraction scripts
Post by: vicent_peris on 2019 May 08 04:40:11
Dear Vicent,

Thank you for your scripting.
Please allways inform about new work from you , our facebookgroup Pixinsight Österreich (you are member)  and  PixInformatik.

I forwarded your work allready.

Franz

Franz, thank you very much for forwarding the message from one board to the another. Next time I'll post in both.

Best regards,
Vicent.
Title: Re: Pattern subtraction scripts
Post by: gvanhau on 2019 May 13 13:46:06
Hello Vicent.

Have you checked it is realy te CCD aging?
I had a simmilar issue on  my CCD Camera (increasing amount of hot pixels), but after several tests, I decided to open de Camera to see if there was any visible damaged component. What I found out was that the Thermal paste between the cooling element a the CDD had lost contact.
After replacing the thermal paste the camera worked again like new.

Regards
Geert
Title: Re: Pattern subtraction scripts
Post by: gianpri on 2019 May 16 04:21:51
Hi Vincent,
unfortunately I can't install your scripts that I decompressed in the PatternSubtraction folder. I get this message:

Title: Re: Pattern subtraction scripts
Post by: Juan Conejero on 2019 May 16 05:05:01
These scripts cannot be featured because they don't include #feature preprocessor directives, so the automatic script featuring process cannot identify them.

You have to load the scripts in Script Editor and modify a few parameters manually to adapt them to your working environment, then you can run them (F9 on Linux and Windows, Ctrl+R on macOS). The scripts provide good information about these parameters and how to customize them.
Title: Re: Pattern subtraction scripts
Post by: gianpri on 2019 May 16 12:37:19
Thank you Juan.
Title: Small bug in
Post by: aworonow on 2019 September 19 07:21:43
In LinearPatternSubtraction, around line 670 (I've added annotations, so the line number maybe off a bit), the following statement occurs:

if ( ! CONFIG.targetIsActiveWindow )

The variable is undefined. I think it should be

if ( ! CONFIG.targetIsActiveImage )

Also, has the write-up on this script been posted somewhere? I cannot find it.

Alex
Title: Re: Pattern subtraction scripts
Post by: Geoff on 2019 October 01 17:15:09
The linear pattern subtraction script works fine on an active image, but when I try to run it on an entire folder I get a message that there are no files to be processed. The files are there, they are fit files read in from Maxim and I have specified the extension in the targetImageExtension parameter. The error occurs in line 688.

Any suggestions?

Geoff
Title: Re: Pattern subtraction scripts
Post by: dgruber on 2019 October 13 03:49:12
Dear Vicent !

Your scripts are very useful. There is a big demand, because i-telescope frames are suffering from bias-problems  (vertical lines) ,  and there are many PixInsight users, which are doing a lot of frames with i-telescope.  (Me too).  Please combine your scripts with image-container,  so it would be more comfortable working with about 150 frames in one session.  As my friend Gerald Wechselberger and me want to presentate your work at the Attersee-PixInsight Meeting we would like to have a more comfortable way,  installing the scripts.  In short : We would like a final version. Many thanks ,  Franz
Title: Re: Pattern subtraction scripts
Post by: dave_galera on 2019 October 13 06:18:36
Hi Vincent,

I know you are short of time as you stated in your original post.

Can I help you in making the scripts installable from a repository and possibly work with image-container? I also use iTelescope and are suffering from the same problems as Franz noted above.

Would be please to help as a thank you for your course I did with you in Spain.
Title: Re: Pattern subtraction scripts
Post by: jkmorse on 2019 October 14 15:43:40
Vincent,

Great script, as usual.  One question.  I did not see any definition of the imageShift parameter.  What criteria should I use to modify that value?

Thanks,

Jim
Title: Re: Pattern subtraction scripts
Post by: oldwexi on 2019 November 10 12:26:32
Hi Vicent!
We have added to your script
LinearDefectDetection and
LinearPatternSubtraction
User Interfaces, so the Script does not need to be touched with Script Editor anymore.

We are asking for your permission to use your extended script in presentations at our next Austrian
PixInsight Meeting in April 2020. On an "as is" base.
It works on the active image or on images in a directory very nicely.

I have sent you with separate E-Mail the image of the UI for LPS.

Please let us know.

Gerald Wechselberger PI Austria
Title: Re: Pattern subtraction scripts
Post by: wlickteig on 2019 November 11 03:33:42
Hi Gerald,

That is great news!  I look forward to not needing to modify the script.

All the best,

Walt
Title: Re: Pattern subtraction scripts
Post by: aworonow on 2019 November 11 06:07:29
Cannot wait to try the new interface. I've struggled, and had limited success, with out-right modifying the script. But I could make good use of it, that for certain!
Alex
Title: Re: Pattern subtraction scripts
Post by: oldwexi on 2019 November 11 07:42:38
Alex, Walt,
We are ready.
We need only the permission of Vicent to give away the extendend scripts on an "as is" base.
Gerald
Title: Re: Pattern subtraction scripts
Post by: vicent_peris on 2019 November 13 08:10:18
Hi,

Thank you so much for adding the GUI!

I'm currently writing an article for the website describing the scripts. I also made some improvements to the code and corrected some bugs. With a bit of luck it won't affect too much the GUI related code.

I'll upload my new code as soon as posible.


Thank you again.
Best regards,
Vicent.
Title: Re: Pattern subtraction scripts
Post by: cliska on 2019 November 14 21:45:25
Hi all!

Vincent, I recently added the UIs for your scripts (as Gerald and Franz told) and had to modify your scripts to open and execute the configuration dialog.

I added the UI to my github-account open for public use: LDD_UI.jsh and LPS_UI.jsh

My repository on github: https://github.com/chkettu/pi-repository

Please let me know, if I can help you to integrate the UI into the newly modified / reworked scripts.

Best,
Chris.