Preprocess Pipeline v0.9.0Beta

[sjbrown]

Hi Everyone,

After many months in development I finally have a pipeline script that is ready to share with the community.

The script has been tested exentsively but I need volunteers to do a final shakedown to get rid of any bugs
that may not be showing because of my particular workflow and preferences. I am currently using a DSLR
so I am particularly interested in CCD users' experiences with the script.

There are many more things on the TODO list including the use of global and/or local preferences, deletion of
intermediate files, a more robust user defineble file/directory scheme, etc..., plus any thoughts and suggestions
from the community for improvements.

I appologize for not including an image of the interface in this post, I currently don't have a means to do so and
the space allowed for attachments can't handle the image size as well as the two necssary attachments.
I have attached the source for the PI Document of the script and it contains all the images needed.

To add the script to PI, do the following:

• Unzip PreprocessPipeline_v090Beta.zip to a folder of your choosing
• Copy or move the PreProcessPipeline.js file to .../PixInsight/src/scripts folder
• In PixInsight, go to Script> and click on Feature Scripts...
• In the Feature Scripts dialog click on the Add button
• In the Select Directory dialog click on the src folder and then click the Select Folder button
• The Feature Scripts dialog will then add the new script to PI

For those of you who have not compiled a PI Document before, please follow these steps:

• Unzip attached source zip file to a folder of your choosing
• In PixInsight, go to Script>Development
• Click on Documentation Compiler
• In the Documentation compiler script, click on the Add button
• Locate the PreprocessPipeline.pidoc file that was unzipped and add it to the script's list
• Click on the OK button, the script will do the rest

I believe this can be done before or after you add the script to PI but it is probably best to add the script first.
You should now be able to view the script documentation in the Documentation Browser of the Process Explorer.

Thanks in advance for your help and I hope you enjoy the script.

John

[vicent_peris]

Hi John,

This is really great! I'm testing it right now. Let me some time to digest it... I think I will have a lot of things to add.


Best regards,
Vicent.

[vicent_peris]

Hi again,

I have set up all the pipeline, but I have a problem: I cannot select the reference image for the registration process. This is what I do:

- Select with the mouse the frame in the input file list, which I want to set up as reference.
- Press the set reference button

Pressing the set reference button does nothing. What happens here?


Regards,
Vicent.

[sjbrown]

To all,

I have repared a critical bug in the debayer process engine that was not propagating the debayered files to the registration process. I had
inadvertantly deleted one line of code when I changed the input file array structure and failed to notice the error. The script does not fail
but the integration output is definately not what it should be. I have replaced the zip file in the top post so if you have already downloaded
it, please download it again and replace the bad file. Thanks.

John

[sjbrown]

I have set up all the pipeline, but I have a problem: I cannot select the reference image for the registration process. This is what I do:

- Select with the mouse the frame in the input file list, which I want to set up as reference.
- Press the set reference button

Pressing the set reference button does nothing. What happens here?

Vincent,

The registration reference file is set by doubleclicking on the 'Reference image' child node in the Register process parameters. A file path edit group will be displayed in the Parameter Modification controls section.
The Set Reference button in the input files list menu is for setting the Integration process reference image.
Sorry for the confusion, I'll see if I can make it a little more clear in the documentation.

Thanks,
John

[vicent_peris]

Hi,

I've tested the script to generate master calibration frames and to calibrate ten light frames. It worked great.

After completion, the console gives an error:

Code: [Select]

*** Error [222]: /home/vicent/PI/src/scripts/Pipeline/PreProcessPipeline.js, line 2749: TypeError: data.imgSet[c].inputFiles[i] is undefined
But this appears not to affect the resulting images. They are saved and correctly processed in the HDD.

I think you must deeply review default parameters. I put below what I think it's better:

Master Bias:

Code: [Select]

**Integrate:
Combination: Average
Weights: Don't care
Normalization: No normalization
Rejection algorithm: Winsorized Sigma clipping (usually we acquire more than 10 bias frames)
Rejection normalization: No normalization
Sigma low: 4
Sigma high: 3
Use file cache: false
Evaluate noise: false
Close output windows: false

Master Dark & Master Flat Dark

Code: [Select]

**Calibrate:
Master Bias enabled: true
Master Dark enabled: false
Master Flat enabled: false

**Integrate:
Combination: Average
Weights: Don't care
Normalization: No normalization
Rejection algorithm: Winsorized Sigma clipping (usually we acquire more than 10 dark frames)
Rejection normalization: No normalization
Sigma low: 4
Sigma high: 3
Use file cache: false
Evaluate noise: false
Close output windows: false

Master Flat

Code: [Select]

**Calibrate:
Master Bias enabled: true
Master Dark enabled: true
Master Flat enabled: false

**Integrate:
Combination: Average
Weights: Noise evaluation
Normalization: Multiplicative
Rejection algorithm: IMO, winsorized (for dome flats) or percentile (for sky flats)
Rejection normalization: Equalize fluxes
Sigma low: 4
Sigma high: 3
Percentile low: 0.2
Percentile high: 0.03
Use file cache: true
Evaluate noise: true
Close output windows: false

Master Image

Code: [Select]

**Calibrate:
Master Bias enabled: true
Master Dark enabled: true
Master Flat enabled: true

**Integrate:
Combination: Average
Weights: Noise evaluation
Normalization: Additive + Scaling
Rejection algorithm: It depends on the number of images... but Winsorized coud be a good default value
Rejection normalization: Scale + zero offset
Sigma low: 4
Sigma high: 3
Percentile low: 0.2
Percentile high: 0.1
Use file cache: true
Evaluate noise: true
Close output windows: false


Hope this helps!
Vicent.

[vicent_peris]

I have set up all the pipeline, but I have a problem: I cannot select the reference image for the registration process. This is what I do:

- Select with the mouse the frame in the input file list, which I want to set up as reference.
- Press the set reference button

Pressing the set reference button does nothing. What happens here?

Vincent,

The registration reference file is set by doubleclicking on the 'Reference image' child node in the Register process parameters. A file path edit group will be displayed in the Parameter Modification controls section.
The Set Reference button in the input files list menu is for setting the Integration process reference image.
Sorry for the confusion, I'll see if I can make it a little more clear in the documentation.

Thanks,
John

Hi, thanks,

then, the reference image for registration is an uncalibrated image?? The reference image should be an already calibrated one.


Regards,
Vicent.

[sjbrown]

then, the reference image for registration is an uncalibrated image?? The reference image should be an already calibrated one.

No. The code at the beginning of the registration process engine propagates the chosen reference image so that it points to
the correctly processed file, if the file selected was in the input file list. If the file was not in the list but somewhere on the disk,
the code verifies that the file exists before executing the registration process. If the code does not find the reference image an error
messge is generated and the script shuts down. (this is an area I need to work on).
For example, selecting foo.fit as the reference with the calibration and debayer processes selected, the code will propagate the reference image
name to foo_c_d.fit and then chek to make sure it's in the propagatd input file list.

Does this help?

John

[vicent_peris]

then, the reference image for registration is an uncalibrated image?? The reference image should be an already calibrated one.

No. The code at the beginning of the registration process engine propagates the chosen reference image so that it points to
the correctly processed file, if the file selected was in the input file list. If the file was not in the list but somewhere on the disk,
the code verifies that the file exists before executing the registration process. If the code does not find the reference image an error
messge is generated and the script shuts down. (this is an area I need to work on).
For example, selecting foo.fit as the reference with the calibration and debayer processes selected, the code will propagate the reference image
name to foo_c_d.fit and then chek to make sure it's in the propagatd input file list.

Does this help?

John

Ok, thanks, this is clear to me now.

This is a great script. I really like the tree idea. Specially useful is the "calibration link" idea.


V.

[sjbrown]

Thanks Vicent,

Once you get used to it the tree is very nice, and quick as well.
The calibration links are vital to the usefulness of the script. Without them the 'pipeline' would be restricted to each individual
image set. This way, if you are not using pre-existing masters, the 'pipeline' extends beyond the individual image set. 8)
I have plans for eventually being able to link the calibration outputs of two or more light frame image sets to the registration and integration
processes of another image set so you could combine image sets of the same object from different nights.

John

[vicent_peris]

One funny bug.

I have set up a pipeline in which I calibrate, register and integrate some images of 5 different objects. For each object, different skyflats are used, and each object has 10 individual frames.

All is going well... until I see the integrated images. The first three images (the script is running right now) are "int_007_G.fit", "int_008_G.fit" and "int_010_R.fit". I load the three images in another instance of PixInsight, and I see that the three images are identical! I mean that "int_008_G.fit" and "int_010_R.fit" are clones of "int_007_G.fit".


Regards,
Vicent.

[Juan Conejero]

Hi John,

This script is precisely what I have been expecting to happen since I released the ImageCalibration module. I'm truly amazed at how well you have designed it from both the structural and procedural points of view. I love it, excellent work.

I'm sure you are aware of the transcendence of this tool, so enough compliments for now and let's get to work. There are several actions that we should take ASAP, if you agree:

- Make this thread a sticky in the New Scripts and Modules section.

- Incorporate this script into the set of standard scripts in all PixInsight Core distributions.

- This script deserves and needs a wide distribution to the whole PixInsight community. For this purpose:

• Everybody should be aware of its existence. I'll be glad to send an email circular to all PI users, pointing them to this thread.
• It should be distributed through the update system, along with its documentation. Are you planning to set up your own PixInsight repository? Do you need help with this? — I'll be glad to support you, as needed.

Now let me point out a feature that I think should be implemented also ASAP: this should be an instantiable script. As you know, an instantiable script in PixInsight is able to generate and import instances of the Script process. This means that the user is able to drag the typical blue triangle to the workspace to generate process icons. These icons can be managed as standard process icons: save them to XPSM files, edit them, launch the corresponding script directly, etc. For example, the TranslucentPlanets example script is instantiable. Note that this is fully compatible and can coexist with your current settings management feature.

If you want, I can help you to make this script instantiable. It is an essential feature IMO. What do you think?

Another feature that would be very nice, is an option to work in two modes: simplified and extended. In simplified mode, only a reduced set of critical parameters would be available to the user. For example, you usually don't need to tweak the RANSAC Tolerance parameter of StarAlignment, etc. In extended mode, all parameters would be available, as now happens.

Along with that, there are a few things of minor importance, such as some buttons not well rendered on Mac OS X (that's a common problem with OS X's Aqua style) —I can fix this for you if you want—, and the fact that all images in your documentation have .PNG file extensions, which prevents them from loading on case-sensitive platforms (all but Windows).

Thank you so much for the big amount of fine work and knowledge that you've invested —will invest, as this is just the beginning of something huge — in this script.

[georg.viehoever]

Wow, what a monumental script! I will try it as soon as time allows.
Georg

[Jack Harvey]

This is a great idea.  By all means lets try to get this to work on OS also insatiable. Juan.

[georg.viehoever]

First attempt to produce a master bias ended in the attached error. I will try again with next version  8)

Georg

Code: [Select]

***************** PreProcess Engine Check *****************

Checking engine state for Master Bias 1

checking input file list length...
Input file list length passed...

checking input file path validity...
Input file paths validated...

Master Bias 1 passed...

Checking engine state for Master Dark 1

Master Dark 1 is disabled... bypassing engine check.

Checking engine state for Master Flat 1

Master Flat 1 is disabled... bypassing engine check.

Checking engine state for Master Image 1

Master Image 1 is disabled... bypassing engine check.


All engine checks passed... ready to execute script engine.

############## Processing Master Bias 1 ##############

Master Bias 1 process time : 0 hours   0 mins   0.000 secs


Finished processing 4 image sets.
Total process time : 0 hours   0 mins   0.000 secs
*** Error [222]: C:/PCL_Georg/scripts/PreprocessPipeline_v090Beta/PreProcessPipeline.js, line 4051: TypeError: this.rootChild[idx] is undefined