Preprocess Pipeline v0.9.0Beta

[georg.viehoever]

...
The preconfiled xpsm's could be part of the official PI release and therfore stringently controlled(no bugs, malware, etc...). What do you think?
...

Good idea!
Georg

[RobF2]

This will be included in the next release. Please note that leaving the reg ref parameter blank in the current version will give you an error message.

Hope this makes sense, not getting much sleep lately.

John

Perfectly.  Thanks John.

[RobF2]

well, the thing about this is that undoubtedly you have one image that has the best FHWM, the least trailing, etc. that's the image you want to use as a registration reference, and it's not necessarily the first image in your sequence. of course right now PI can't really automagically evaluate an image for you... i usually use DSS to score my files and then pick the best one as the registration reference in PI.

Point taken, but if I haven't collected a lot of data, and have already culled the less optimal images (after using Nikolay's Blink script usually) there isn't usually a lot of variation and I'd often like to throw everything together for a quick impression of how the data will stack up (umm, no pun intended).  

I guess I should get more precise about identifying the optimal reference though for best final outcome.  I know this isn't automatic yet, but I'm sure I've read about FWHM tools/scripts people have written for PI though?

[mmirot]

Ultimately, I will be using Nikolay's Blink script first too.
I right now it does give FWHM calcuation but I am sure this can be added soon.

By just, blinking I can find a good reference frame. The bloated ones are easy to spot with the animation. 

By the way, is there reason to have different reference frame for integration too.

Max

[RBA]

I'm trying to play with this module and I've noticed that quite often, the text in the window where the tree is displayed, sometimes disappears. And once it does, it never comes back. So I end up with a blank box with the checkmarks only.

Win7x64

[mmirot]

I am making and saving my masters first then running the script again on the master image.
I seem to be running into problems with the flats.

If I want the flats work properly how should I set this up. They need to be bias subtracted at some point. Should I calibrate ( bias subtract the flats) at the time I create the master flat? 

Is there a way to use a uncalibrated flat on a master image so it works correctly?  It still needs to have a bias or dark subtracted at some point.

Any pointers.

By the way is a lovely script.

Max

[mmirot]

I am making and saving my masters first then running the script again on the master image.
I seem to be running into problems with the flats.

If I want the flats work properly how should I set this up. They need to be bias subtracted at some point. Should I calibrate ( bias subtract the flats) at the time I create the master flat? 

Is there a way to use a uncalibrated flat on a master image so it works correctly?  It still needs to have a bias or dark subtracted at some point.

Any pointers.

By the way is a lovely script.

Max

Anyone?

[sjbrown]

PreprocessPipeline v 0.9.35 Beta release has been attatched to the first post in this thread.

This release includes two additions; the registration reference image now defaults to the first image in the input files list and an independant, selectable process for format conversion to the 'fits' format is now available.

This release also fixes a bug in the ImageIntegration look-ahead code which would point to an
incorrect image name when 'Close image windows' parameter is set to true.

The next release out will probably be after the release of PI 1.7 to handle any changes in the preprocessing
PCL modules.

John

[sjbrown]

I am making and saving my masters first then running the script again on the master image.
I seem to be running into problems with the flats.

If I want the flats work properly how should I set this up. They need to be bias subtracted at some point. Should I calibrate ( bias subtract the flats) at the time I create the master flat? 

Is there a way to use a uncalibrated flat on a master image so it works correctly?  It still needs to have a bias or dark subtracted at some point.

Any pointers.

By the way is a lovely script.

Max

Anyone?

Hi Max,

I believe you can calibrate your flats at either time. I usually run all my calibration frames separatly as you are doing and when creating a master flat I calibrate the subs with my master bias (I use a DSLR so I have very short exposure subs and don't use darks to calibrate flat subs). But you can have the flat calibrated when you run calibration on your light frames by setting 'Calibrate flat' to true.

Hope this helps,

John

[sjbrown]

I'm trying to play with this module and I've noticed that quite often, the text in the window where the tree is displayed, sometimes disappears. And once it does, it never comes back. So I end up with a blank box with the checkmarks only.

Win7x64

I apologize for the slow response. 
Does this occur when you are changing the value of a process parameter?

John.

[mmirot]

Hi,

I've tested the script to generate master calibration frames and to calibrate ten light frames. It worked great.

After completion, the console gives an error:

Code: [Select]

*** Error [222]: /home/vicent/PI/src/scripts/Pipeline/PreProcessPipeline.js, line 2749: TypeError: data.imgSet[c].inputFiles[i] is undefined
But this appears not to affect the resulting images. They are saved and correctly processed in the HDD.

I think you must deeply review default parameters. I put below what I think it's better:

Master Bias:

Code: [Select]

**Integrate:
Combination: Average
Weights: Don't care
Normalization: No normalization
Rejection algorithm: Winsorized Sigma clipping (usually we acquire more than 10 bias frames)
Rejection normalization: No normalization
Sigma low: 4
Sigma high: 3
Use file cache: false
Evaluate noise: false
Close output windows: false

Master Dark & Master Flat Dark

Code: [Select]

**Calibrate:
Master Bias enabled: true
Master Dark enabled: false
Master Flat enabled: false

**Integrate:
Combination: Average
Weights: Don't care
Normalization: No normalization
Rejection algorithm: Winsorized Sigma clipping (usually we acquire more than 10 dark frames)
Rejection normalization: No normalization
Sigma low: 4
Sigma high: 3
Use file cache: false
Evaluate noise: false
Close output windows: false

Master Flat

Code: [Select]

**Calibrate:
Master Bias enabled: true
Master Dark enabled: true
Master Flat enabled: false

**Integrate:
Combination: Average
Weights: Noise evaluation
Normalization: Multiplicative
Rejection algorithm: IMO, winsorized (for dome flats) or percentile (for sky flats)
Rejection normalization: Equalize fluxes
Sigma low: 4
Sigma high: 3
Percentile low: 0.2
Percentile high: 0.03
Use file cache: true
Evaluate noise: true
Close output windows: false

Master Image

Code: [Select]

**Calibrate:
Master Bias enabled: true
Master Dark enabled: true
Master Flat enabled: true

**Integrate:
Combination: Average
Weights: Noise evaluation
Normalization: Additive + Scaling
Rejection algorithm: It depends on the number of images... but Winsorized coud be a good default value
Rejection normalization: Scale + zero offset
Sigma low: 4
Sigma high: 3
Percentile low: 0.2
Percentile high: 0.1
Use file cache: true
Evaluate noise: true
Close output windows: false


Hope this helps!
Vicent.

I am have lots of trouble understanding calibration setting options. I am end up with gray images etc.
Perhaps someone could explain the true / false settings for the calbration part of script.  I don't understand the application of the these switches.
Also what are the recommend True / false if you create master cals seperately and if you link to masters cals created with the master image.

Thanks

Max

[mmirot]

Some of these options are not very clear on how the should work. Many are should be grayed out to prevent confusion.

Master Dark & Master Flat Dark
Code:
**Calibrate:
Master Bias enabled: true
Master Dark enabled: false
Master Flat enabled: false

Obviously applying a flat and dark subtraction is not helpful when creating these. It should not be selectable.
I am not sure why you want to bias subtract these at this time either. Seems like it is source of error down the road too.


Master Flat
Code:
**Calibrate:
Master Bias enabled: true
Master Dark enabled: true
Master Flat enabled: false

Again, why is applying a flat even a option here?

It is not clear what the other options do either. Usuallu,  you master flat-dark subtract or master bias subtract the flat not both. That what this looks like to me???

Master Image

Code:
**Calibrate:
Master Bias enabled: true
Master Dark enabled: true
Master Flat enabled: true

Ok this makes sense to some degree.
However, what does bias do here?
A source for flat subtraction?
A souce for dark scaling and optimisation?

I am concerned with calbration conflicts. It seems the pipeline should adopt some conventions to prevent double subtraction and duplicate flat fieldling.

I need help as it stands now. I am fine with registration and integration.

Max

[mmirot]

I guess I am the only one interested in this script currently
I still have not figured out how to set up calibration.
 
I have thinking.  I have a few good suggestions for John.

Get rid of link back for calibration source.

Every cal master input acts a source and path whether one image or a set that requires integration.
A sole image is treated and used as previously created master by default.


If you choose to add image(s) for dark flat it automatically defaults to a dark flat subtract.
Otherwise it automatically defaults to a bias subtraction for the flat.

The true false options under the calibration process would only apply to the master image never to calibration master bias flat etc.

The original location for the input of the master cal  would be the default location of created masters frames unless the the user changes the location under the process options.

This is a lot more logical to me and will reduce confusion.
 I hope you can follow my logic


What do you think?

Max

[mmirot]

I am trying to keep this thread on life support.  I think it needs the paddles 

max

[Juan Conejero]

Hi John,

Vicent Peris and I have been testing your script. First I have made a couple modifications because the script didn't work in PI version 1.7. I have attached the modified version to this message. The first change initializes several properties of the ImageSet object (ImageSet.cnvParms, ImageSet.calParms, etc.). This has required some code reordering to make the necessary objects (CnvParms, CalParms, etc.) visible to ImageSet. The changes are marked with a "### MODIFIED ###" string.

The second change is to consider FITS files those with the following suffixes: .fits, .fit and .fts. In your original code you were only considering .fit. This was causing unnecessary format conversions. I have also changed all default file extensions to ".fits" instead of ".fit".

I want to revise your code in more depth, but right now we have a few suggestions:

- Add a "Delete" button to the Node Controls section. In this way one would be able to remove the current tree node (right now you can only disable a node in case you make a mistake).

- There are some layout issues with the Input Controls TreeBox.

- Use a set of predefined ImageIntegration parameters. There should be different default parameters for integration of master bias, master dark, master flat and lights.

- The script has in excess of 7000 lines now. I would split it in several .js files and #include all of them from a main file with the script entry point (which would be the executable JavaScript file with feature preprocessing directives).

If you don't mind I'll try to fix these problems by modifying your code. I'll let you know when I have it ready.

You're doing a wonderful work and an essential contribution to PixInsight with this script. We want to express our gratitude here and encourage you to continue development of this great tool.