Preprocess Pipeline v0.9.0Beta

[mmirot]

John,
Here I my first impressions
1)
It does really need to be an instantiable script.
It can take a long time to set up.
I would like to have the options and paths saved. Repeating the process is no fun for minor set up mistake.

2)
Inherited from a currently created master verses a path set up for a prior master is a bit confusing.
 I would be nice if this was more straightforward.

The link source button should have two simple choices.
For example "Inherit newly created  bias master verses the the path of a prior completed bias master.

This eliminates the need to add each path to the calibration common parameter list too.
In fact, removing the all three of the  master " " enable , master " " path sets simplifies the set up a lot.

The remaining calibration common parameters should be different for an master image and a flat.
There really needs to be either calibrate flat with a bias or a dark. It not needed to have a option to flat field the flat.

3)
When I select a reference frame image, I don't see a indicator to confirm the choice. Did I miss something?
4)
I can we get a delete node option?
5)
Could we start the integration parameters with better defaults such as multiplicative normalization for flats etc.
6)
Clone button the tree for another filter or way to have quick way to make a tree for more filters/master images.

7)
Eventually a wizard will help newbies get started faster

Nice work so far

Max
 
 

 

[sjbrown]

Hi Max,

I'll try to answer most of your concerns. There are a couple that I am unsure of.
Most of the concerns, specifically 1, parts of 2, 5, and 6 are addressed with the next version which is instantiable. Process icons work very well as a means of saving specific process parameter values, file paths, output directories, and any number of workflows! However, the parameter settings function already handles some of that but I haven't been able to adequately convey that. This is what happens every time the script starts up :

• The 9 (now 12) process parameter variables are created with the PCL module default parameters.
  
• In the new version, the GlobalDaa importParameters function is run. If the script is being instantiated from a script icon, the script will recreate the contents of that icon (re-instantiation of a saved instance).
  
• The sript then checks for a key in the settings file to see if any saved parameters exist, if not, the default parameters are saved to the settings file(this only happens on the very first run of the script!).
  
• Next, the script loads each of the nine parameter variables with their respective values.
  
• Once loaded, the parameter variables will pass on their values evey time a process is created for an image set.
  
• After the process engine finishes the last thing the script does is save the nine parameter variables, with any changes made, to the Settings file.
  

Due to the fact that there are no saved parameters in the Settings file the very first time the script is run, the only parameters available will be the default PCL module parameters. The second time you run the script, any changes you made to the default parameters will show up in each process' parameter list. I hope this helps make things clearer.

3)
If I understand correctly, you are refering to the 'Set Reference' button on the Input Files list. When you click on an input file and then click the Set Reference button, the file you selected is moved to the top of the list. This function sets the ImageIntegration process' reference image, NOT the StarAlignment(registration) process' reference image.

4)
When I designed the script I did consider this option but the code required to do it was both much longer and more complex than the code required to disable a node. And it ultimately would provide no more functionality than simply disabling a node. One of the reasons for prefering shorter code is that when a script is run by the user, the core must compile the source code for the script at runtime. The longer the code is, the longer it takes the core to compile the script. So I am trying to be very conservative on the length of the code.

7)
Eventually I would like to create at least a tutorial on the script. (Maybe someone will create a vidoe!!)

If I haven't been clear, please let me know.

John

[sjbrown]

PreProcessPipeline v 0.9.20 Beta is now available in the initial post in this thread.
This version adds instantiability to the script and some other minor changes.

Instantiablility allows the user to create "preset" workflows and save them to the workspace as script icons.
These icons can be saved individually or in groups, just as process icons. There are two caveats of script icons you need to be aware of:

• Running a script from an icon is a two step process. Double clicking on the icon will create an instance of the PCL Script module. This module contains the name of the script, the md5checksum of the script and all of the script parameters. From the Script module, click on the 'Apply Global' button (the blue globe) and the script will start up with the re-created instance that is stored in the script icon.
  
• All script icons made with a specific version of the script will only run with that version of the script. This means that every time a new version of the script is released, the existing script icons will need to be updated. Fortunately this task is fairly easy to do. I recommend that you do this on copies of your existing icons.
  
  • Start up the new version of the script. Without adding anything to the tree, create a new instance icon on the workspace.
    
  • Double click the icon you just created to bring up the Script module. In the Script module, highlight the md5checksum value, right click it and select 'Copy'.
    
  • Either close the current Script module interface or reset it.
    
  • Double click a script icon that needs to be updated to bring up a new Script module, or populate the existing one.
    
  • Highlight the md5checksum value, rightclick it and select 'Paste'.
    
  • Drag the Script module's 'New Instance' button on to the script icon you are updating and release the button.
    
  • In the dialog that appears, click yes. If you want, verify that the icon was updated correctly to the current script version by trying to run it.
    
  Once finished with your updates, save the changed icons to the originals (overwrite the originals).
  

Other changes:

• Added code for three more saved parameter groups. Flat dark calibration, bias integration and flat dark integration parameters are now saved separately.
  
• Added a Sort button to the Input Files menu allowing the user to perfom an alpha-numeric sorting of the list. Input Files are no longer automatically sorted when files are added.
  
• Added a Clear button to the File Path Modification controls group. This will aid in creating preset icons.
  
• Added a Clear Console button to the dialog control buttons group. Useful if you want to save the script output to file.
  
• Added two more checks to the engine check function to deal with any image sets that contain no processes and to deal with a condition in which the process tree contains no viable image sets.
  

Update:  Please note that the first time you run this version you will get a long list of warnings caused by no entries in the Settings file for the three
added parameter save variables.

I hope you enjoy the new version, I do

John

p.s. I haven't updated the documentation yet.

[mmirot]

Great John just read both posts.
I will download the next script and get back to you. 

Max

[mmirot]

I need some help with saving an instance.
I tried to drag the triangle once I populated the dialog with input files and some processes with options.
The triangle in the lower left does not do anything right now.

I was trying to set up the following operation and it crashed on me.

I wanted to take my set of flats and calibrate them by bias subtracting them and then integrating them.
I am not sure if I am setting this up correctly.

I have made the master bias and have it saved already.

I had master bias enabled as "true" and a valid path to the master bias
Everything else was set to "false" except for calibrate bias.

Integration options were average, noise evaluation , Multiplicative, Percentile clipping, equalize flux , clip setting 0.014 high and 0.014 low.

Out paths were set for both cal and integration.

Do have this right. Why the crash?

Max

Do have the set up

[pfile]

i tried this script for the first time today, downloading the instantiable one. nice work, this is going to prevent me from accidentally choosing the wrong files at each manual step of the flow

one problem though - it seems like it does not work well when input files have the same name. for instance, each night my canon software starts over from IMG_0001.CR2, so in general i may have several files with that name. the calibration portion of the script seems to do the right thing - i'll get multiple IMG_0001 files (IMG_0001_c.fit, IMG_0001_c_1.fit, IMG_0001_c_2.fit, etc.) however at the debayer step, it seems like the files are getting overwritten because i only see filenames of the form IMG_0001_c_d.fit in that directory, and there are too few files in the directory. strangely the script must still have the idea that there are multiple IMG_0001s, because it seems to have then registered the same IMG_0001_c_d.fit several times (i see IMG_0001_c_d_r.fit and IMG_0001_c_d_r_1.fit in the registered directory).

also as max says i was not able to drag the triangle to my pixinsight desktop and it does not move.

thanks for the script, it is going to be very useful for me.

edit: strange, on the 2nd try through (started over with the calibrated files as the first step of the pipe), i was able to drag the triangle. this might be related to gui weirdness on the 64-bit osx version of pixinsight.

[mmirot]

"Running a script from an icon is a two step process. Double clicking on the icon will create an instance of the PCL Script module. This module contains the name of the script, the md5checksum of the script and all of the script parameters. From the Script module, click on the 'Apply Global' button (the blue globe) and the script will start up with the re-created instance that is stored in the script icon."

I have no idea what you mean by running a script as icon.

Max

[mmirot]

Don't worry about my crash. I have not been able to repeat it.

Max

[sjbrown]

I need some help with saving an instance.
I tried to drag the triangle once I populated the dialog with input files and some processes with options.
The triangle in the lower left does not do anything right now.

Hi Max,
A couple of questions...

What OS are you running? There may be an issue with the OSX platform.

I was trying to set up the following operation and it crashed on me.

Did it crash during the ImageCalibration process?

Your setup looks correct except for possibly:

Everything else was set to "false" except for calibrate bias.

Are you using overscan regions? If not, then set 'calibrate bias' to false.

John

oops, You posted before I did

[sjbrown]

"Running a script from an icon is a two step process. Double clicking on the icon will create an instance of the PCL Script module. This module contains the name of the script, the md5checksum of the script and all of the script parameters. From the Script module, click on the 'Apply Global' button (the blue globe) and the script will start up with the re-created instance that is stored in the script icon."

I have no idea what you mean by running a script as icon.

Max

Max,

Perhaps I should have said "Starting a script by double clicking on a script icon is a two step process."
I wrote this because it is different from what happens when you double click a process icon.

Does this help?

John

[mmirot]

No the crash was during integration after calibration with a flat master.
I think it is ok now.

My OS is windows 64.

Is there documentation somewhere? I see the source code only and the file.

Max

[mmirot]

Not really.
 I start script the from my utility script list.  Under the major heading in PI titled "Script" right next to "Workspace".
 I have used execute script too...works the same.
 Not as you describe

Max

[sjbrown]

No the crash was during integration after calibration with a flat master.
I think it is ok now.

My OS is windows 64.

Is there documentation somewhere? I see the source code only and the file.

Max

I will post a copy of the compiled documentation when I get the new version(of the docs) completed.

Thanks,

John

[sjbrown]

Not really.
 I start script the from my utility script list.  Under the major heading in PI titled "Script" right next to "Workspace".
 I have used execute script too...works the same.
 Not as you describe

Max

The script icon is the "saved instance" that you were trying to create on the workspace area. I built the script on a Windows 32bit machine so it should
work on your 64bit machine. I do notice that the 'action' of the button while dragging it is not the sane as when dragging the button from a process
module. It seems to take a while for it to work.

John

[mmirot]

Some will have to do a  screen shot or two this eludes me

Max