Preprocess Pipeline v0.9.0Beta

[sjbrown]

...
one problem though - it seems like it does not work well when input files have the same name. for instance, each night my canon software starts over from IMG_0001.CR2, so in general i may have several files with that name. the calibration portion of the script seems to do the right thing - i'll get multiple IMG_0001 files (IMG_0001_c.fit, IMG_0001_c_1.fit, IMG_0001_c_2.fit, etc.) however at the debayer step, it seems like the files are getting overwritten because i only see filenames of the form IMG_0001_c_d.fit in that directory, and there are too few files in the directory. strangely the script must still have the idea that there are multiple IMG_0001s, because it seems to have then registered the same IMG_0001_c_d.fit several times (i see IMG_0001_c_d_r.fit and IMG_0001_c_d_r_1.fit in the registered directory).

...

I understand and have isolated the problem. The propagation code is not able to "see" the actual files names output from the calibration process module and is therefore propagating files without the concatenated 1, 2, 3 etc. that the process module is correctly adding to the input files. Because of this the DeBayer engine from each consecutive image set is opening the files that were output from the first image set's calibration output. I am designing a 'look ahead' section of code that will work in conjunction with the propagation code to anticipate this condition and allow the propagation code to attach the proper serialized postfix to the input filenames. This may take a couple days to code and test so, in the mean time,  you may want to try outputing to unique output directories.

Sorry for this inconvenience and thanks for reporting this error,

John

[pfile]

you may want to try outputing to unique output directories.

Sorry for this inconvenience and thanks for reporting this error,

John

by this you mean to set up separate pipelines for each night's work, each with it's own set of directories, and then have the integration process look into multiple directories for it's files?

no problem, this script is really great, glad to help debug it.

[mmirot]

"Running a script from an icon is a two step process. Double clicking on the icon will create an instance of the PCL Script module. This module contains the name of the script, the md5checksum of the script and all of the script parameters. From the Script module, click on the 'Apply Global' button (the blue globe) and the script will start up with the re-created instance that is stored in the script icon."

I have no idea what you mean by running a script as icon.

************
Ok I think I found the answer. If you drag the script symbol from the process explorer you get a icon.
  This is quite different from how  many will start a script.  I do it from the main script menu by double clicking.

Max

[mmirot]

I will post a copy of the compiled documentation when I get the new version(of the docs) completed.

Thanks,

John
______________________________________________________________


Thanks John,
This the doc will be most helpful.

It might answer some questions.
It is hard to make helpful suggestions on organization of elements without understanding all the options in the script.  I can see already it is quite flexible. My goal is help make it a bit more intuitive for a first time user, if at all possible. Avoiding that set up error message is another goal. It will save a lot extra forum help posts too. Here is an example. I did not realize that integration had a reference frame too. I got an  error message saying I forgot this. I had selected the integration frame will the button but not from the registration option. Ultimately it would best to select both at the same time in the manor , that is with the button somehow. Thanks again Max





Thanks again for this major contribution to the platform.
Max

[j.w.]

John,

sorry for the long post, when I launch the new script I get the following console messages.  This happens in both the 32 and 64 bit versions running in OSX.  I haven't run files through the script yet - perhaps that would clear these warnings?

Thanks, John


run --execute-mode=auto "/Users/johnwhite/Downloads/PixInsight Scripts/PreProcessPipeline.js"

Processing script file: /Users/johnwhite/Downloads/PixInsight Scripts/PreProcessPipeline.js
Warning : failed to read PPP/flatDarkCal/pedestal
Warning : failed to read PPP/flatDarkCal/pedestalMode
Warning : failed to read PPP/flatDarkCal/pedestalKeyword
Warning : failed to read PPP/flatDarkCal/overscanEnabled

< etc., etc. - thanks for the answer John, I'll remove most of this and clean the post up some  )

[sjbrown]

sorry for the long post, when I launch the new script I get the following console messages.  This happens in both the 32 and 64 bit versions running in OSX.  I haven't run files through the script yet - perhaps that would clear these warnings?

Hi John,

Yes, running the script through with files will clear this up. It occurs the first time you start up v 0.9.20 because of the three additional parameter
save variables added and causes no harm. I tought I had mentioned it in the announcement post but it slipped my mind. I will update that
post.

Thanks,

John

[sreilly]

How does one go about installing these scripts properly?

[sjbrown]

How does one go about installing these scripts properly?

Hi Steve,

Did you try this:

To add the script to PI, do the following:

• Unzip PreprocessPipeline_v09xxBeta.zip to a folder of your choosing
  
• Copy or move the PreProcessPipeline.js file to .../PixInsight/src/scripts folder
  
• In PixInsight, go to Script> and click on Feature Scripts...
  
• In the Feature Scripts dialog click on the Add button
  
• In the Select Directory dialog click on the src folder and then click the Select Folder button
  
• The Feature Scripts dialog will then add the new script to PI
  

Once the initial script is finalized it will be included in the PI Update system and all further updates to the script will be
handled automatically.

John

[sreilly]

Thanks John,

I guess I hadn't read down far enough. I just created master bias and master dark frames and got an error message at the end of execution but I believe it was about not saving the image.

[j.w.]

Hi John,

I've run into another issue that has me scratching my head.  I tried processing a lot of files for the first time this week, and on each occasion PixInsight never makes it through the entire process but just hangs (Mac version 64 bit version) or gives me an "out of memory" error (Mac version 32 bit version).  In each case, it seems that the script is gobbling up memory but not releasing it after each process is completed.  I'm running 205 total files (82 lights, 30 flats, 30 flat-darks, 30 darks and 33 bias), using about 4 GB of memory.  Any ideas?  It could just be the way PixInsight works (Juan?), since I've never really run through the entire process on this many files before without closing and restarting PixInsight for one reason or another.

Thanks, John

[sjbrown]

PreProcessPipline v 0.9.25 is now available for download. The documentation has also been updated to include
theinstantiation feature and new ontrol buttons. The docs are ailable in both source and compiled form.

This release is a bug fix release that makes the following corrections:

• Added a 'look ahead' function to the calibration and registration engines to 'see' if those modules will postfix a serialized number to the filename and then update the input files accordingly. This corrects a problem encountered when images with the same name from different image sets are output to the same output directory.
  
• Added code to check for input file existence while parsing the parameters from a script icon. A warning message is issued and a message is also sent to the console window.
  
• Fixed a bug in the 'Remove Disabled' input file management code that was destroying the integrety of the input files list.
  
• Fixed a bug in the 'Add Files' code that would allow the duplication of input files.
  
• Fixed a bug that was not setting the image set type counters when reconstructing the script from an icon. This was causing image set titles to be duplicated.
  

All preprocess script icons that were created with version 0.9.20 will require updating their md5checksum values before they can be used with this version. The documentation now includes a description for updating the icons.

Thanks for your input,

John

[sjbrown]

Hi John,

I've run into another issue that has me scratching my head.  I tried processing a lot of files for the first time this week, and on each occasion PixInsight never makes it through the entire process but just hangs (Mac version 64 bit version) or gives me an "out of memory" error (Mac version 32 bit version).  In each case, it seems that the script is gobbling up memory but not releasing it after each process is completed.  I'm running 205 total files (82 lights, 30 flats, 30 flat-darks, 30 darks and 33 bias), using about 4 GB of memory.  Any ideas?  It could just be the way PixInsight works (Juan?), since I've never really run through the entire process on this many files before without closing and restarting PixInsight for one reason or another.

Thanks, John

John,

I have the same problem on my 32 bit machine. The garbage collector in scripts is very slow. One suggestion is to reduce the buffer size in Image Integration
from the default 16M to 8M. This may give you some breathing room. You may also try splitting it into two runs, one for your cal frames and a second run
for the image frames. I don't know of any way to improve memory management in the script, if I did this would never be an issue.

Hope this helps,

John

[sjbrown]

Thanks John,

I guess I hadn't read down far enough. I just created master bias and master dark frames and got an error message at the end of execution but I believe it was about not saving the image.

Steve,
I'm not sure, but I don't think hyphens are allowed in fits image names. That could be the reason for the error.

John

[sjbrown]

...
one problem though - it seems like it does not work well when input files have the same name. for instance, each night my canon software starts over from IMG_0001.CR2, so in general i may have several files with that name. the calibration portion of the script seems to do the right thing - i'll get multiple IMG_0001 files (IMG_0001_c.fit, IMG_0001_c_1.fit, IMG_0001_c_2.fit, etc.) however at the debayer step, it seems like the files are getting overwritten because i only see filenames of the form IMG_0001_c_d.fit in that directory, and there are too few files in the directory. strangely the script must still have the idea that there are multiple IMG_0001s, because it seems to have then registered the same IMG_0001_c_d.fit several times (i see IMG_0001_c_d_r.fit and IMG_0001_c_d_r_1.fit in the registered directory).

...

Version 0.9.25 should take care of this problem. Let me know if you still have the same result.

John

[pfile]

...
one problem though - it seems like it does not work well when input files have the same name. for instance, each night my canon software starts over from IMG_0001.CR2, so in general i may have several files with that name. the calibration portion of the script seems to do the right thing - i'll get multiple IMG_0001 files (IMG_0001_c.fit, IMG_0001_c_1.fit, IMG_0001_c_2.fit, etc.) however at the debayer step, it seems like the files are getting overwritten because i only see filenames of the form IMG_0001_c_d.fit in that directory, and there are too few files in the directory. strangely the script must still have the idea that there are multiple IMG_0001s, because it seems to have then registered the same IMG_0001_c_d.fit several times (i see IMG_0001_c_d_r.fit and IMG_0001_c_d_r_1.fit in the registered directory).

...

Version 0.9.25 should take care of this problem. Let me know if you still have the same result.

John

thanks, i will give it a try.