Preprocess Pipeline v0.9.0Beta

[georg.viehoever]

I know I am kind of asking for a golden door knob: Could you add some way to convert file formats where necessary (mainly in those places that do an integrate with the input). For DSLR users we otherwise get the error below:

Georg

Code: [Select]

...
############## Processing Master Bias 1 ##############

-------- Running Image Integration --------

ImageIntegration: Global context

* Empty file cache

Opening files:
E:/Benutzer/Oeffentlich/Pictures/2010_07_11_pluto_schwan/bias (1).CR2
*** Error: E:/Benutzer/Oeffentlich/Pictures/2010_07_11_pluto_schwan/bias (1).CR2: The DSLR_RAW format cannot perform incremental file read operations.

[vicent_peris]

Hi again,

I think this script needs a new functionality to be really useable. The user needs to set up predefined process settings for each type of master image. This could be a tab with a second tree, where you set up the processes. Then, each time you select a process in the windows that's already in the script, you get those predefined parameters.

One example. I have a set of images where I have a very small PSF (about 1 pixel wide), so in the registration process, I don't want to remove the hot pixels (it would remove the stars also). For each object, I have to modify this parameter from 1 to 0. This is not useful for a pipeline because you take longer to set up the processes in the pipeline than manually doing all the preprocessing.

Best regards,
Vicent.

[sjbrown]

Hmmm... This is what happens when I take a break for a few hours of sleep. I wake up and find I am waaaay behind
in my responses.

First of all, thankyou all for your responses, accolades, suggestions and problem reports. I started designing this script last April/May
with the goal of providing a tool that would be useful to everyone in the community. I believe this script, in it's current state, provides
the necessary foundation to meet this goal. I am now at a point in the development where I will need the help of other, more experienced
programmers so all you programmers out there feel free to join in.

Second, I am not all that familiar with using a forum so my responses may be slow in coming. Please bear(sp?) with me.

Now on to the responses...

[sjbrown]

Hi Juan,

- Make this thread a sticky in the New Scripts and Modules section.

How do I do this? Do I ceck the 'Lockthis topic' checkbox or is that for something else? I believe I have just earned the
right of the Newbie classification.

- Incorporate this script into the set of standard scripts in all PixInsight Core distributions.

This sounds like something only you can do. If you feel it's ready, then please do so.

- This script deserves and needs a wide distribution to the whole PixInsight community. For this purpose:


Everybody should be aware of its existence. I'll be glad to send an email circular to all PI users, pointing them to this thread.
It should be distributed through the update system, along with its documentation. Are you planning to set up your own PixInsight repository? Do you need help with this? — I'll be glad to support you, as needed.

I agree to both of these items, especially the update system. Using attachments is ugly! You do realize that after you send out the Email circular,
I will be swamped with posts to respond to! 
I do not now have a web site nor do I forsee one in the near future so your help with this will be greatly appreciated.

If you want, I can help you to make this script instantiable. It is an essential feature IMO. What do you think?

I do like this idea but at this point implementing this type of feature is beyond me. I accept your offer for help.

[Another feature that would be very nice, is an option to work in two modes: simplified and extended. In simplified mode, only a reduced set of critical parameters would be available to the user. For example, you usually don't need to tweak the RANSAC Tolerance parameter of StarAlignment, etc. In extended mode, all parameters would be available, as now happens.

Good idea. This would reduce the amount of scroling currently required. This is going onto the TODO list. Thanks.

Along with that, there are a few things of minor importance, such as some buttons not well rendered on Mac OS X (that's a common problem with OS X's Aqua style) —I can fix this for you if you want—

Please do. And if it's something that can be coded in the script, please post a code snippet so I can do the same in future scripts(yes, I have more ideas).

and the fact that all images in your documentation have .PNG file extensions, which prevents them from loading on case-sensitive platforms (all but Windows).

Would changing them all to lower case take care of the problem? I used the Snippit tool in Vista which produced the upper case extension.

One last thing before I forget, thankyou very much for the Documentatoin Compiler tool. It is an excellent addition to the developer's tool
chest and without it I would not have even tried to tackle the HTL code and style sheets(ugh). 

John

[sjbrown]

One funny bug.

I have set up a pipeline in which I calibrate, register and integrate some images of 5 different objects. For each object, different skyflats are used, and each object has 10 individual frames.

All is going well... until I see the integrated images. The first three images (the script is running right now) are "int_007_G.fit", "int_008_G.fit" and "int_010_R.fit". I load the three images in another instance of PixInsight, and I see that the three images are identical! I mean that "int_008_G.fit" and "int_010_R.fit" are clones of "int_007_G.fit".


Regards,
Vicent.

Hi Vicent,

This is perplexing. Are the Int_007_G.fit, Int_008_G.fit and Int_010_G.fit the output file names you assigned to the individual image sets?
Each image set that is added to the treeview is a new instance of the ImageSet class and therefore each image set will have a unique set of children. I can't see how the code can produce identical outputs for unique inputs.

Did you perhaps save the console output?

John

[vicent_peris]

This is perplexing. Are the Int_007_G.fit, Int_008_G.fit and Int_010_G.fit the output file names you assigned to the individual image sets?

Yes.

Did you perhaps save the console output?

++Sorry, I didn't save anything...


V.

[sjbrown]

Code: [Select]

***************** PreProcess Engine Check *****************

Checking engine state for Master Bias 1

checking input file list length...
Input file list length passed...

checking input file path validity...
Input file paths validated...

Master Bias 1 passed...

...

############## Processing Master Bias 1 ##############

Master Bias 1 process time : 0 hours   0 mins   0.000 secs


Finished processing 4 image sets.
Total process time : 0 hours   0 mins   0.000 secs
*** Error [222]: C:/PCL_Georg/scripts/PreprocessPipeline_v090Beta/PreProcessPipeline.js, line 4051: TypeError: this.rootChild[idx] is undefined

Hi Georg,

This output is telling me that you have a list of bias input files but you did not add the Integrate process to the Processes node.
If you do so it should apply the ImageIntegration module to the input files.

This is also pointing out a couple of flaws, one coding and one design. I will need to incorporate either a warning or an error to indicate
to the user that no processes have been selected for a given image set. As for the design flaw, all bias image sets will only ever use the integration
process so it would probably be a good idea to create that node automatically when adding the Master Bias image set type. I will put this in
my TODO list as well.

Try running the script again and let me know how it turns out.

Thanks for your input,

John

[sjbrown]

Quote from: sjbrown on Today at 17:31
Did you perhaps save the console output?
++Sorry, I didn't save anything...

Vicent,

Would it be possible for you to post the sets of images and calibration frames somewhere for me to download?
I would like to run them to identfy what is happening here. This is completely abherant behaviour of the script
and I need to get to the bottom of this ASAP.

Thanks,

John

[vicent_peris]

Ok, tomorrow I will upload them.


V.

[sjbrown]

I know I am kind of asking for a golden door knob: Could you add some way to convert file formats where necessary (mainly in those places that do an integrate with the input). For DSLR users we otherwise get the error below:

Georg

Code: [Select]

...
############## Processing Master Bias 1 ##############

-------- Running Image Integration --------

ImageIntegration: Global context
 
* Empty file cache

Opening files:
E:/Benutzer/Oeffentlich/Pictures/2010_07_11_pluto_schwan/bias (1).CR2
*** Error: E:/Benutzer/Oeffentlich/Pictures/2010_07_11_pluto_schwan/bias (1).CR2: The DSLR_RAW format cannot perform incremental file read operations.

Hi again Georg,

From the console output you posted it looks like you figured out what was wrong in this post:

First attempt to produce a master bias ended in the attached error. I will try again with next version  

Your request emphasizes why the script is in Beta release. In this instance I failed to think outside of my own workflow and therefore did not
anticipate presenting CR2 raws to the integration process. I always convert my CR2s before running anything on them(I use a Cannon XSi, huge files -> loooong process times ).
It should actually be very easy to implement this change and will be ncluded , hopefully, in the next release.

Thanks,

John

[sjbrown]

Ok, tomorrow I will upload them.


V.

Thanks Vicent.

[David Raphael]

I will definitely try this out...

This is exciting! 

[Harry page]

Hi

When I use the reset button I can not reslect any Item without reloading the script 

Keep up the good work

Harry

[Harry page]

Hi

I cannot get the docs to load and get the following error


Harry

[sjbrown]

Hi Harry,

When I use the reset button I can not reslect any Item without reloading the script

I have just confirmed this bug and will hunt it down in the code. It is a strange one. In the mean time the workaround is easy.
After the tree is clear, click on the root node "Preprocess Engine' and this will reactivate the menu buttons.

I cannot get the docs to load and get the following error

I think you'll need Juan's help on this. Could it be something to do with admin privelages?

Thanks for the input,

John