Processing 240 images at once

[dbock]

I'm interested in performing the same processes on multiple files.
For example I have 240 images of a comet and merely wish to do a couple of processes on all 240 at once.
How do I do that in PI?

Processes. AutomaticBackgroundExtraction would be the primary function across all 240 images.
Since this one process is hands off, I would think the ability to do this across all 240 images would be a simple script. Correct?
If that is the case, I need to learn how to do that.
Any help?

If I could do a couple of other processes and have them apply to all that would be useful as well.

[Carlos Milovic]

Take a look at Process Containers and Image Containers.
http://pixinsight.com.ar/en/docs/5/pixinsight-apply-to-image-container.html
http://pixinsight.com.ar/en/docs/33/pixinsight-image-container.html
http://pixinsight.com.ar/en/docs/56/pixinsight-process-container.html
http://pixinsight.com.ar/en/docs/125/pixinsight-process-container-example.html

[dbock]

Excellent. Thank you.

[Niall Saunders]

As Carlos suggests - you shouldn't need to 'script' anything (manually).

If each image os to have the same process, or series of processes, applied to it then apply what is needed to the first image in your group. If you then look at the HistoryExplorer for that image, you will be able to grab the 'ProcessInstance' icon (the little blue left-pointing triangle) at the bottom-left of the HistoryExplorer window and drag and drop this onto your WorkSpace background.

If you now double-click this new ProcessContainer icon, the full sequence of steps will be presented to you. If you are happy with the sequence, then you are half-way there.

Now you need an ImageContainer process, loaded with the list of images that you need to process (don't include the image that you have already processed).

Finally - and slightly counter-intuitively - you need to drag the little blue triangle icon from the ProcessContainer over to the bottom line of the ImageContainer process. And, you should now see all of your images being processed.

However - what I have never tried is to actually use this method to 'save' each processed image, automatically, at the end of each process run!

[dbock]

Thanks. I'll experiment with this. I take a lot of comet sequences in a night, and wanted to clean up the raw images for all of them, the same way, before rendering a video.

[msmythers]

I used a process container for 864 subs during the Perseids in 2015. For my images I used ABE/Autohistogram/Curves. Real basic stuff. Then I used the Blink tool to make the video. Using the parameters for ffmpeg I was even able to resize the video within Blink.

https://drive.google.com/file/d/0B13r3kYqTf8NVGR1X0xCNG83QVE/view?usp=sharing

Mike

[dbock]

Yes, those are the 3 processes I use.
I just ran the ABE on 240 images, provided an output directory, but the ABE files weren't saved there.
They're currently in the active window, but not on disk.
It appears the sequence was saved, but they look the same as the raw images.
The on screen ABE files are what I want for the next step.
If I select Views to add to the image container, the ABE along with the backgrounds are listed.
So I'm figuring I have to add all the ABE's manually, and exclude the background ones.

[pfile]

you need to set up ABE like so:

discard background model
replace target image

these are checkboxes in the "target image correction" pane

rob

[dbock]

Thanks. I'm really exercising my computer right now. All 4 drives are flying with pagefile writes to manage all this data.
I have 64GB of ram, and it is fully utilize as I finish the curve transformation.
I think if I had used your suggestion, I'd have about 1/3 of the memory requirements.
It appears to be running through them though.

Thanks again for the tips.

[Niall Saunders]

Remember to let us see the results of all your efforts!

[dbock]

I finally got this done tonight. Given each frame is a single 30 second shot, and I didn't do dark frames, its a bit noisy.
However, I did learn how to run 4 or 5 processes on all 240 images, 1 process per pass of 240 at a time.

Comet 41p from April 2, 2017
240 x 30 second subs
ZWO asi071MC camera
10" f/8 RC prime focus
Processed in PixInsight 1.8
Rendered in Vegas Movie Studio HD 11

https://youtu.be/6UK-g_mS9qg

[msmythers]

Very nicely done.   


Mike

[Niall Saunders]

Well done Doug - a result well worth all the effort that you put in to it!

[dbock]

I ended up working on this 4 times, since resources were being gouged, and I didn't understand a few things along the way.
I've figured out how to do this now, 1 step at a time, and limit the computer resource usage to a minimum.
It still eats up about 250 GB of disk space to accomplish this, 50GB per pass.
Since each pass creates new files, I had to recognize that the image container from the previous pass had to be removed, and a new one created with the last passes results. Plus, I then could remove the 50 GB's of files from the previous pass input, to free up storage.
I wasn't going to run out of storage with 4TB's, but I realized the interim steps weren't needed after the 3rd pass, and just removed the previous ones.

The final video was only 180 MB in MP4 format, before uploading to YouTube.

[msmythers]

You reminded me of the one thing I did for my images. When I first started this I did a short test of 50 images and I realized the resources and speed were going to be a problem for 800+ images. Since the end video was going to be a much lower resolution then the original images and the images were only 6 second exposures I decided the first step in the process container was to to use the SampleFormatConversion tool and set every image to 8-bit integer as the first step. Not ideal but ..



Mike