Running PixInsight processes from other apps

[mstriebeck]

Hi,

I want to use Pixinsight from another app.

More concretely: I want to write an program that optimizes the parameters in ImageIntegration (rejection algorithm, rejection parameters...) by running ImageIntegration inside an optimization loop. And once I have that, parallelize this and then use MANY examples (hope to get examples from others for this) to develop and train a model that gives recommendations.

The first step would just be to run ImageIntegration, providing images and parameter values and capturing the output.

Is this possible? Are there any examples out there?

Thanks
     Mark

[Niall Saunders]

Hi Mark,

Have you considered implementing your optimisation routine(s) from within PixInsight itself?

There are means made available to you to both as JavaScript and as external code (I know a little of the former, and virtually nothing of the latter  - however, please feel free to look at my fully documented Batch CMYG DeBayer Script if you feel that JavaScript may give you all the resources that you need).

Hope this helps.

[Niall Saunders]

Hi again Mark,

I did a quick Forum Search and came across this thread - which may also be helpful for you:

https://pixinsight.com/forum/index.php?topic=10441.msg65601#msg65601

[mstriebeck]

Thanks for the pointers Niall.

In order to run the automatic optimization and then training a model, I have to parallelize it and dispatch to many machines. That would be much easier if I just use the Pixinsight processes from within an external program (I can just link everything together to one binary).

     Mark

[Andres.Pozo]

When launching PI you can add a command line option to run a script that launches the processes that you want. Using this method your optimization program can launch several parallel instances of PI doing some tasks.

If you need to pass parameters to the script, there are several techniques that you can use. For example, your application can write the parameters in a text file so the script can read them.

I think the biggest problem is that you have to launch PixInsight  for each optimization step and this can be quite slow. I think that there is no way to launch a script in a running instance of PixInsight.

Andrés

[georg.viehoever]

Of course, you can write a Javascript script running inside PixInsight that scans the file system for parameter files, does the operation you want with those parameters and then creates a result file evaluated by the optimizer. Or you could do something via http requests etc.

In fact, that's a mode of operation used by a lot of commercial engineering tools, because many of them have quite considerable startup times, just like PI.

Georg

[vicent_peris]

Hi Mark,

If you want to populate a group of machines with PixInsight instances running specific processes, I think the best way is to write bash scripts that send the commands to each machine. This would be a good approach IMO:

First, we write a small JavaScript script to be run in each PixInsight instance. This script is simply an execution of an ImageIntegration instance with a set of known parameters:

Code: [Select]

var P = new ImageIntegration;
P.images = [ // enabled, path, drizzlePath
   [true, "/home/vicent/Image01.xisf", ""],
   [true, "/home/vicent/Image02.xisf", ""],
   [true, "/home/vicent/Image03.xisf", ""],
   [true, "/home/vicent/Image04.xisf", ""],
   [true, "/home/vicent/Image05.xisf", ""],
   [true, "/home/vicent/Image06.xisf", ""],
   [true, "/home/vicent/Image07.xisf", ""],
   [true, "/home/vicent/Image08.xisf", ""],
   [true, "/home/vicent/Image09.xisf", ""],
   [true, "/home/vicent/Image10.xisf", ""],
   [true, "/home/vicent/Image11.xisf", ""],
   [true, "/home/vicent/Image12.xisf", ""]
];
P.inputHints = "";
P.combination = ImageIntegration.prototype.Average;
P.weightMode = ImageIntegration.prototype.NoiseEvaluation;
P.weightKeyword = "";
P.weightScale = ImageIntegration.prototype.WeightScale_IKSS;
P.ignoreNoiseKeywords = false;
P.normalization = ImageIntegration.prototype.AdditiveWithScaling;
P.rejection = ImageIntegration.prototype.WinsorizedSigmaClip;
P.rejectionNormalization = ImageIntegration.prototype.Scale;
P.minMaxLow = 1;
P.minMaxHigh = 1;
P.pcClipLow = 0.200;
P.pcClipHigh = 0.100;
P.sigmaLow = 4.000;
P.sigmaHigh = 3.000;
P.linearFitLow = 5.000;
P.linearFitHigh = 2.500;
P.ccdGain = 1.00;
P.ccdReadNoise = 10.00;
P.ccdScaleNoise = 0.00;
P.clipLow = true;
P.clipHigh = true;
P.rangeClipLow = true;
P.rangeLow = 0.000000;
P.rangeClipHigh = false;
P.rangeHigh = 0.980000;
P.mapRangeRejection = true;
P.reportRangeRejection = false;
P.largeScaleClipLow = false;
P.largeScaleClipLowProtectedLayers = 2;
P.largeScaleClipLowGrowth = 2;
P.largeScaleClipHigh = false;
P.largeScaleClipHighProtectedLayers = 2;
P.largeScaleClipHighGrowth = 2;
P.generate64BitResult = false;
P.generateRejectionMaps = true;
P.generateIntegratedImage = true;
P.generateDrizzleData = false;
P.closePreviousImages = false;
P.bufferSizeMB = 16;
P.stackSizeMB = 1024;
P.useROI = false;
P.roiX0 = 0;
P.roiY0 = 0;
P.roiX1 = 0;
P.roiY1 = 0;
P.useCache = true;
P.evaluateNoise = true;
P.mrsMinDataFraction = 0.010;
P.noGUIMessages = true;
P.useFileThreads = true;
P.fileThreadOverload = 1.00;

P.executeGlobal();

Once you have this script, it can be executed within a PixInsight shell script (SCP). This script will run the JS and then save the integrated image:

Code: [Select]

run -x "/home/vicent/ImageIntegration.js"
save integration -p=/home/vicent/integrated-image.xisf --nodialog --nomessages --noverify
This SCP can be automatically executed when you launch the PixInsight instance in each machine.

You need then a bash script that regenerates the JS for each machine with the specific settings to be tested.

Finally, you can execute the SCP on each machine with the following command:

Code: [Select]

PixInsight -n --no-startup-check-updates --no-startup-gui-messages --no-splash -r=/home/vicent/PI-shell-script.scp --force-exit
If you're executing this instance on multiple machines in an automated way, you'll need to specify the DISPLAY variable value in which the PixInsight should be run. Of course, you'll need a graphical session running in each machine.


Please let me know if you need any further info.
Best regards,
Vicent.

[mstriebeck]

Thanks a lot for the detailed instructions Vicent! That looks like a good approach.

    Mark

[mstriebeck]

... making progress ...

One more question: is it possible to capture the process output?

[David Serrano]

I think the biggest problem is that you have to launch PixInsight  for each optimization step and this can be quite slow. I think that there is no way to launch a script in a running instance of PixInsight.

Maybe PI could be arranged to be launched only once, and the script would poll for the existence of the config file containing the parameters. When the file appears, the script would do its processing, delete the file and poll again. Then the user can write another file with different parameters to cause the script to fire again without the overhead of launching PI.

[mstriebeck]

I actually tried that by first starting PixInsight in the background and then executing the script. I get the output:

PixInsight Core 01.08.04.1198 Ripley (x64)
Copyright (c) 2003-2016 Pleiades Astrophoto

* Yielded execution to running application instance #1

Unfortunately, nothing happens in the running instance :-(

[vicent_peris]

... making progress ...

One more question: is it possible to capture the process output?

Yes. Type "help log" in the Process Console. You can execute the log command in the SCP file to control the output of the ImageIntegration process. Once you exit PixInsight, you can then analyze externally the log file.

[vicent_peris]

I actually tried that by first starting PixInsight in the background and then executing the script. I get the output:

PixInsight Core 01.08.04.1198 Ripley (x64)
Copyright (c) 2003-2016 Pleiades Astrophoto

* Yielded execution to running application instance #1

Unfortunately, nothing happens in the running instance :-(

PixInsight -n=2 will launch a second instance of PixInsight. Type "PixInsight --help" from the system console.

V.

[mstriebeck]

Yes. Type "help log" in the Process Console. You can execute the log command in the SCP file to control the output of the ImageIntegration process. Once you exit PixInsight, you can then analyze externally the log file.

Perfect - that's exactly what I needed. Thanks!

[mstriebeck]

PixInsight -n=2 will launch a second instance of PixInsight. Type "PixInsight --help" from the system console.

V.

Actually, what I tried to achieve was to start PixInsight in the background and then run my scripts in the running PixInsight instance (i.e. not in a new instance).