WBPP 2.5: scripting the pipeline

robyx

Administrator
Staff member
Hi all,

This thread will provide more details about the new pipeline script introduced with WBPP 2.5.
The script can be enabled under the Pipeline tab:

WBPP_PipelineScript.png

The WBPP pipeline execution flow is depicted in the following flow chart:

1662698126408.png

As shown, the pipeline script is invoked each time a trackable operation starts and ends. The list of trackable operations is the following:
  • Calibration​
  • LPS-LDD​
  • Cosmetic Correction​
  • Debayer​
  • Measurements​
  • Weights generation​
  • Bad frames rejection​
  • Writing weights​
  • Reference frames selection​
  • Plate solving reference frames​
  • Registration​
  • Local normalization reference frame generation (both automated or interactive mode)​
  • Local normalization​
  • Integration​
  • Drizzle Integration​
  • RGB Recombination​
The pipeline script should take into consideration the following constraints:
  1. It is invoked within a dedicated scope: each local variable will run out of scope after its execution.
  2. It should be stateless: avoid trying to persist information by modifying/injecting data into any existing object, even if it's accessible from the scope code.
  3. The script code runs within a try/catch block to protect the execution integrity, so it is safe to raise an exception to terminate the script operation.
  4. Any information about the operation is embedded into an injected variable env; use it to retrieve any information about the operation.
As mentioned, if the script needs to access some information about the current operation or the associated group, it should access the env object; this variable is injected within the scope of the script before the execution and should be used to access that information. In particular, all operations provide the following properties:

  • env.event [String ]: the operation event that triggered the script. IT could be "start" or "done"
  • env.operationsCount [Number ]: the total number of operations in the pipeline
  • env.operationIndex [Number ]: the zero-based index of the current operation

Depending on the operation, more information may be available in the env object. The following list provides more details for each operation:

Operationenv.nameproperties of env object
Calibration"Calibration".name
.status
.statusMessage
[Group information]
LPS+LDD"LPS".name
.status
.statusMessage
Cosmetic correction"Cosmetic Correction".name
.status
.statusMessage
[Group information]
Debayer"Debayer".name
.status
.statusMessage
[Group information]
Measurements"Measurements".name
.status
.statusMessage
Weights generation"Weights generation".name
.status
.statusMessage
Bad frames rejection"Bad frames rejection".name
.status
.statusMessage
Weights writing"Write weights".name
.status
.statusMessage
Reference frame selection"Reference frame selection".name
.status
.statusMessage
Plate solving"Plate solving reference frames".name
.status
.statusMessage
Registration"Registration".name
.status
.statusMessage
[Group information]
Local Normalization reference frame selection"LN reference generation".name
.status
.statusMessage
[Group information]
Local Normalization reference frame selection [interactive]"LN reference [interactive]".name
.status
.statusMessage
[Group information]
Local Normalization"Local Normalization".name
.status
.statusMessage
[Group information]
Image Integration"Integration".name
.status
.statusMessage
[Group information]
Drizzle Integration"Drizzle Integration".name
.status
.statusMessage
[Group information]
RGB channel recombination"RGB Combination".name
.status
.statusMessage
[Group information]

The env.status property assumes the values defined by the OperationBlockStatus object; in particular, the following are possible:
  • OperationBlockStatus.RUNNING = 1: the operation is started
  • OperationBlockStatus.DONE = 2: the operation terminated successfully
  • OperationBlockStatus.FAILED = 3: the operation failed
  • OperationBlockStatus.CANCELED = 4; the operation has been canceled
For the operations where the group information is available, extra properties are accessible from the env object, in particular:
  • env.group.size [String]: string size in the format [width]x[height]
  • env.group.binning [Number]: the group binning
  • env.group.filter [String]: the filter name
  • env.group.exposure [Number]: the group exposure reference
  • env.group.color[String]: the group color space, it can be
    • "Mono" for monochromatic
    • "CFA" for bayered images in calibration mode
    • "RGB" for debayered images in post-calibration mode
  • env.group.mode [String]: the group mode, it could be "calibration" or "post-calibration"
  • env.group.channel[String]: non empty string indicates that the group belongs to an associated channel, in particular
    • "_R", "_G", _B": separated gray channel group
    • "combined RGB": the rgb recombination group
  • env.group.keywords [Object]: a map between the keywords name and the value associated to the group
  • env.group.frames[Array]: an array listing the active frames. An active frame is an object with the following relevant properties
    • current [String]: the filepath of the last associated file in the file history. Each time a file item is processed, the resulting output file is associated with the file item as the "current" one. For example, after the calibration, the calibrated file is associated with the file item as the "current" and this property will point to the calibrated file path. Once this file item is debayered, the debayered file will be set as the new "current" file and this property will point to the debayered file path.
    • descriptor[Object]: a descriptor contains the measurements of the file; the list or properties is the following:
      • FWHM
      • eccentricity
      • noise
      • numberOFStars
      • PSFSignalWeight
      • PSFSNR
      • SNR
      • median
      • mad
      • Mstar
    • localNormalizationFile [String]: the path of the associated local normalization file
    • drizzleFile [String]: the path of the associated drizzle file
    • isReference [Boolean]: true if the file item is a reference for star alignment
If the operation is not associated with a specific group, the env.group property remains undefined.

Below, an example of a simple script that prints the current operation and the number of frames involved:

JavaScript:
if (env.event == "start") {
   console.noteln("[SCRIPT LOG] Operation ", env.name, " started");
   if (env.group) {
      console.writeln("[SCRIPT LOG] processing " + env.group.frames.length +" frames");
   }
}

if (env.event == "done") {
   if (env.status == OperationBlockStatus.DONE)
      console.noteln("[SCRIPT LOG] ", env.name, " successfully executed");
   else if (env.status == OperationBlockStatus.FAILED)
         console.noteln("[SCRIPT LOG] ", env.name, " failed, " + env.statusMessage);
   else if (env.status == OperationBlockStatus.CENELED)
         console.noteln("[SCRIPT LOG] ", env.name, " canceled");
}

By printing to the console, the output of this script can be read in the log file. Here are some samples extracted from a log file:

1662712040841.png


1662712020765.png


1662712085691.png


In a more advanced example, we could run bash commands to integrate WBPP with virtually any system:

JavaScript:
let say = function(sentence) {
   // spd-say is a text to speech command
   ExternalProcess.execute( "spd-say", ["-w", sentence]);
   sleep(1);
}

if (env.event == "start") {
   say(env.name)
   if (env.group) {
      say("processing " + env.group.frames.length +" frames");
   }
}

if (env.event == "done") {
   if (env.status == OperationBlockStatus.DONE)
      say(env.name + " successfully executed");
   else
      if (env.status == OperationBlockStatus.FAILED)
         say(env.name  + "failed, " + env.statusMessage);
}
 

Attachments

  • 1662697642859.png
    1662697642859.png
    35.2 KB · Views: 123
  • 1662711944712.png
    1662711944712.png
    142.6 KB · Views: 134
Last edited:
I am not sure if I should post this here but I think it has to do with the latest WBPP update (2.5.0 and 2.5.1).
I am running Windows 11 with the latest updates on an Intel NUC, also with current updates. Pixinsight is also current.
Typically, after processing comet images in WBPP (2.4.5) I would load the registered images into StarAlign and output those to their respective folder, then I would use the CometAlignment module to do the same for the comet aligned images. This all worked fine with 2.4.5
When I upgraded to 2.5.0, and even today with the 2.5.1 upgrade, here is the issue I run in to with CometAlignment module.

If I load several 'registered' comet images processed by WBPP 2.5.0/2.5.1 in to CometAlignment, I can then click the first comet frame and select the comet nucleus, or a star, and it works like normal. however, when I open the last comet image the green circle is already showing on the image and I cannot move it. If I click on a star or the comet nucleus, the parameters get updated, and you can see them change with the position that you choose, but the green circle does not move and stay with the new selection. If I go back to the first comet image and select a star, then you can see the same star selected in the last comet image so the green circle appears to be following the first image. I just cannot get it to move in the last image. I can find the comet parameters and enter then in manually, but even with that if I go ahead and run the CometAlignment process, then perform ImageIntergration, the comet is blurred in that image, when before, it would be stacked on the comet nucleus.

This only occurs with comet data that I have processed with WBPP 2.5.0 or 2.5.1. If I go back to August 31, 2022 comet data, and load the comet images from the registered frames that WBPP 2.4.5 ran, I do not have this issue, meaning, the last comet image I can select a star or the comet nucleus without issue and the green circle moves and the parameters get updated with the move. And after I run it through image intergration, the comet is stacked appropriately and its not blurred.

I have tried resetting in the CometAlignment tool but it didnt help. I have tried clearing the cache in WBPP 2.5.0 and re-processing but that doesnt help. I have tried uninstall and reinstalling Pixinsight but that didnt help. I can also repeat this issue on my M1 MacBook Air. Any data that was processed with WBPP 2.4.5 does not appear to have this issue and I have tried a couple different comet data sets. If I use data that WBPP 2.5.0 processed, I get the issue in the CometAlignment module as described above. It's possible I am missing something, and I dont really know if this has to do with WBPP 2.5.0/2.5.1 or with the CometAlingment module not liking something with the new update.

Can someone confirm with comet data if they are seeing the same issue? I'd be happy to share three of four comet files if needed.
If this is a non-issue for others then I can hopefully get guidance on how to try and reset Pixinsight. Its just weird that I have no issues when I use old comet data that WBPP 2.4.5 processed.

Thank you
 
I am not sure if I should post this here but I think it has to do with the latest WBPP update (2.5.0 and 2.5.1).
I am running Windows 11 with the latest updates on an Intel NUC, also with current updates. Pixinsight is also current.
Typically, after processing comet images in WBPP (2.4.5) I would load the registered images into StarAlign and output those to their respective folder, then I would use the CometAlignment module to do the same for the comet aligned images. This all worked fine with 2.4.5
When I upgraded to 2.5.0, and even today with the 2.5.1 upgrade, here is the issue I run in to with CometAlignment module.

If I load several 'registered' comet images processed by WBPP 2.5.0/2.5.1 in to CometAlignment, I can then click the first comet frame and select the comet nucleus, or a star, and it works like normal. however, when I open the last comet image the green circle is already showing on the image and I cannot move it. If I click on a star or the comet nucleus, the parameters get updated, and you can see them change with the position that you choose, but the green circle does not move and stay with the new selection. If I go back to the first comet image and select a star, then you can see the same star selected in the last comet image so the green circle appears to be following the first image. I just cannot get it to move in the last image. I can find the comet parameters and enter then in manually, but even with that if I go ahead and run the CometAlignment process, then perform ImageIntergration, the comet is blurred in that image, when before, it would be stacked on the comet nucleus.

This only occurs with comet data that I have processed with WBPP 2.5.0 or 2.5.1. If I go back to August 31, 2022 comet data, and load the comet images from the registered frames that WBPP 2.4.5 ran, I do not have this issue, meaning, the last comet image I can select a star or the comet nucleus without issue and the green circle moves and the parameters get updated with the move. And after I run it through image intergration, the comet is stacked appropriately and its not blurred.

I have tried resetting in the CometAlignment tool but it didnt help. I have tried clearing the cache in WBPP 2.5.0 and re-processing but that doesnt help. I have tried uninstall and reinstalling Pixinsight but that didnt help. I can also repeat this issue on my M1 MacBook Air. Any data that was processed with WBPP 2.4.5 does not appear to have this issue and I have tried a couple different comet data sets. If I use data that WBPP 2.5.0 processed, I get the issue in the CometAlignment module as described above. It's possible I am missing something, and I dont really know if this has to do with WBPP 2.5.0/2.5.1 or with the CometAlingment module not liking something with the new update.

Can someone confirm with comet data if they are seeing the same issue? I'd be happy to share three of four comet files if needed.
If this is a non-issue for others then I can hopefully get guidance on how to try and reset Pixinsight. Its just weird that I have no issues when I use old comet data that WBPP 2.4.5 processed.

Thank you
Hi @midmoastro,
the only potential conflict I see is that WBPP 2.5 integrates the automated generation of the astrometric solution in the registered frames. I don't know much about the comet script but before digging into the source code, I suggest running WBPP 2.5 unchecking the astrometric solution checkbox, and seeing if this way the script works as expected.

If this is the case then we've found what is the cause and we can investigate further to see why it happens.

If this is not the case I suggest creating a dedicated thread to discuss this separately.
 
Robyx,

Thank you. Unchecking 'astrometric solution' did solve the issue on my Mac. I suspect it will on my Windows 11 PC but I think uninstalling and reinstalling might have caused some issues. If I try to run WBPP now I get 'File I/O Error: Invalid or Empty File Name'. I'll take that error to a different thread.
EDIT: I just did a 'reset' in WBPP and that fixed my other issue.
Final update: unchecking 'astrometric solution' resolved the issue I had on both Mac and Win11 PC.
Thank you again, unsure how I missed this box.

Finally, I'm sure its not said enough, thank you for all the work everyone is doing here, it is very much appreciated. Looking forward to all the new features.
 
Last edited:
Back
Top