The NetworkService process in PixInsight 1.8.8-10

Juan Conejero

PixInsight Staff
Staff member
The NetworkService module implements the necessary software infrastructure to run PixInsight in a fully automated environment as an on-demand service. This module is at the heart of our AstroBin+PixInsight collaboration, where PixInsight is constantly providing accurate and feature-rich astrometric solutions, as well as high-quality SVG annotations and PNG finding charts, for thousands of deep-sky images. With version 1.8.8-10 of PixInsight we have released the generic NetworkService module as an open-source product to make it available to all developers on the PixInsight/PCL platform.

NetworkService implements an asynchronous, network-based, distributed client-server task processing system that can be controlled and monitored through interprocess communication (IPC) messages. The general workflow is:
  • A new task request is generated somewhere and stored in a task requests database, which is publicly accessible on the Internet through a task provider URL.
  • NetworkService is running somewhere on PixInsight and checking the task provider URL periodically.
  • When a new task request is available, a customized NetworkService process retrieves it (with secure authentication) and executes the specified task.
  • When NetworkService completes the task, it sends the corresponding data and notification to a callback URL, which has been specified in the task request.
This system is simple, flexible and, if properly implemented and deployed, secure. You can have an unlimited number of running NetworkService instances on multiple machines distributed all over the world, executing tasks that are being generated anywhere, at any time, in any order. In this document I'll describe the NetworkService communication protocol with all the necessary practical implementation details, including how to control it via IPC messages and how to customize it to perform specific tasks.


1. Task Databases

Given the asynchronous nature of the NetworkService system, the easiest and most efficient way to control this process is by means of a database to store task requests and authenticated task users. Here are templates for the structures of two MySQL tables that implement this system.

1.1 task_requests MySQL Database Table

Table name: task_requests

Table structure:
FieldNameTypeCollationNullDefault
1serial_numbervarchar(32)utf8_general_ciNoNone
2activetinyint(1)No1
3task_typevarchar(40)utf8_general_ciNoNone
4task_paramstextutf8_general_ciNo
5request_usertextutf8_general_ciNo
6request_ipvarchar(15)utf8_general_ciNoNone
7request_utcvarchar(14)utf8_general_ciNoNone
8callback_urltextutf8_general_ciNo

Table indexes:
KeynameTypeUniquePackedColumnCollationNull
PRIMARYBTREEYesNoserial_numberANo

Tasks are identified by their unique serial number, which for performance reasons should be used as the primary key for indexed table access.

A task can be either active or inactive. Active tasks are pending tasks ready to be performed. When a task is performed, it is tagged as inactive but the record is not deleted, which allows us to keep an archive of executed tasks for future reference. We also store other ancillary data items for basic server control purposes, including the IP address from which the task request was created, the request time in the UTC timescale, and the name of the user that sent the request (see the task_users table below).

Each task belongs to a unique task type which identifies the customized version of NetworkService able to perform it. The task type also classifies tasks for execution by a subset of authorized users. Each task can also have optional task-specific parameters.

Finally, the callback_url field stores the URL to which NetworkService will send back the resulting data after task completion. This will be better understood when we describe the request creation and querying scripts later.

1.2 task_users MySQL Database Table

Table name: task_users

Table structure:
FieldNameTypeCollationNullDefault
1user_nametextutf8_general_ciNo
2user_passwordvarchar(40)utf8_general_ciNoNone
3task_typetextutf8_general_ciNo

Note that passwords are stored encrypted as SHA-1 digests.

Each user is allowed to perform tasks of a specific type, except if the special wildcard task type '*' is specified as the value of the task_type field, which gives the right to execute any tasks. For example, to perform tasks where task_type="EXAMPLE_TASK", you should create a user with the following credentials:

user_name = "Example"
task_type = "EXAMPLE_TASK"

We also suggest creating a single 'root' user able to manage all tasks:

user_name = "Admin"
task_type = "*"

Important — Please use extremely secure passwords, as this service can be vulnerable otherwise. Always follow this advice, even if you are implementing a NetworkService-based system on a local network. Extremely ugly things can happen if PixInsight is exploited through this service, and you will be the sole responsible in such case.


2. Database Control Scripts

These scripts (or equivalent resources) must be available on each server supporting a tasks database. We need two scripts: one accepting new task requests to populate the task_requests table, and a second one to manage task queries. These scripts must be accessible at specific URLs and, needless to say, must be implemented and deployed with strict security requirements, or the entire service can be compromised and exploited. For this purpose the use of secure server connections is mandatory. We present here two well tested PHP scripts that we use in the continuous development of our AstroBin+PixInsight collaboration project.

In both scripts, you have to define the $MyDatabaseName, $MyDatabaseUserName and $MyDatabaseUserPassword variables with the actual values appropriate for your system. The scripts assume that two database tables exist with the names and structures described above. These scripts use SQL locks to allow for concurrent database accesses, which is an essential feature to support the asynchronous nature of the NetworkService system.

2.1 Task Request Generation Script

This script creates a new record in the task_requests database table.

new-task.php
PHP:
<?php

/******************************************************************************
 * NetworkService - Task Request Script
 *****************************************************************************/

function DieMySQLError( $dbh, $what )
{
   echo "*** MySQL Error: $what\n" . mysqli_error( $dbh ) . "\n";
   exit();
}

function DieError( $what )
{
   echo "*** Error: $what\n";
   exit();
}

function SerialNumber( $n = 32 )
{
   $K = "A9B0CD8E1F7G2H6I3J4K5L4MN5OP3Q6R2S7TU1V8W0X9YZ";
   $nK = strlen( $K ) - 1;
   $key = "";
   while ( $n-- > 0 )
      $key .= $K[round( mt_rand( 0, $nK ) )];
   return $key;
}

// ----------------------------------------------------------------------------

/*
 * Force secure connections. However, invoking this script using an insecure
 * connection generates a security risk that cannot be fixed. NEVER use this
 * script through an HTTP connection. ALWAYS use HTTPS.
 */
if ( !isset( $_SERVER["HTTPS"] ) || $_SERVER["HTTPS"] != "on" )
{
    header( "Location: https://" . $_SERVER["HTTP_HOST"] . $_SERVER["REQUEST_URI"], true, 301 );
    exit;
}

/*
 * Database name, user and password - change to actual values.
 */
$MyDatabaseName = "foo"
$MyDatabaseUserName = "bar";
$MyDatabaseUserPassword = "foobar";

/*
 * Remote connection data
 */
$ip    = $_SERVER["REMOTE_ADDR"];
$port  = $_SERVER["REMOTE_PORT"];
$agent = $_SERVER["HTTP_USER_AGENT"];
$utc   = gmdate( "YmdHis" );

/*
 * POST data
 */
$userName     = $_POST["userName"];
$userPassword = $_POST["userPassword"];
$taskType     = $_POST["taskType"];
$taskParams   = rawurldecode( $_POST["taskParams"] );
$callbackURL  = rawurldecode( $_POST["callbackURL"] );

/*
 * Validate input data
 */
if ( strlen( $userName ) < 4 || strlen( $userName ) > 32 )
   DieError( "Missing or invalid user identifier." );
for ( $i = 0; $i < strlen( $userName ); ++$i )
   if ( $userName[$i] < 'a' || $userName[$i] > 'z' )
      if ( $userName[$i] < 'A' || $userName[$i] > 'Z' )
         if ( $userName[$i] < '0' || $userName[$i] > '9' )
            DieError( "Malformed request." );
 
if ( strlen( $userPassword ) < 8 || strlen( $userPassword ) > 32 )
   DieError( "Missing or invalid user password." );
 
if ( strlen( $taskType ) < 1 || strlen( $taskType ) > 32 )
   DieError( "Missing task type." );
for ( $i = 0; $i < strlen( $taskType ); ++$i )
   if ( $taskType[$i] < 'a' || $taskType[$i] > 'z' )
      if ( $taskType[$i] < 'A' || $taskType[$i] > 'Z' )
         if ( $taskType[$i] < '0' || $taskType[$i] > '9' )
            DieError( "Malformed request." );

/*
 * Database connection
 */
$dbh = mysqli_connect( "localhost", $MyDatabaseUserName, $MyDatabaseUserPassword );
if ( !$dbh )
   DieMySQLError( $dbh, "Cannot open MySQL connection" );
if ( !mysqli_select_db( $dbh, $MyDatabaseName ) )
   DieMySQLError( $dbh, "Cannot select database" );

/*
 * Authenticate
 */
$sql = sprintf( "SELECT task_type FROM task_users WHERE user_name = '%s' AND user_password = '%s'",
                  mysqli_real_escape_string( $dbh, $userName ),
                  sha1( $userPassword ) );
$result = mysqli_query( $dbh, $sql );
if ( !$result )
   DieMySQLError( $dbh, "Unable to select a task_users record" );
if ( mysqli_num_rows( $result ) != 1 )
   DieError( "Incorrect user credentials." );
$row = mysqli_fetch_assoc( $result );
if ( $row['task_type'] != "*" )
   if ( $row['task_type'] != $taskType )
      DieError( "Task not allowed." );

/*
 * New task request
 */
$serialNumber = SerialNumber();
$sql = "INSERT INTO task_requests ( " .
                            "serial_number, " .
                            "active, " .
                            "task_type, " .
                            "task_params, " .
                            "request_user, " .
                            "request_ip, " .
                            "request_utc, " .
                            "callback_url ) " .
               "VALUES ( " .
                           "'$serialNumber', " .
                           "1, " .
                           "'$taskType', " .
                           "'$taskParams', " .
                           "'$userName', " .
                           "'$ip', " .
                           "'$utc', " .
                           "'$callbackURL' )";

$result = mysqli_query( $dbh, $sql );
if ( !$result )
   DieMySQLError( $dbh, "Unable to create a new task_requests record" );

echo "OK\n" .
     "serialNumber=$serialNumber\n";
?>

The script expects to receive the following POST variables:

userName=<name>

where <name> is the username of a registered user in the task_users table.​

userPassword=<password>

where <password> is the plain text password of the specified user.​

taskType=<type>

where <type> is the task type as registered in the task_users table.​

taskParams=<params>

where <params> is a text string specifying task-specific parameters. <params> is expected to be URL-encoded.​

callbackURL=<url>

where <url> is the callback URL to which the NetworkService process will send the resulting data upon completion of the specified task. <url> is expected to be URL-encoded (see note below).​

Upon successful execution, a new record is generated in the task_requests database table and the script outputs a newline-separated list with two items: the word "OK" and the serial number of the newly created task request. In the event of error, the script outputs human-readable error information. The caller must check if the output starts with the "OK" token to detect a successful operation.

Important — URL-encoded fields must comply with RFC 3986, which means that all characters that are not a-z, A-Z, 0-9, -, ., _ or ~ must be replaced with their percent-encoded representation %NN, where NN is a zero-padded, two-digit hexadecimal number. Note that if you are using PHP to implement POST message receivers and generators, the PHP urlencode() and urldecode() functions won't work; you should use the rawurlencode() and rawurldecode() functions instead.

2.2 Task Query Script

This script is invoked by running NetworkService processes at regular intervals to query information on pending tasks. This script must be accessible at the task provider URL that we have described above.

next-task.php
PHP:
<?php

/******************************************************************************
 * NetworkService - Task Query Script
 *****************************************************************************/

function DieMySQLError( $dbh, $what )
{
   echo "*** MySQL Error: $what\n" . mysqli_error( $dbh ) . "\n";
   exit();
}

function DieError( $what )
{
   echo "*** Error: $what\n";
   exit();
}

// ----------------------------------------------------------------------------

/*
 * Force secure connections. However, invoking this script using an insecure
 * connection generates a security risk that cannot be fixed. NEVER use this
 * script through an HTTP connection. ALWAYS use HTTPS.
 */
if ( !isset( $_SERVER["HTTPS"] ) || $_SERVER["HTTPS"] != "on" )
{
    header( "Location: https://" . $_SERVER["HTTP_HOST"] . $_SERVER["REQUEST_URI"], true, 301 );
    exit;
}

/*
 * Database name, user and password - change to actual values.
 */
$MyDatabaseName = "foo"
$MyDatabaseUserName = "bar";
$MyDatabaseUserPassword = "foobar";

/*
 * Remote connection data
 */
$ip    = $_SERVER["REMOTE_ADDR"];
$port  = $_SERVER["REMOTE_PORT"];
$agent = $_SERVER["HTTP_USER_AGENT"];
$utc   = gmdate( "YmdHis" );

/*
 * POST data
 */
$userName     = $_POST["userName"];
$userPassword = $_POST["userPassword"];
$taskType     = $_POST["taskType"];

/*
 * Validate input data
 */
if ( strlen( $userName ) < 4 || strlen( $userName ) > 32 )
   DieError( "Missing or invalid user identifier." );
for ( $i = 0; $i < strlen( $userName ); ++$i )
   if ( $userName[$i] < 'a' || $userName[$i] > 'z' )
      if ( $userName[$i] < 'A' || $userName[$i] > 'Z' )
         if ( $userName[$i] < '0' || $userName[$i] > '9' )
            DieError( "Malformed request." );
 
if ( strlen( $userPassword ) < 8 || strlen( $userPassword ) > 32 )
   DieError( "Missing or invalid user password." );
 
if ( strlen( $taskType ) < 1 || strlen( $taskType ) > 32 )
   DieError( "Missing task type." );
for ( $i = 0; $i < strlen( $taskType ); ++$i )
   if ( $taskType[$i] < 'a' || $taskType[$i] > 'z' )
      if ( $taskType[$i] < 'A' || $taskType[$i] > 'Z' )
         if ( $taskType[$i] < '0' || $taskType[$i] > '9' )
            DieError( "Malformed request." );

/*
 * Database connection
 */
$dbh = mysqli_connect( "localhost", $MyDatabaseUserName, $MyDatabaseUserPassword );
if ( !$dbh )
   DieMySQLError( $dbh, "Cannot open MySQL connection" );
if ( !mysqli_select_db( $dbh, $MyDatabaseName ) )
   DieMySQLError( $dbh, "Cannot select database" );

/*
 * Authenticate
 */
$sql = sprintf( "SELECT task_type " .
                "FROM task_users " .
                "WHERE user_name = '%s' AND user_password = '%s'",
                mysqli_real_escape_string( $dbh, $userName ), sha1( $userPassword ) );
$result = mysqli_query( $dbh, $sql );
if ( !$result )
   DieMySQLError( $dbh, "Unable to select a task_users record" );
if ( mysqli_num_rows( $result ) != 1 )
   DieError( "Incorrect user credentials." );
$row = mysqli_fetch_assoc( $result );
if ( $row['task_type'] != "*" )
   if ( $row['task_type'] != $taskType )
      DieError( "Task not allowed." );

/*
 * Protect table from possible concurrent accesses
 */
$result = mysqli_query( $dbh, "LOCK TABLES task_requests WRITE" );
if ( !$result )
   DieMySQLError( $dbh, "Unable to lock task_requests table" );

/*
 * Database query
 */
$result = mysqli_query( $dbh, "SELECT serial_number, task_type, task_params, request_user, request_utc, callback_url " .
                              "FROM task_requests " .
                              "WHERE active = 1 AND task_type = '$taskType' " .
                              "ORDER BY request_utc DESC " .
                              "LIMIT 1" );
if ( !$result )
   DieMySQLError( $dbh, "Unable to perform search in task_requests table" );

if ( mysqli_num_rows( $result ) > 0 )
{
   /*
    * Fetch request data
    */
   $row = mysqli_fetch_assoc( $result );
   $serialNumber = $row["serial_number"];
   $taskType     = $row["task_type"];
   $taskParams   = rawurlencode( $row["task_params"] );
   $requestUser  = $row["request_user"];
   $requestUTC   = $row["request_utc"];
   $callbackURL  = rawurlencode( $row["callback_url"] );

   /*
    * Disable task request
    */
   $result = mysqli_query( $dbh, "UPDATE task_requests " .
                                 "SET active = 0 " .
                                 "WHERE serial_number = '$serialNumber'" );
   if ( !$result )
      DieMySQLError( $dbh, "Unable to update task_requests table" );

   /*
    * Output request data
    */
   echo "OK\n" .
        "serialNumber=$serialNumber\n" .
        "taskType=$taskType\n" .
        "taskParams=$taskParams\n" .
        "requestUser=$requestUser\n" .
        "requestUTC=$requestUTC\n" .
        "callbackURL=$callbackURL\n";
}

mysqli_query( $dbh, "UNLOCK TABLES" );
?>

This script receives the following POST variables:

userName=<name>

where <name> is the username of a registered user in the task_users table.​

userPassword=<password>

where <password> is the verbatim, plain text password of the specified user.​

taskType=<type>

where <type> is the task type as registered in the task_users table.​

On successful execution, the script will output a newline-separated list with the following plain text items:

OK

Indicates a successful task query operation.​

serialNumber=<serial-number>

The serial number of a pending task that the caller must execute.​

taskType=<task-type>

The type of the pending task.​

taskParams=<task-params>

The parameters of the pending task. The format, encoding and meaning of <task-params> are task-specific.​

requestUser=<request-user>

The username of the user who performed the task request, that is, who generated the corresponding record in the task_requests database table.​

requestUTC=<request-utc>

The date and time the task request was generated, in ISO 8601 format.​

callbackURL=<callback-url>

The callback URL. <callback-url> is the URL where the NetworkService process will send the task's output data. The contents, encoding and format of the callback data are task-specific.​

In the event of error, the script outputs human-readable error information. The caller must check if the output starts with the 'OK' token to detect a successful operation.


3. Controlling Running Instances of PixInsight with IPC Commands

PixInsight supports up to 256 concurrent running instances of the core application on the same machine. To optimize resources, you may want to run 4 instances or perhaps more, depending on the complexity of the tasks you are going to implement and on how powerful is the hardware (in terms of CPU and RAM) where you are going to deploy a customized NetworkService process. We call instance slot the number (from 1 to 256) of a given running instance of PixInsight.

Each running instance of PixInsight can be controlled with special IPC (interprocess communication) commands. IPC commands are sent to selected application instances with specific command-line executions. You need five commands to (1) run PixInsight, (2) start, (3) stop and (4) query the state of the NetworkService process, plus one more to (5) force termination of a given instance of PixInsight. All of these actions can be performed and controlled with a shell script.

3.1 Run a New Instance of PixInsight in Automation Mode

PixInsight --automation-mode -n=<slot>

where <slot> is the instance slot (from 1 to 256) of the new running instance. If no slot is specified (just -n), the new instance will get the first free slot available.​

The --automation-mode argument enables a special working mode optimized for non-interactive or unattended execution, available since version 1.8.8-6 of PixInsight. It disables all fancy graphical effects (animations, window translucency, etc) and most informative/warning messages normally shown to the user. In automation mode, the application does not check for updates at startup, does not show a splash screen, does not show dialog boxes, and does not save any preferences settings upon termination.​
For example, to run four instances of PixInsight in a controlled way, you can do the following from a script:​

PixInsight --automation-mode -n=1
PixInsight --automation-mode -n=2
PixInsight --automation-mode -n=3
PixInsight --automation-mode -n=4

This would allow you to run four NetworkService processes at the same time on a given machine.​

3.2 Start the NetworkService Process

This one is better described with an example:

PixInsight --start-process="1:NetworkService,taskProviderURL=https://example.com/tasks/next-task.php,workerName=foo@Example,taskType=EXAMPLE_TASK,userName=Foo,userPassword=FooBar"

where:
  • 1: (optional) is the instance slot (from 1 to 256).
  • taskProviderURL (required) is the URL of a task provider, such as the next-task.php script that we have described above.
  • workerName (optional) identifies the machine where the service is running. This is not essential, but can be very useful for debugging purposes.
  • taskType (required) must be the type of the task to execute, as declared in the task_requests and task_users database tables.
  • userName and userPassword (required) must correspond to a user in your task_users table with permission to perform tasks of the specified task type. Passwords are verbatim.
Once you have started execution of a NetworkService process (or, actually, of a customized version of it), it will continue working as a background process indefinitely, calling the task provider URL at regular intervals. When a pending task is available, it will try to perform it and will send back the result to the corresponding callback URL.

3.3 Stop the NetworkService Process

PixInsight --stop-process=<slot>:NetworkService

3.4 Query State of the NetworkService Process

PixInsight --get-process-status=<slot>:NetworkService

This invocation will write some text to stdout. The text stream will end with the following fragment:

<slot>:NetworkService: <status>

where:
  • <slot> is the instance slot (from 1 to 256).
  • <status> is a status code: 0 if the process is not running, 1 if it is running normally, -1 if not running because of an error.
3.5 Terminate PixInsight Execution

PixInsight --terminate=<slot>

This will terminate the running instance at the specified <slot> unconditionally. Never send this command while the NetworkService process is still running. Send a --stop-process command and then use --get-process-status in a loop until you get a 0 or -1 status code.


4. Customizing the NetworkService Module

The NetworkService module included in the standard PixInsight distribution is a generic, baseline implementation of the NetworkService concept without any real task execution capabilities. This of course means that you must customize it. We strongly recommend that you read and study the source code of this module to understand how it works and how it can be adapted to your specific requirements.

The NetworkServiceTask class defines the prototype required for all classes implementing NetworkService tasks. By reimplementing its virtual member functions in a derived class (or modifying the same class, if preferred), you implement an actual task in a customized version of the NetworkService module. here are two relevant virtual member functions that should always be reimplemented:

String NetworkServiceTask::TaskType() const

Returns the task type, as declared in the task_requests and task_users database tables.​

void NetworkServiceTask::Perform( const TaskData& data, NetworkServiceLogger& log )

Performs a new task.​

data is a reference to a task request data structure, which describes the task request and includes task-specific parameters. See the TaskData structure for complete information.​
log is a reference to the logger object being used by the network service. A reimplementation of this member function must use this object to provide detailed information on the work performed, as well as on any errors or warning conditions that may arise. See the NetworkServiceLogger class.​

In addition, the constructor of each derived class of NetworkServiceTask must call the following static member function to register its class with the NetworkService process:

void NetworkServiceTask::RegisterTask( NetworkServiceTask* task )

where the task argument must be a pointer to the newly created object. In this way the process will be able to identity the task and invoke its Perform() member function automatically when it receives pending tasks of the corresponding type.


5. The NetworkService Graphical Interface

The NetworkService process has an associated tool window that can be very useful for testing purposes:

1637178391892.png


Of course, when NetworkService is running in an automated environment this interface is not necessary, although it can also be used in these cases to monitor ongoing tasks directly on the local machine.

____________

I hope this document will serve as an introductory reference to the new NetworkService module. Feel free to ask us if you need additional information or clarification on any aspect of the NetworkService module or its customization.
 
Last edited by a moderator:
Back
Top