PixInsight Reference Documentation — PixInsight Update Repositories

Introduction

PixInsight 1.6.9 introduced the PixInsight Update System, an open infrastructure for PixInsight software updates based on special repositories and update packages. We already have described PixInsight's update system from the end user perspective.

In this document I'll describe the structure and requirements of a PixInsight repository, the XML Repository Information format (XRI), and how to prepare and deploy update packages. This information is relevant to all developers working on the PixInsight platform, either with the PixInsight Class Library (PCL) or the PixInsight JavaScript Runtime (PJSR).

PixInsight Repositories

A PixInsight repository is just a URL from which the PixInsight Core application can retrieve information on available updates. If you already have a website, we recommend creating a dedicated subdomain for your repository; for example pixinsight.yourdomain.com. This is just what we have done to implement the official PixInsight Updates repository: update.pixinsight.com.

Communication between a repository and the PixInsight Core application requires a few conditions:

Protocols supported: HTTP and FTP. However, HTTP is simpler to implement in practice because it doesn't require authentication.
No redirections allowed. For security reasons, a PixInsight repository cannot redirect to a different URL, even if it is located on the same server.
Repository information. The repository must provide a special XML document in XRI format (described below): updates.xri. This document provides information about the repository and all the update packages that it can provide for different versions of PixInsight and supported operating systems and architectures.
Update packages must be compressed archives in one of the following formats: zip, tar.gz, tar.bz2 and 7z. Not all formats are supported on all platforms, as we'll describe later. For the sake of compatibility, the recommended formats are tar.gz and zip.

The XRI format

XRI stands for XML Repository Information. The XRI format is strict XML 1.0 encoded as UTF-8 plain text. Each PixInsight repository must provide an updates.xri file directly available at the repository URL —no redirections are allowed, as noted above.

Instead of describing the structure and syntax of an XRI document in formal terms (i.e. by means of a DTD or XSD schema), I think it is much more practical to define it with the help of some examples. For instance, the following XRI document is the updates.xri file available at updates.pixinsight.com as of the date of release of PixInsight version 1.6.9 (December 20, 2010):

<?xml version="1.0" encoding="UTF-8"?>
<xri version="1.0">
  <description>
     <p>
        PixInsight Updates — The main PixInsight repository at Pleiades Astrophoto.
     </p>
  </description>
   <platform os="all" arch="noarch" version="1.6.9:1.6.9">
      <package fileName="20101220-refdoc.tar.gz"
               sha1="5f62e6c9995333b13491eff968f95c38818173ed"
               type="documentation"
               releaseDate="20101220">
         <title>
            PixInsight Reference Documentation Snapshot - 2010 Dec 20
         </title>
         <remove>
            doc
         </remove>
         <description>
            <p>
               This update installs an initial snapshot of the PixInsight Reference Documentation
               System for PixInsight Core version 1.6.9, released December 20, 2010.
            </p>
            <p>
               The PixInsight Reference Documentation is a work in progress. We are currently
               releasing snapshots of the complete documentation system, including the set of
               existing tool documents as we are producing them. For more information on the
               PixInsight Reference Documentation System please refer to this official
               announcement thread on PixInsight Forum:
            </p>
            <p>
               http://pixinsight.com/forum/index.php?topic=2403
            </p>
            <p>
               Copyright (c) 2010 Pleiades Astrophoto S.L. All Rights Reserved.
            </p>
         </description>
      </package>
   </platform>
</xri>

With the help of this simple document we are going to describe the main XML elements recognized for the XRI 1.0 format in the following subsections.

XRI Elements

A valid XRI 1.0 document includes the following elements:

<description> ... </description>

This element should be present in updates.xri for all repositories. It provides a general description of the repository, which can include its purpose and features, the names and credits of its owner(s), and copyright information, as appropriate.

The information provided by this element can be organized as paragraphs with the HTML <p> tag, as in the example above. The full set of ISO-8859-1 HTML4 entities is supported.

<platform> ... </platform>

Defines a platform for which a set of update packages are applicable. The <platform> element is mandatory and must have the following attributes:

os = operating_system_spec

where operating_system_spec can be one of:

"any" "all" "freebsd" "linux" "macosx" "windows"

The "any" and "all" attribute values have the same meaning and denote that a set of packages are platform-independent. Usually this is the case with JavaScript scripts developed with the PJSR framework.

arch = architecture_spec

where architecture_spec must be one of:

"noarch" "any" "all" "x86" "i386" "i586" "i686" "x86_64" "x64"

The "noarch", "any" and "all" attribute values have the same meaning and denote that a set of packages are architecture-independent. The "x86", "i386", "i586" and "i686" attribute values have the same meaning and correspond to packages targeting 32-bit versions of PixInsight. "x86_64" and "x64" have the same meaning and correspond to packages targeting 64-bit versions of PixInsight.

version = version_range_spec

where version_range_spec specifies a range of versions of the PixInsight Core application to which a set of packages is applicable. version_range_spec must have the form:

"from_version : to_version"

where both from_version and to_version are version specifiers in one of the following formats:

M.m.r
M.m.r.b

where M, m, r and b are valid decimal representations of version numbers: M is a major version, m is a minor version, r is a revision number, and the optional b is a build number. If the build number b is not specified, its value is assumed to be zero for the from_version specifier and 9999 for the to_version specifier.

Examples of valid version range specifiers:

"1.6.9:1.6.9"
"1.6.9:1.6.9.644"

The first example addresses all existing builds in the 1.6.9 version of PixInsight Core. The second example provides an update only for version 1.6.9 and builds less than or equal to 644 of the Core application.

<package> ... </package>

Defines an update package. The <package> element can only occur inside a parent <platform> element; it is illegal specifying packages outside platform specifications.

The <package> element must have the following attributes:

fileName = package_file_name

where package_file_name is an URI specification to an update package archive, relative to the current repository URL.

sha1 = package_sha1_checksum

where package_sha1_checksum is the lowercase hexadecimal representation of the SHA1 digest of the package archive file. In other words, it is the output of the standard sha1sum UNIX/Linux command, or the output of the corresponding command of the openssl utility in Mac OS X. Windows users will need a file checksum utility, such as Microsoft's File Checksum Integrity Verifier (FCIV).

SHA1 checksums are used by the PixInsight Core application to verify integrity of all downloaded update packages. If the checksum is incorrect, or if the computed SHA1 digest for a downloaded package differs from the value specified in the repository, the package will not be accepted.

type = package_type_spec

where package_type_spec must be one of:

"generic" "file" "core-u" "core-c" "core-t" "corelibs" "updater" "module" "script" "documentationsystem" "documentation"

For PixInsight module files, type="module" should be specified. For script files, type="script" must be used. type="documentation" must be used for tool or script documentation files. The rest of package type specifications are reserved for official repository usage and must not be used by third-party repositories.

releaseDate = release_date_spec

where release_date_spec is a release date specification in one of the following formats:

YYYYMMDD
YYYYMMDDhh
YYYYMMDDhhmm
YYYYMMDDhhmmss

where YYYY is a four-digit year number, MM is a month number in the range [1,12], DD is a day number in the range [1,31], hh is the (optional) hour in [0,23], mm is the (optional) minute in [0,59] and ss is the (optional) second in [0,59].

The specified release date must be strictly valid; nonexisting dates (such as "20110229") will be rejected. This is to ensure that proper information will always be shown to the user.

In addition, the <package> element can have the following optional attribute:

metadata = unique_metadata_id: where unique_metadata_id is the unique identifier of a <metadata> element. Metadata elements can be used to save package descriptions when the same package must be deployed on several platforms and architectures.

The <package> element can have the following subelements:

<title> ... </title>

Specifies the package title. Package titles are used to provide brief descriptions of update packages to the user. They should not be longer than some 80 characters; ideally no more than 50 characters. Although this subelement is not mandatory, it should be included for each package to identify it uniquely in a readable form.

<remove> ... </remove>

Specifies a list of files and/or directories that will be removed completely before installing this package. The set of items to remove must be specified as a comma-separated list, and they will be interpreted as relative file or directory path specifications under the PixInsight installation directory.

This subelement can be used to perform a cleanup of unnecessary files prior to installation.

WARNING: Improper use of the <remove> package subelement can destroy the PixInsight installation on the user's machine. Always use this element after testing it on your own machine with a sandbox repository.

<description> ... </description>

Specifies a description of this package. This subelement must always be included for all update packages. The description text can be organized as a set of paragraphs with the HTML <p> element. The full set of ISO-8859-1 HTML4 entities is supported.

<metadata> ... </metadata>

This element can be used to provide metadata information shared by a set of packages. It is particularly efficient for binary packages (e.g., modules and core applications) that are to be deployed on multiple operating systems and architectures. In these cases the <metadata> element can effectively save large amounts of XML source code.

The <metadata> element must have the following attributes:

id = unique_metadata_id

where unique_metadata_id is a unique identifier associated to this <metadata> element. In case multiple <metadata> elements share the same unique identifier, only the last one seen in updates.xri will be taken into account.

The value of this attribute can be specified as the value of the metadata attribute of an existing <package> element. When this is done, the <package> element in question will inherit all the metadata items declared by this <metadata> element: release date, title, description, etc.

releaseDate = release_date_spec

where release_date_spec is a package release date specification. The syntax is the same used for the releaseDate attribute of the <package> element.

The <metadata> element can have the following subelements:

<title> ... </title>: Specifies a package title. The text defined by this element will be used as the title shared by all packages associated to this <metadata> element.
<description> ... </description>: Specifies a package description. The text defined by this element will be used as the description shared by all packages associated to this <metadata> element.

Update Packages

Packages are compressed archives that you must provide on your repository. For an update package to be available on the update system you must declare it in your updates.xri document, along with its associated metadata, file name and checksum, as we have seen in the preceding section.

Supported Formats

PixInsight update packages must always be compressed archives —uncompressed packages are not allowed and will be rejected by the update system. As of the date of releasing this document, the following compressed archive formats are supported:

Supported Package Formats
Format	FreeBSD	Linux	Mac OS X	Windows
tar.gz	Yes	Yes	Yes	Yes
tar.bz2	Yes	Yes	Yes	Yes
tar.xz	*	*	No	Yes
zip	Yes	Yes	Yes	Yes
7z	Yes	Yes	No	Yes

(*) The tar.xz archive compression will only be supported on FreeBSD and Linux if XZ compression is already available on the host machine; PixInsight does not provide support for decompressing tar.xz archives.

For the sake of compatibility, the tar.gz format is strongly recommended. To generate tar.gz archives on Windows we recommend the excellent 7-Zip freeware utility by Igor Pavlov.

Deployment Directories

An update package must contain one or more files ready to be deployed on the user's machine. This means that the update files must be correctly distributed on the same directories that the user has within his/her own PixInsight installation. The directories within an update package will always be considered relative to the PixInsight installation on the user's machine.

The following table specifies all the subdirectories relevant to deployment of update files.

Update Deployment Directories
Directory	Contents and purpose
bin	Binary files on FreeBSD, Linux and Windows. The bin directory contains the PixInsight Core application and all module shared objects (-pxm.so and -pxm.dll, respectively for FreeBSD/Linux and Windows).
MacOS	Binary files on Mac OS X. The MacOS directory contains the PixInsight Core application and all module shared objects within the PixInsight application bundle on Mac OS X (*-pxm.dylib).
include/pjsr	PJSR include files. Use to deploy JavaScript include files (*.jsh) that will be available to all PJSR-based scripts.
src/scripts	Script source code. Use to deploy the executable source code of PJSR-based scripts, including dynamically compiled C and C++ source code (when this feature is available).
doc/docs	Generic documents. Use to deploy documents not related to any particular module or script.
doc/tools	Tool documentation files. Use to deploy documentation on PCL-based tools and modules.
doc/scripts	Script documentation files. Use to deploy documentation on PJSR-based scripts.

Other directories within PixInsight's installation tree are reserved for deployment of official update packages and must not be used by third-party developers.

Important— When deploying module binaries for Mac OS X, please remember that on this operating system all binary files must be located on the MacOS installation folder, instead of the bin folder used for FreeBSD, Linux and Windows. The MacOS folder is part of the structure of the PixInsight application bundle on Mac OS X.

Package Examples

Example 1 — Module update package for Windows

Suppose you have developed a MyFooTool module that you want to distribute as an update for all supported platforms from your own PixInsight repository. For Windows, the contents of your package should be as follows:

                     + bin
                       |
                       + - MyFooTool-pxm.dll

Note that newly deployed modules are automatically installed after the update.

Example 2 — Module update package + tool documentation for Mac OS X

Now you have written some documentation for your MyFooTool, which you have authored in PIDoc format and compiled with the PIDoc compiler script to generate the corresponding XHTML documents. To deploy both your module and its documentation on Mac OS X, your update package should contain something like this:

                     + MacOS
                     | |
                     | + - MyFooTool-pxm.dylib
                     |
                     + doc
                       |
                       + tools
                         |
                         + MyFooTool
                           |
                           + images
                           | |
                           | + - Figure01.jpg
                           | + - Figure02.jpg
                           | + - Foo_screenshot.png
                           |
                           + - MyFooTool.html

In this way you can distribute an update with your module and its documentation ready to be integrated with PixInsight's reference documentation system.

Note that the doc subtree is exactly the same for all platforms and architectures; only the module's binary file must be different.

Example 3 — PJSR Script + documentation

PJSR-based scripts are platform-independent by nature. This greatly simplifies their deployment, as you don't have to bother yourself with different versions for all supported operating systems and architectures. Let's say you have written a MyBarScript.js piece of code that you want to share with the PixInsight community, along with some documentation, as in the preceding example. You module should be similar to this:

                     + src
                     | |
                     | + scripts
                     |   |
                     |   + - MyBarScript.js
                     |
                     + doc
                       |
                       + scripts
                         |
                         + MyBarScript
                           |
                           + images
                           | |
                           | + - SomethingNice.jpg
                           | + - YouMustSeeThis.jpg
                           |
                           + - MyBarScript.html

As before, what you are deploying is your script's source code along with its documentation, which will be integrated with the PixInsight reference documentation system automatically.

Note that the name of both the documentation folder and the .html documentation file must be the same specified with the #feature-id directive in your script's source code; otherwise the documentation won't be integrated and hence it won't be directly accessible from the PixInsight Core application.

As happens with new modules, newly deployed scripts are automatically installed (featured) after the update.

Testing Updates: Sandbox Repositories

Please be aware that incorrect or malformed update packages can cause a lot of problems, and even destroy or invalidate the PixInsight installation on the user's machine. Before releasing a new update, always make sure that your packages will be installed correctly. In particular, it is extremely important to ensure that your updates won't remove or replace any existing files that they should not touch.

Be aware that when you release an update, the whole PixInsight community will be able to install it by just adding your repository URL to their update repository lists. When properly used, the update system is an extremely nice and cool way to distribute your software; however, any error or lack of attention on your part can easily turn it into a nightmare.

The best way to test your updates without distributing them to all PixInsight users is by means of a sandbox repository. A sandbox repo is just an alternate URL that you should have on your website, where you upload your updates.xri document and your update packages for testing purposes. Then you add your sandbox repository URL to your update repository list in your own PixInsight installation, and test it until you are completely sure that everything works as expected.

As we have already recommended for your 'production' repository, your sandbox repository should also be a subdomain of your website. For example, you could use something like sandbox.yourdomain.com. To avoid unwanted accesses, please be sure to delete the updates.xri file in your sandbox repository, once you have finished testing it and your packages. When your updates.xri and packages are fully verified, move them to your actual repository (pixinsight.yourdomain.com or whatever you want to call it) and rely on the update system to notify all your users.

Future versions of the PixInsight update system will implement more sophisticated features for update testing and verification, as authenticated access to sandbox repositories.

Finally, something that must not be overlooked is the possibility of viruses and other malware infecting binary update files. This is particularly important on Windows, where most (all?) malware lives. Always make sure that the binaries that you distribute are virus-free.

Important— Please don't hesitate to contact us when you set up your PixInsight repository. We'll always be glad to take a look at your updates.xri document and update packages to help you in validating and testing your updates.

Future Directions

It is evident that the PixInsight update system is, in its current state, just an initial release with minimal functionality. These are some particularly important features that will be implemented in future versions:

Repository certification. Right now nothing stops you from releasing a powerful virus as a PixInsight update, which you can distribute to infect the whole PixInsight community quickly and efficiently. Of course, you must convince some PixInsight users to add your repository URL to their lists in order to do so, which is really difficult if you are a bad guy. We are well aware of the need for a reliable system for repository certification and authentication. Naturally, the certification authority for PixInsight repositories will be Pleiades Astrophoto S.L., the company that produces the PixInsight platform. We have the certification system almost ready and we'll release it as soon as possible.
Better support for testing updates. As we have noted above, future versions of the update system will implement more and more powerful tools for testing updates. In particular, sandbox repositories will be implemented with authenticated access to prevent unauthorized accesses to working data. Along with this feature, the update system will be more robust to detect and prevent common deployment errors automatically.
Better and richer XRI format. The XRI format must be extended for improved flexibility and efficiency to describe and implement updates. For example, right now we can't define package dependencies. If we are deploying a package that depends on another package being already installed, we are now forced to include both packages in the same update, even if that involves rewriting already installed files. This will change when we are able to specify package dependencies in the updates.xri document.

PixInsight Update Repositories

Contents