Access to metadata from PixInsight

[bulrichl]

In an attempt to learn more about JavaScript programming in PixInsight I had a look at the Script "BatchFormatConversion". I did not understand the following code in function FileData:

Code: [Select]

if ( outputFormat.canStoreMetadata && instance.format.canStoreMetadata )
   this.metadata = instance.metadata;
else
   this.metadata = undefined;

"instance" is the FileFormatInstance that is returned by function "this.readImage".

Currently the properties "FileFormat.canStoreMetadata" and "FileFormatInstance.metadata" do not exist according to the documentation in Object Explorer. I confirmed this by I inserting the following lines:

Code: [Select]

console.writeln(instance.format); // output: instance.format = XISF
console.writeln("instance.format.canStoreMetadata exists?\t", ("canStoreMetadata" in instance.format)); // output: instance.format.canStoreMetadata exists? false
console.writeln("instance.metadata exists?\t", (metadata in instance)); // output: instance.metadata exists? false

My question is: how does PixInsight offer a way to access the metadata?

The View Explorer shows e. g. the following 8 properties:

metadata
properties
   XISF:BlockAlignmentSize
   XISF:CreationTime
   XISF:CreatorApplication
   XISF:CreatorModule
   XISF:CreatorOS
   XISF:LoadTime
   XISF:MaxInlineBlockSize
   XISF:ResourceURL

LoadTime and ResourceURL are not stored in the XISF header but are generated during FileFormatInstance.open. The remainig 6 properties are fetched from the XISF header of the file.

Because View Explorer shows these properties I guess Pixinsight offers a way to access the metadata. The property identifiers of all 8 properties are accessible by fileFormatInstance.properties. However I only got the property identifiers and was not able to access the property types and property values in this way. I tried FileFormatInstance.readProperty(String id), but an access violation was the result.

How can I manage to get type and value of the properties?

Bernd

[Juan Conejero]

Hi Bernd,

There are no "FileFormat.canStoreMetadata" and "FileFormatInstance.metadata" properties, as you have found and demonstrated. So you've found a "strange thing" in the BatchFormatConversion script. It is innocuous but obviously useless; the "this.metadata" property will necessarily be undefined.

I have no clue as to how this code has been produced. Probably me in a moment of confusion. I'll revise this script, and will include more format data items that have been added in recent versions of PixInsight. Thank you for discovering!

LoadTime and ResourceURL are not stored in the XISF header but are generated during FileFormatInstance.open.

Yes, the XISF format support module generates these properties automatically.

How can I manage to get type and value of the properties?

These are the relevant methods of the View object:

Boolean View.hasProperty( String id )
Object|null View.propertyValue( String id )
uint|null View.propertyType( String id )

The first method returns true if a property with the specified identifier exists for a view. The other two methods provide the data type and value of an existing property (they return null if the required property does not exist). The following property of View:

Array View.properties

is an array of String with the identifiers of all properties currently defined for a View object. You can use this array to iterate view properties.

[bulrichl]

Hi Juan,

thank you very much for your quick answer.

It is innocuous but obviously useless; the "this.metadata" property will necessarily be undefined.

Sure, but  the property 'instance.metadata' doesn't exist either, because there is no FileFormatInstance.metadata; what does exist is FileFormatInstance.properties.

This seems somewhat inconsistent to me, because in the XISF files there is no 'properties' tag but a 'metadata' tag. The 'metadata' tag has a number of 'property' child tags though. Also in the XISF specification, introduction to Chapter 8, is stated: "Two high-level data objects have been defined in this XISF specification: Property and Image." And in the introduction to chapter 11: "Five main core elements have been formalized in this specification to support a number of essential components of XISF units: Property, Structure, Table, Metadata, and Image.".

Summarizing: the XISF specification and XISF files contain a 'metadata' tag, whereas the FileFormatInstance supplies a 'properties' object. After all maybe this different wording is the cause for the "strange thing" in the BatchFormatConversion Script. However, presumably this cannot be altered yet, one has to take it into consideration.
---

However, currently my main problem again is a practical programming one, i. e. the proper application of View: how is the connection between View and the object I want to retrieve type and value from to be established? I tried a little bit but have not been successful with View by now. Unfortunately I could not find a script that would clarify this question either. Please take into account that I am yet rather ignorant and unexperienced about object oriented programming...

Bernd

[bulrichl]

As Juan didn't answer my last post: please can someone of the experienced script programmers chime in and help me?

The following is a clipping from the script "BatchFormatConversion.js":

Code: [Select]

this.readImage = function( filePath )
{
   var suffix = File.extractExtension( filePath ).toLowerCase();
   var F = new FileFormat( suffix, true/*toRead*/, false/*toWrite*/ );
   if ( F.isNull )
      throw new Error( "No installed file format can read \'" + suffix + "\' files." );

   var f = new FileFormatInstance( F );
   if ( f.isNull )
      throw new Error( "Unable to instantiate file format: " + F.name );

   var d = f.open( filePath, this.inputHints );
   if ( d.length < 1 )
      throw new Error( "Unable to open file: " + filePath );
   if ( d.length > 1 )
      throw new Error( "Multi-image files are not supported by this script: " + filePath );

   var bitsPerSample = (this.bitsPerSample > 0) ? this.bitsPerSample : d[0].bitsPerSample;
   var floatSample = (this.bitsPerSample > 0) ? this.floatSample : d[0].ieeefpSampleFormat;
   var image = new Image( 1, 1, 1, ColorSpace_Gray, bitsPerSample, floatSample ? SampleType_Real : SampleType_Integer );
   if ( !f.readImage( image ) )
      throw new Error( "Unable to read file: " + filePath );

// >>> Here I want to access f.properties. <<<

   var data = new FileData( image, d[0], f, this.outputFormat );

   f.close();

   return data;
};

My Problem:
I want to access f.properties (i. e. the metadata belonging to an image file) at the marked position of the code. I only got the property identifiers and did not manage to get the property types and the property values.

Juan answered that I had to construct a View object and retrieve values and types with the methods "View.propertyValue" and "View.propertyType".

I tried to do that but when I use e. g.

Code: [Select]

var view = new View();or

Code: [Select]

var view = new View(f.properties[1]);
it doesn't work. The result is an empty object 'view' and accordingly the view.id is undefined. Obviously I am missing some basic important information relating to the usage of view, i. e., how to establish the connection between "f.properties" and the view.

I appreciate any help, maybe a short example of code can clarify it best.

Bernd

[georg.viehoever]

Hi Bernd,
the following works for me:

Code: [Select]

readImage = function( filePath )
{
   var suffix = File.extractExtension( filePath ).toLowerCase();
   var F = new FileFormat( suffix, true/*toRead*/, false/*toWrite*/ );
   if ( F.isNull )
      throw new Error( "No installed file format can read \'" + suffix + "\' files." );

   var f = new FileFormatInstance( F );
   if ( f.isNull )
      throw new Error( "Unable to instantiate file format: " + F.name );

   var d = f.open( filePath, "" );
   if ( d.length < 1 )
      throw new Error( "Unable to open file: " + filePath );
   if ( d.length > 1 )
      throw new Error( "Multi-image files are not supported by this script: " + filePath );
   Console.writeln("keywords");
   Console.writeln(f.keywords);
   return;

};

readImage("D:/Users/Public/Pictures/Astro/2016_10_01_LaPalma/20161005_Packman_Georg/light_packman (1).CR2")

Output:

Code: [Select]

run --execute-mode=auto "D:/Users/Georg/PixInsight/NewScriptFile1.js"

Processing script file: D:/Users/Georg/PixInsight/NewScriptFile1.js
keywords
{COMMENT,,Decoded with PixInsight 01.08.04.1195},{COMMENT,,Decoded with DSLR_RAW module 01.03.05.0324},{COMMENT,,Decoded with dcraw version 9.27},{INSTRUME,Canon EOS 80D,Camera model},{DATE-OBS,2016-10-05T21:44:59,Camera timestamp},{EXPTIME,29.900000,Exposure time in seconds},{ISOSPEED,800,ISO speed as specified in ISO 12232}

Georg

[bulrichl]

Hi Georg,

yes, but I was not interested in the FITS keywords but in the 'metadata'. Have a look at the following XISF header as it is in the XISF file:

Code: [Select]

<?xml version="1.0" encoding="UTF-8"?>
<!--.Extensible Image Serialization Format - XISF version 1.0.Created with PixInsight - http://pixinsight.com/.-->
<xisf version="1.0" xmlns="http://www.pixinsight.com" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.pixinsight.com http://pixinsight.com/xsd/xisf-1.0.xsd">
   <Image geometry="5344:3516:1" sampleFormat="UInt16" colorSpace="Gray" location="attachment:4096:37579008">
      <FITSKeyword name="SIMPLE" value="T" comment="file does conform to FITS standard"/>
      <FITSKeyword name="BITPIX" value="16" comment="number of bits per data pixel"/>
      <FITSKeyword name="NAXIS" value="2" comment="number of data axes"/>
      <FITSKeyword name="NAXIS1" value="5344" comment="length of data axis 1"/>
      <FITSKeyword name="NAXIS2" value="3516" comment="length of data axis 2"/>
      <FITSKeyword name="EXTEND" value="T" comment="FITS dataset may contain extensions"/>
      <FITSKeyword name="COMMENT" value="" comment="FITS (Flexible Image Transport System) format is defined in 'Astronomy"/>
      <FITSKeyword name="COMMENT" value="" comment="and Astrophysics', volume 376, page 359; bibcode: 2001A&amp;A...376..359H"/>
      <FITSKeyword name="BZERO" value="32768" comment="offset data range to that of unsigned short"/>
      <FITSKeyword name="BSCALE" value="1" comment="default scaling factor"/>
      <FITSKeyword name="PROGRAM" value="'PixInsight 01.08.04.1195'" comment="Software that created this HDU"/>
      <FITSKeyword name="COMMENT" value="" comment="PixInsight Class Library: PCL 02.01.01.0784"/>
      <FITSKeyword name="COMMENT" value="" comment="FITS module version 01.01.04.0359"/>
      <FITSKeyword name="COLORSPC" value="'Grayscale'" comment="PCL: Color space"/>
      <FITSKeyword name="RESOLUTN" value="100." comment="PCL: Resolution in pixels per resolution unit"/>
      <FITSKeyword name="RESOUNIT" value="'inch    '" comment="PCL: Resolution unit"/>
      <FITSKeyword name="COMMENT" value="" comment="Decoded with PixInsight 01.08.04.1195"/>
      <FITSKeyword name="COMMENT" value="" comment="Decoded with DSLR_RAW module 01.03.05.0324"/>
      <FITSKeyword name="COMMENT" value="" comment="Decoded with dcraw version 9.27"/>
      <FITSKeyword name="INSTRUME" value="'Canon EOS 600D'" comment="Camera model"/>
      <FITSKeyword name="DATE-OBS" value="'2016-08-08T21:45:01'" comment="Camera timestamp"/>
      <FITSKeyword name="EXPTIME" value="361." comment="Exposure time in seconds"/>
      <FITSKeyword name="ISOSPEED" value="800." comment="ISO speed as specified in ISO 12232"/>
      <FITSKeyword name="FOCALLEN" value="50." comment="Focal length in mm"/>
      <FITSKeyword name="APTDIA" value="8.84" comment="Aperture diameter in mm"/>
      <Resolution horizontal="100" vertical="100" unit="inch"/>
   </Image>
   <Metadata>
      <Property id="XISF:CreationTime" type="String">2017-03-24T13:04:33Z</Property>
      <Property id="XISF:CreatorApplication" type="String">PixInsight 01.08.04.1195</Property>
      <Property id="XISF:CreatorModule" type="String">XISF module version 01.00.06.0107</Property>
      <Property id="XISF:CreatorOS" type="String">Windows</Property>
      <Property id="XISF:BlockAlignmentSize" type="UInt16" value="4096"/>
      <Property id="XISF:MaxInlineBlockSize" type="UInt16" value="3072"/>
   </Metadata>
</xisf>


The 'Metadata' section is appended to the 'FITSKeyword' section. In a script I can get the property Ids of the metadata but do not manage to access type and value. However, in the View Explorer, the complete metadata information is represented.

So I would like to learn the correct usage of 'View'.

Bernd

[georg.viehoever]

Hi Bernd,
I think what you need to understand is the relationship between the different image related types. It is probably best understood by going through the C++ documentation (http://pixinsight.com/developer/pcl/doc/html/classpcl_1_1ImageWindow.html#details), which unfortunately also still has gaps.

- ImageWindow: The frame that you see in the PI GUI, usually related to one image file: has one or more Views
- View: One view in the ImageWindow, i.e. main view or one of the previews. Has one image, and belongs to an ImageWindow
- Image: The data structure holding the actual image data.

What you see in the View Explorer apparently is a mix of attributes provided from all 3 of these.

The script below outputs the XISF properties that you are apparently interested in, but certainly can be extended to display the other data present in the View explorer as well. Note that the script is quick and dirty: It does not properly dismiss the loaded images/views/ImageWindows.

Georg

Code: [Select]

readImage2 = function( filePath )
{
   var imageWindows=ImageWindow.open(filePath);
   if ( imageWindows.length < 1 )
      throw new Error( "Unable to open file: " + filePath );
   if ( imageWindows.length > 1 )
      throw new Error( "Multi-image files are not supported by this script: " + filePath );
   Console.writeln("ImageWindows:",imageWindows);
   var imageWindow=imageWindows[0]
   Console.writeln("ImageWindow:",imageWindow);
   Console.writeln("ImageWindow.keywords",imageWindow.keywords);
   var view=imageWindow.mainView;
   Console.writeln("view:",view);
   var properties=view.properties;
   Console.writeln("view.properties:",properties);
   var propertiesLength = properties.length;
   for(var i=0;i<propertiesLength;++i)
   {
      var name=properties[i];
      Console.writeln("properties:",name,",value:",view.propertyValue(name));
   }
   
   return;

};

readImage2("D:/Users/Georg/PixInsight/processed.xisf")
Output:

Code: [Select]

Reading image:
D:/Users/Georg/PixInsight/processed.xisf
Loading image: w=6024 h=4022 n=3 RGB Float32
29 FITS keyword(s) extracted.
ICC profile extracted: 'sRGB IEC61966-2.1', 3144 bytes.
ImageWindows:[object ImageWindow]
ImageWindow:[object ImageWindow]
ImageWindow.keywords{COMMENT,,PixInsight image preprocessing pipeline},{COMMENT,,Master frame generated with Batch Preprocessing Script v1.43},{IMAGETYP,Master Light,Type of image},{XBINNING,1,Binning factor, horizontal axis},{YBINNING,1,Binning factor, vertical axis},{COMMENT,,Integration with PixInsight 01.08.04.1195},{HISTORY,,Integration with ImageIntegration module version 01.09.04.0318},{HISTORY,,Integration with ImageIntegration process},{HISTORY,,ImageIntegration.pixelCombination: average},{HISTORY,,ImageIntegration.outputNormalization: additive + scaling},{HISTORY,,ImageIntegration.weightMode: noise evaluation},{HISTORY,,ImageIntegration.scaleEstimator: iterative k-sigma / BWMV},{HISTORY,,ImageIntegration.rangeRejection: range_low=0.000000 range_high=0.980000},{HISTORY,,ImageIntegration.pixelRejection: Winsorized sigma clipping},{HISTORY,,ImageIntegration.rejectionNormalization: scale + zero offset},{HISTORY,,ImageIntegration.rejectionClippings: low=yes high=yes},{HISTORY,,ImageIntegration.rejectionParameters: sigma_low=4.000 sigma_high=3.000},{HISTORY,,ImageIntegration.numberOfImages: 75},{HISTORY,,ImageIntegration.totalPixels: 1817139600},{HISTORY,,ImageIntegration.totalRejectedLow: 3898301(0.215%) 3923064(0.216%) 3900441(0.215},{HISTORY,,ImageIntegration.totalRejectedHigh: 60538831(3.332%) 34645967(1.907%) 75489628(4},{HISTORY,,ImageIntegration.finalNoiseEstimates: 1.7240e-005 1.6114e-005 1.8101e-005},{HISTORY,,ImageIntegration.finalScaleEstimates: 2.7189e-005 3.0254e-005 2.7818e-005},{HISTORY,,ImageIntegration.finalLocationEstimates: 1.5866e-004 1.9206e-004 1.4611e-004},{HISTORY,,ImageIntegration.referenceNoiseReductions: 1.2883 1.4614 1.2664},{HISTORY,,ImageIntegration.medianNoiseReductions: 1.2683 1.4380 1.2477},{NOISE00,1.7240e-005,Gaussian noise estimate for channel #0},{NOISE01,1.6114e-005,Gaussian noise estimate for channel #1},{NOISE02,1.8101e-005,Gaussian noise estimate for channel #2}
view:[object View]
view.properties:XISF:BlockAlignmentSize,XISF:CreationTime,XISF:CreatorApplication,XISF:CreatorModule,XISF:CreatorOS,XISF:LoadTime,XISF:MaxInlineBlockSize,XISF:ResourceURL
properties:XISF:BlockAlignmentSize,value:4096
properties:XISF:CreationTime,value:2017-04-17T15:14:04Z
properties:XISF:CreatorApplication,value:PixInsight 01.08.04.1195
properties:XISF:CreatorModule,value:XISF module version 01.00.06.0107
properties:XISF:CreatorOS,value:Windows
properties:XISF:LoadTime,value:2017-04-17T16:11:59Z
properties:XISF:MaxInlineBlockSize,value:3072
properties:XISF:ResourceURL,value:file:///D:/Users/Georg/PixInsight/processed.xisf


[bulrichl]

Hi Georg,

thanks a lot for your answer. Your script supplies the data that I was interested in and your explanation (ImageWindow, View, Image) gave me a rough idea of what I am missing. However, I am not a software developer, never programmed in C++ and therefore have so far avoided to look at the C++ documentation. On the other side PixInsight's scripting abilities are too complex for applying a trial and error method. Especially for a beginner, a systematical approach is needed: an introduction which contains an explanation of the main concepts of PixInsight and some really basic information about PixInsight's objects, methods, their usage and dependencies. The information in the object explorer is just too scarce for me, and it doesn't help much that it is ordered alphabetically and not in a more systematical way. I really would like to make progress with programming JavaScript in Pixinsight but obviously I am not able to manage the starting troubles, what a pity.

Anyway, the board 'PCL and PJSR Development' seems to be meant also for putting questions related to JavaScript programming. So I will collect some basic questions of mine and continue to send them, hoping that someone is willing to respond.

Bernd

[georg.viehoever]

Hi Bernd,

I do understand how difficult it is. Given past experience I dont have much hope that the documentation situation for PJSR and PCL will change in the near future. But if you know what to ask, you usually get an answer -- sooner or later

Georg

[bulrichl]

Hi again,

that encourages me a little bit. I also just had a look at the C++ documentation - yes, there is a bunch of descriptions. For a beginner like me, who doesn't recognize the structure of the whole, it is difficult to search at the right places and find the answers. It will take much time.

Bernd