ImageIntegration to transcribe more properties/keywords to product

[dghent]

Hi all,

While creating master bias and dark libraries for new cameras or refreshing masters for older cameras, there is one particularly repetitive and manual procedure one must do to the master frames after their creation by ImageIntegration that could possibly be automated.

When ImageIntegration produces an integrated image product, only certain keywords from the source frames are included in the product's header. INSTRUME, X/YPIXSZ, IMAGETYP, and so-on. However there are some common keywords that are critical for identifying frames such as a master bias or dark that are missing even though the information exists in all source frames for the integration:

EXPTIME
GAIN and/or EGAIN
CCD-TEMP

Additionally, some non-standard but widely-used keywords could also be included, if present in the source frames:

READOUTM (text string, readout mode name)
OFFSET (integer, histogram offset)

The lack of EXPTIME finding its way into the FITS header or XISF image properties is particularly annoying, as that prevents it from being properly categorized when loaded into batch preprocessing scripts such as WBPP. So, when I create an integrated master calibration frame of some sort, I usually add an additional finishing step of adding these keywords and their proper values using the FITSHeader tool.

I'm not sure how this could be implemented, but perhaps an additional metadata field in ImageIntegration would allow the user to arbitrarily specify properties or keywords that ImageIntegration would carry over from the source files into the integrated product it produces. There would need to be some consideration for fields with data that might vary from source frame to source frame (eg; CCD-TEMP reading -10.0 on some frames and -9.90 on others. Annoying but common). Even just allowing EXPTIME and EXPOSURE keywords to be transcribed would be helpful, as the value for those keywords/properties should be consistent across the source frames and would allow the master calibration frames to be immediately used in tools such as WBPP.

 

[Juan Conejero]

The CCD-TEMP FITS keyword as well as the Instrument:Sensor:Temperature XISF property are already being preserved by ImageIntegration. The same is true for EGAIN and Instrument:Camera:Gain. As for your suggestion about a validation range when there are very slight variations:

There would need to be some consideration for fields with data that might vary from source frame to source frame (eg; CCD-TEMP reading -10.0 on some frames and -9.90 on others.

I agree that this could be useful. I'll think how this can be implemented in a future version of the ImageIntegration tool, thank you for suggesting this feature.

Exposure time metadata is not generated in the integrated result because generating it on a regular basis would be wrong. An integrated image is not equivalent to an image acquired with an exposure time equal to the sum of exposure times in the integrated subframes, or, in case all integrated subframes have the same exposure time, equal to that exposure time. In both cases an EXPTIME keyword or an Instrument:ExposureTime property, which refer exclusively to the actual exposure time of an individual raw frame, would be wrong and misleading.

For master frames the EXPTIME keyword (not the Instrument:ExposureTime property, since its definition in the XISF standard forbids this use) is being generated by the WeightedBatchPreprocessing script for convenience (although it should not be generated for integrated light frames, IMO). For a master dark or flat frame generated manually you can encode exposure times in file names (as WBPP does for example) and/or in other metadata items that you can define ad-hoc for your specific needs, including a (loosely defined, strictly wrong) EXPTIME keyword that you can add with the FITSHeader process or with a very simple script.

READOUTM (text string, readout mode name)
OFFSET (integer, histogram offset)

We are strongly against proliferation of arbitrary nonstandard FITS keywords. There is no use or need for these keywords in PixInsight, so we have no reason to consider them.

 

[dghent]

Hey Juan, thanks for weighing in on this. I appreciate your thoughts on the topic.

I think after some reflection of the situation, it occurred to me that master/main calibration frames have insufficient metadata facilities to describe them, and this is an area (granted, far outside this specific thread yet connected to it) that could use some thoughtful development. Resorting to concocting descriptive file names that contain all the relevant information is a sign of this need and I do not think that master/main calibration frames are so exotic that they should be excluded from having some standard way to describe them in metadata.

Such a system would let the user be certain of a given frame's source and configuration and catalog it accordingly without having to decipher a file name, and it being standardized would allow tools such as WBPP to perform automatic sanity checks to ensure the specified master/main calibration frames are appropriate for the subframes they are being directed to calibrate, and emit a gripe if there is an inconsistency.

I think we are starting to see the need for this as cameras, and certainly CMOS-based cameras, are starting to sport feature sets and parameters that affect frame contents beyond the usual triad of gain, exposure time, and sensor temperature. These additional parameters must be accounted for when describing a master/main calibration frame as well, and preferably in a form of standardized metadata rather than a bespoke system.