PixInsight 1.6.1 - New FITS Format Preferences Options

[Juan Conejero]

The FITS Format Preferences dialog provides a new configuration option: Signed integer images store physical pixel data. This option is necessary to provide compatibility with some applications that store raw pixel data as signed integer FITS images. In such images, the negative half of the available numeric range (for example, from -32768 to -1 in 16-bit signed images) is not used. This decrements by one bit the effective numeric range of these images (for example, from 16 to 15 bits per sample).

Although these files are strictly valid as per the FITS standard, in our opinion storing raw pixel data as signed integer numbers is conceptually incorrect and creates compatibility problems between applications unnecessarily.

Please note that enabling this option breaks compatibility with signed integer FITS images written by PixInsight. This is because PixInsight always uses the whole available numeric range to store pixel data (for example, from -32768 to +32767 in a 16-bit signed integer image), since PixInsight stores and manipulates unsigned integer data internally. For this reason, you should enable this option only when you have to load problematic signed FITS files, and disable it as soon as you've finished working with them. For the sake of compatibility, our advice is to avoid the use of signed integer images completely: always write unsigned integer images with PixInsight.

[georg.viehoever]

Wow, there are so many new features in 1.61 that I did not even notice. Thanks for these articles!

Questions:
- is it possible to store sets of file format options? There is no blue triangle in those dialogs...
- is it possible to modify file format options from PJSR?

Georg

[bitli]

I also very much interested in the possibility to set FIST options by PJSR (I did not find a way to do it it and questions on the subject remained unanswered).  For the time being I gave up trying to use PI with remote telescope images,  as there is no way to automate the process without setting fits image options by script.
--- bitli

[Juan Conejero]

Hi Georg and bitli,

- is it possible to store sets of file format options? There is no blue triangle in those dialogs...

Nope. These are preferences dialogs, not process interfaces. However I'll try to implement a Save Settings / Load Settings feature in a future version. Thanks for the suggestion.

- is it possible to modify file format options from PJSR?

No. Module settings are write-protected and read-unprotected by default. You can freely retrieve them but you can't change them from other module or from a script. This is part of PixInsight's security system. As you know, the whole security system will change in the next version and will be much more flexible (and more secure).

I also very much interested in the possibility to set FIST options by PJSR

I don't understand why you need to modify FITS settings from a script. Changing a module settings item from a 'foreign' module or script is is in fact contrary to PixInsight's modular design. Your script should simply load FITS images (creating ImageWindow objects) and rely on the FITS module, which will apply the current settings. Only the user has the right to change settings. This is how all standard modules work (ImageCalibration, StarAlignment, ImageIntegration, ...).
 
Please let me know if I am missing/misunderstanding something. As noted above you can read module settings from your scripts. For example, try this command from the console:

j Settings.readGlobal( "/FITS/FITSBottomUp", 0 );

This will write the current state of the FITS Coordinate origin setting (true by default). In this way your script can know the orientation of FITS files, in case it needs it. This little script retrieves and prints all FITS settings on the console:

Code: [Select]

#include <pjsr/DataType.jsh>

function FITSSettings()
{
   this.lowerRange                        = Settings.readGlobal( "FITS/FITSLowerRange", DataType_Double );
   this.upperRange                        = Settings.readGlobal( "FITS/FITSUpperRange", DataType_Double );
   this.outOfRangePolicy                  = Settings.readGlobal( "FITS/FITSOutOfRangePolicy", DataType_UInt32 );
   this.outOfRangeFixMode                 = Settings.readGlobal( "FITS/FITSOutOfRangeFixMode", DataType_UInt32 );
   this.overrideICCProfileEmbedding       = Settings.readGlobal( "FITS/FITSOverrideICCProfileEmbedding", DataType_Boolean );
   this.embedICCProfiles                  = Settings.readGlobal( "FITS/FITSEmbedICCProfiles", DataType_Boolean );
   this.overrideMetadataEmbedding         = Settings.readGlobal( "FITS/FITSOverrideMetadataEmbedding", DataType_Boolean );
   this.embedMetadata                     = Settings.readGlobal( "FITS/FITSEmbedMetadata", DataType_Boolean );
   this.overrideThumbnailEmbedding        = Settings.readGlobal( "FITS/FITSOverrideThumbnailEmbedding", DataType_Boolean );
   this.embedThumbnails                   = Settings.readGlobal( "FITS/FITSEmbedThumbnails", DataType_Boolean );
   this.bottomUp                          = Settings.readGlobal( "FITS/FITSBottomUp", DataType_Boolean );
   this.unsignedIntegers                  = Settings.readGlobal( "FITS/FITSUnsignedIntegers", DataType_Boolean );
   this.writeFixedFloatKeywords           = Settings.readGlobal( "FITS/FITSWriteFixedFloatKeywords", DataType_Boolean );
   this.writeScalingKeywordsForSignedData = Settings.readGlobal( "FITS/FITSWriteScalingKeywordsForSignedData", DataType_Boolean );
   this.signedIntegersArePhysical         = Settings.readGlobal( "FITS/FITSSignedIntegersArePhysical", DataType_Boolean );
}

FITSSettings.prototype = new Object;

console.show();
var F = new FITSSettings;
for ( var i in F )
   console.writeln( i, '=', eval( "F."+i ) );

[Niall Saunders]

Hi Juan,

I think I can see where Georg and Bitli are coming from. When I think back to the problems that I had trying to read-in the crap FITS format used by the Meade DSI images, and when you pointed out 'how' to resolve the issues using the FITS Format settings, their request was certainly one of my desires at the time as well.

Basically, I only wanted to 'change' the FITS Format settings, on the fly, JUST to be able to read in the images that were going to be Batch deBayered by my PJSR script - I didn't 'really' want to change the FITS Format settings 'forever'.

I still would have a similar requirement even now - again because of the 32-bit Float format used by Meade's Envisage to save my raw data. Right now I get around the problem be creating a 'second instance' of PI - and have tweaked the config file of that second instance to allow me to get the images read in to PI, and then saved back out in a more appropriate 16-bit format. Once I have batch converted all my image-session RAW data (using ProcessContainers) I then close PI(2) and carry on working in PI(1).

So, like Georg and Bitli, I would also certainly benefit from being able to 'automate' or 'simplify' this 'change of basic setup' operation.

[georg.viehoever]

Hi Juan,

I was thinking along the same line as Niall. Concrete use cases I had in mind:
-a script that writes images as FITS files suitable for the Aladin viewer. In this case, I would like to temporarily change the FITS format options when writing the FITS file, restoring the defaults afterwards.
-a script reading .CR2 files without debayering, restoring my default debayer/black point options afterwards (I usually use the debayering as provided by PI's dcraw, but sometimes it is necessary to work on the undebayered data...)
- a script that writes a .jpg file with a predefined maximum size, changing the jpeg quality options.

There are other similar use cases that would require a temporary change of the default (or customized) file format options.
Georg

[Juan Conejero]

Hi bitli, Niall, Georg,

(BTW, welcome back Georg )

I understand your request, and can see the necessity for the features that you want. I'll think on a way to implement this functionality, but please understand that this isn't an easy task. The problem here is that we cannot allow a module or script to change any module settings permanently. For example, if your script changes settings of the DSLR_RAW module, then it should restore the original settings (the existing settings before your script runs) before terminating execution, and it should do that in a completely transparent way.

PixInsight is governed by solid software design principles and a strictly defined and maintained software architecture. I know some users don't believe in these things or don't see their value, but they are the real strength of PixInsight. Two of these principles are modularity and encapsulation. If I'd simply allow free changes to module settings from other modules, then both principles would be betrayed and that would be the beginning of a general catastrophe.

So I need to design a special interface for modules and scripts to ensure that no platform rules can be violated under any circumstance. For example, a script would gain global write access to module settings in a way similar to this:

// Request write access to global settings of the FITS module
Settings.beginGlobalChange( 'FITS' );

// Modify some global settings of the FITS module
Settings.writeGlobal( "FITS/FITSBottomUp", DataType_Boolean, false );
...
// Do your stuff here with custom modified FITS settings
...
// Finished working with custom FITS settings, inform the platform
Settings.endGlobalChange( 'FITS' );

// Now the original FITS settings have been restored

If your script terminates without the necessary calls to Settings.endGlobalChange(), an exception is thrown, the user is informed about the error, then all the original settings are restored.

This feature should also be available to modules, with equivalent static member functions of the pcl::Settings class. In the case of a module, pcl::Settings::BeginGlobalChange() and pcl::Settings::EndGlobalChange() will only be callable from reimplementations of pcl::ProcessImplementation::ExecuteOn() or pcl::ProcessImplementation::ExecuteGlobal().

That's it, more or less. This needs some work but count on it implemented in a 1.6.x version.

[georg.viehoever]

Hi Juan,

yes, I am back to the US southwest. More on that later.

Your idea of implementing this is one possible way to go. I am not sure if the same idea would work for PCL modules that want to change those settings, you probably need some kind of guard pattern there (similar to smart pointers or other means to provide for exception safe code).

Another option would be to make the temporary changes to settings part of the image.save() call, something like image.save(filename, settingModifiers).

Georg

[bitli]

Sorry, I  did not follow the forum for a while. Getting Light Bucket images on PI ended up to be to cumbersome and not scriptable enough, so I went to some other occupations.  I hope to be able to take advantage DSLR processing this winter, as there seem there was some progress in this area.

I disagree thst it is not possible load the files in the format require load because PI is modular. All operations should have as little dependencies on global context as possible and indeed should not have to modify a global value to achieve an operation. I would expect that an operation like:
 

Code: [Select]

loadFitsFile(name:'abc', maxValue:4025, ...) should be possible without impacting a global context (the global context is a set of default value of convenience to the user).

Because some operation works on file, they should accept a file descriptor (something like [name: 'abc', maxValue:4095]) instead of just a file name.

Similarly the script operations  should return some description of what was done (in an object, not as global changes). For example the name of the view that was created, or any key values that was calculated.

I do not see where this would be any less modular or less secure that using global values.
Anyhow thank for the tip on reading the parameters. Maybe I can at least check them and ask the user to change them if needed.

-- bitli