Batch edit fits headers

why is it so difficult to just add a parameter and value to a fits header?

The question has been asked many times ( in this forum and elsewhere)
- obviously this is a common occurrence/need for many fits users/generators

what would be an example of the script code to add/append a field and corresponding value?
 
FITS files are made up of one or more HDUs (Header and Data Units). Each HDU must have a Header, which is made up of one or more fixed size (2880 byte) records, each containing up to 36 fixed length (80 byte) keyword records. The end of the header is indicated by an "END" keyword record.
To insert a new keyword (and it must always be inserted, since it must come before the "END" keyword), all the following keywords must shift up one 80 byte record; if there are multiple header blocks (i.e. more than 36 keywords), this will involve shifting all keywords in all blocks coming after the insertion, moving them all up one place. If this causes an overflow of the "END" record at the end of the last block, a new block must be created (and inserted in the file before the corresponding data unit). This means that inserting a new keyword may involve rewriting quite a bit of the file.
Editing a keyword is much easier; since keyword records are fixed length, they can be edited in place without changing the structure of the header.
So we have utilities for editing keywords, but in general not for adding keywords.
When in doubt, go back to the FITS standard and look.
 
It is worth understanding that the FITS format was originally designed (in 1981) as a data capture format for recording image data on sequential magnetic tape storage. At that time there was no intent (or capability) to edit the data. It was not designed as an editable intermediate image format. It is generally wise to recognise this by using it only as an input format.
 
Last edited:
I have written a batch script which allows keywords to be edited, added or removed from fits files. It is my first script and I have not tested it thoroughly so I can only offer it "as is". I am an ASIAIR user so I have used it to add filter information successfully . Also I use a MacBook Pro so have not tested on any other OS. However, there seems to be a demand for something like this so I am happy to share. Please let me know any comments/feedback/bugs.
 

Attachments

  • BatchFITSKeywordEdit.js.zip
    9.8 KB · Views: 758
I knew it must be possible to script this in PI, since PI often modifies the FITS header content, but I'm not surprised that it takes more than 1000 lines of code to do it! I've tried a few quick tests and it seems to work fine (1.8.8-9 on Windows 10 Pro)!
 
One observation: a value added as an integer (e.g. "1") is inserted as "1.". The FITS value format standard is ambiguous here: "1" could be integer or float, but "1." is definitely float. While it is not a major issue, I suggest that if a value is entered as a valid FITS integer, the integer format should be retained. Arguably, a utility like this should just replicate the input value string (provided it comprises only legal characters), since the type will in general be unknown (who knows whether or not leading / trailing zeros, signs etc. are significant for this particular keyword).
 
Last edited:
fredvanner said:
I knew it must be possible to script this in PI, since PI often modifies the FITS header content, but I'm not surprised that it takes more than 1000 lines of code to do it! I've tried a few quick tests and it seems to work fine (1.8.8-9 on Windows 10 Pro)!
Click to expand...
Thanks for the feedback Fred. The user interface adds a lot of those lines of code!
 
fredvanner said:
One observation: a value added as an integer (e.g. "1") is inserted as "1.". The FITS value format standard is ambiguous here: "1" could be integer or float, but "1." is definitely float. While it is not a major issue, I suggest that if a value is entered as a valid FITS integer, the integer format should be retained. Arguably, a utility like this should just replicate the input value string (provided it comprises only legal characters), since the type will in general be unknown.
Click to expand...
Thanks for this comment Fred. I agree your point. My code uses the Pixinsight routine for writing the new FITS data to file and I think this behaviour is a function of that routine. I am reluctant to reinvent the wheel just to cover this point!
 
Mike1485 said:
I think this behaviour is a function of that routine.
Click to expand...
I believe that you are right. PI outputs a number of other integer parameters with this floating format. I guess that as long as only PI is going to process the value this doesn't matter. Hopefully, most 3rd party software will be resilient to this minor issue.
 
The FITS format does not distinguish between integers and floating point numbers for keyword values. They are just 'numeric' it they are not 'alphanumeric' or 'logical'. All FITS files are written directly by the CFITSIO library in PixInsight, including their header keywords.

EDIT - Sorry, yes, in sections 4.2.3 and 4.2.4 version 4 of the standard makes a distinction. However, the difference is merely 'aesthetic' and 1. and 1 are actually representing the same value.
 
Juan Conejero said:
The FITS format does not distinguish between integers and floating point numbers for keyword values. They are just 'numeric' it they are not 'alphanumeric' or 'logical'. All FITS files are written directly by the CFITSIO library in PixInsight, including their header keywords.

EDIT - Sorry, yes, in sections 4.2.3 and 4.2.4 version 4 of the standard makes a distinction. However, the difference is merely 'aesthetic' and 1. and 1 are actually representing the same value.
Click to expand...
Thanks for the clarification Juan.
 
Mike1485 said:
Please let me know any comments/feedback/bugs.
Click to expand...
I appreciate that a Batch FITS keyword editor has been programmed. I briefly tested your script (Windows 10 Pro, PixInsight 1.8.8-9). It takes into account that HDUs require multiples of 2880 Bytes (this was apparently an obstacle for other trials to implement the addition of keywords).

However, the script changes some things that were not asked for. I wanted to just add one FITS keyword:

XPIXSZ 3.78 Pixel Width in microns (with binning)

to a FITS file that already contained 35 FITS keywords. The resulting FITS file contains 46 FITS keywords, and is larger by 112600 Bytes (a thumbnail image) than the original file. The following keywords were added:

EXTEND (together with 2 COMMENT keywords)
PROGRAM (together with 2 COMMENT keywords)
COLORSPC
RESOLUTN
RESOUNIT
THUMBIMG
ROWORDER

Besides the keyword CREATOR, which was written by the capture software, was deleted.

In my view it is not negative that software which is used to modify files leaves its mark (PROGRAM and 2 COMMENT keywords). I don't mind that the keyword COLORSPC is added as well. However, the keywords RESOLUTN and RESOUNIT are unnecessary, and it was not required at all to append a thumbnail image. I understand that the keyword ROWORDER should be written only by the capture software. Finally, in no case a keyword should be deleted unintentionally.

Otherwise, the script seems to be very useful to me. Perhaps you can consider my request for changes?

Bernd
 
bulrichl said:
Perhaps you can consider my request for changes?
Click to expand...

Bernd - thanks for your comments.

The code uses the Pixinsight routine for writing the new FITS data to file. This Pixinsight routine adds the keywords as you have noted and also deletes the CREATOR keyword. Even if I were to change my script to ensure these things are not changed (which would be a non-trivial change!) then as soon as the file was processed in any other way in Pixinsight these changes would likely be made in any case. Hopefully the script will be useful despite these issues.
 
Ran into the problem of missing FILTER keywords after a recent software upgrade. Then found this script. Does exactly do the job. Fantastic! Thank you!
 
You must log in or register to reply here.
Back
Top