Batch edit fits headers

snolan

New member
Aug 3, 2019
2
0
why is it so difficult to just add a parameter and value to a fits header?

The question has been asked many times ( in this forum and elsewhere)
- obviously this is a common occurrence/need for many fits users/generators

what would be an example of the script code to add/append a field and corresponding value?
 

fredvanner

Well-known member
Apr 17, 2019
2,492
472
71
Wells, Somerset, UK
FITS files are made up of one or more HDUs (Header and Data Units). Each HDU must have a Header, which is made up of one or more fixed size (2880 byte) records, each containing up to 36 fixed length (80 byte) keyword records. The end of the header is indicated by an "END" keyword record.
To insert a new keyword (and it must always be inserted, since it must come before the "END" keyword), all the following keywords must shift up one 80 byte record; if there are multiple header blocks (i.e. more than 36 keywords), this will involve shifting all keywords in all blocks coming after the insertion, moving them all up one place. If this causes an overflow of the "END" record at the end of the last block, a new block must be created (and inserted in the file before the corresponding data unit). This means that inserting a new keyword may involve rewriting quite a bit of the file.
Editing a keyword is much easier; since keyword records are fixed length, they can be edited in place without changing the structure of the header.
So we have utilities for editing keywords, but in general not for adding keywords.
When in doubt, go back to the FITS standard and look.
 

fredvanner

Well-known member
Apr 17, 2019
2,492
472
71
Wells, Somerset, UK
It is worth understanding that the FITS format was originally designed (in 1981) as a data capture format for recording image data on sequential magnetic tape storage. At that time there was no intent (or capability) to edit the data. It was not designed as an editable intermediate image format. It is generally wise to recognise this by using it only as an input format.
 
Last edited:

Mike1485

Well-known member
Jun 15, 2021
126
62
I have written a batch script which allows keywords to be edited, added or removed from fits files. It is my first script and I have not tested it thoroughly so I can only offer it "as is". I am an ASIAIR user so I have used it to add filter information successfully . Also I use a MacBook Pro so have not tested on any other OS. However, there seems to be a demand for something like this so I am happy to share. Please let me know any comments/feedback/bugs.
 

Attachments

fredvanner

Well-known member
Apr 17, 2019
2,492
472
71
Wells, Somerset, UK
I knew it must be possible to script this in PI, since PI often modifies the FITS header content, but I'm not surprised that it takes more than 1000 lines of code to do it! I've tried a few quick tests and it seems to work fine (1.8.8-9 on Windows 10 Pro)!
 

fredvanner

Well-known member
Apr 17, 2019
2,492
472
71
Wells, Somerset, UK
One observation: a value added as an integer (e.g. "1") is inserted as "1.". The FITS value format standard is ambiguous here: "1" could be integer or float, but "1." is definitely float. While it is not a major issue, I suggest that if a value is entered as a valid FITS integer, the integer format should be retained. Arguably, a utility like this should just replicate the input value string (provided it comprises only legal characters), since the type will in general be unknown (who knows whether or not leading / trailing zeros, signs etc. are significant for this particular keyword).
 
Last edited:

Mike1485

Well-known member
Jun 15, 2021
126
62
I knew it must be possible to script this in PI, since PI often modifies the FITS header content, but I'm not surprised that it takes more than 1000 lines of code to do it! I've tried a few quick tests and it seems to work fine (1.8.8-9 on Windows 10 Pro)!
Thanks for the feedback Fred. The user interface adds a lot of those lines of code!
 

Mike1485

Well-known member
Jun 15, 2021
126
62
One observation: a value added as an integer (e.g. "1") is inserted as "1.". The FITS value format standard is ambiguous here: "1" could be integer or float, but "1." is definitely float. While it is not a major issue, I suggest that if a value is entered as a valid FITS integer, the integer format should be retained. Arguably, a utility like this should just replicate the input value string (provided it comprises only legal characters), since the type will in general be unknown.
Thanks for this comment Fred. I agree your point. My code uses the Pixinsight routine for writing the new FITS data to file and I think this behaviour is a function of that routine. I am reluctant to reinvent the wheel just to cover this point!
 

fredvanner

Well-known member
Apr 17, 2019
2,492
472
71
Wells, Somerset, UK
I think this behaviour is a function of that routine.
I believe that you are right. PI outputs a number of other integer parameters with this floating format. I guess that as long as only PI is going to process the value this doesn't matter. Hopefully, most 3rd party software will be resilient to this minor issue.
 
  • Like
Reactions: Mike1485

Juan Conejero

PixInsight Staff
Sep 2, 2004
8,693
682
58
Valencia, Spain
pixinsight.com
The FITS format does not distinguish between integers and floating point numbers for keyword values. They are just 'numeric' it they are not 'alphanumeric' or 'logical'. All FITS files are written directly by the CFITSIO library in PixInsight, including their header keywords.

EDIT - Sorry, yes, in sections 4.2.3 and 4.2.4 version 4 of the standard makes a distinction. However, the difference is merely 'aesthetic' and 1. and 1 are actually representing the same value.
 

Mike1485

Well-known member
Jun 15, 2021
126
62
The FITS format does not distinguish between integers and floating point numbers for keyword values. They are just 'numeric' it they are not 'alphanumeric' or 'logical'. All FITS files are written directly by the CFITSIO library in PixInsight, including their header keywords.

EDIT - Sorry, yes, in sections 4.2.3 and 4.2.4 version 4 of the standard makes a distinction. However, the difference is merely 'aesthetic' and 1. and 1 are actually representing the same value.
Thanks for the clarification Juan.
 

bulrichl

PTeam Member
Nov 2, 2016
1,444
226
La Palma, Canary Islands
Please let me know any comments/feedback/bugs.
I appreciate that a Batch FITS keyword editor has been programmed. I briefly tested your script (Windows 10 Pro, PixInsight 1.8.8-9). It takes into account that HDUs require multiples of 2880 Bytes (this was apparently an obstacle for other trials to implement the addition of keywords).

However, the script changes some things that were not asked for. I wanted to just add one FITS keyword:

XPIXSZ 3.78 Pixel Width in microns (with binning)

to a FITS file that already contained 35 FITS keywords. The resulting FITS file contains 46 FITS keywords, and is larger by 112600 Bytes (a thumbnail image) than the original file. The following keywords were added:

EXTEND (together with 2 COMMENT keywords)
PROGRAM (together with 2 COMMENT keywords)
COLORSPC
RESOLUTN
RESOUNIT
THUMBIMG
ROWORDER


Besides the keyword CREATOR, which was written by the capture software, was deleted.

In my view it is not negative that software which is used to modify files leaves its mark (PROGRAM and 2 COMMENT keywords). I don't mind that the keyword COLORSPC is added as well. However, the keywords RESOLUTN and RESOUNIT are unnecessary, and it was not required at all to append a thumbnail image. I understand that the keyword ROWORDER should be written only by the capture software. Finally, in no case a keyword should be deleted unintentionally.

Otherwise, the script seems to be very useful to me. Perhaps you can consider my request for changes?

Bernd
 
  • Like
Reactions: Mike1485

Mike1485

Well-known member
Jun 15, 2021
126
62
Perhaps you can consider my request for changes?
Bernd - thanks for your comments.

The code uses the Pixinsight routine for writing the new FITS data to file. This Pixinsight routine adds the keywords as you have noted and also deletes the CREATOR keyword. Even if I were to change my script to ensure these things are not changed (which would be a non-trivial change!) then as soon as the file was processed in any other way in Pixinsight these changes would likely be made in any case. Hopefully the script will be useful despite these issues.
 

wjdrijfhout

New member
Apr 8, 2019
3
1
Ran into the problem of missing FILTER keywords after a recent software upgrade. Then found this script. Does exactly do the job. Fantastic! Thank you!
 
  • Like
Reactions: Mike1485