FITS PEDESTAL values in files calibrated using Maxim DL

[GrahamJC]

Master calibration frames and calibrated light frames created by Maxim DL include a pedestal value to avoid negative pixels. The pedestal value is written in the FITS header using the PEDESTAL keyword:

PEDESTAL  -100  Correction to add for zero-based ADU

Note that the default pedestal value written by Maxim is negative (-100).

When PI uses a master calibration frame it reads the pedestal value from the FITS header and *subtracts* it from each pixel. I think this is incorrect - as noted in the FITS header comments the pedestal value written by Maxim DL should be *added* to get zero-based ADU values.

I am not sure how (or if) PI uses the pedestal value in light frames but maybe a similar problem exists.

Graham

[Juan Conejero]

Our ImageCalibration module has an output pedestal parameter whose default value is zero. When this parameter is nonzero, it is added to all calibrated frames and PEDESTAL keywords are generated with positive values as follows:

PEDESTAL  100  Value in DN added to enforce positivity

The ImageCalibration and ImageIntegration tools read these keywords from input files and subtract the corresponding values if they are greater than zero. In general, pedestals are only necessary to calibrate bias and dark frames in some cases.

So our pedestal values are always positive or zero; they are always added to output frames and subtracted from input frames. Negative PEDESTAL keyword values are ignored.

We don't claim to be compatible with other applications. In general, images calibrated with other applications cannot be used with PixInsight. You should perform the whole preprocessing work (calibration, registration and integration) in PixInsight to get coherent results.

[bitli]

You should perform the whole preprocessing work (calibration, registration and integration) in PixInsight to get coherent results.

This is true but not very helpful in case you get masters from a remote telescope, you may only get the masters for calibration (which is a problem IMHO, but there is often no choice).

In practice you can subtract 200 with the PixelMath and save the images (you should be sure that this will NOT create negative values).  You may be able to use some other program (ooops...) to change the FITS keyword value to +100.  This can even be done with a good editor supporting binary mode (this is an advantage of FITS file, they can be hacked).  Admittedly not the ideal solution.

If you have access to the raw images, you should indeed use PI to make the masters.

-- bitli

[GrahamJC]

My initial post was indeed prompted by trying to use master calibration files from a remote telescope system (iTelescope in this case). In addition to the PEDESTAL problem (you can use PI to modify the FITS header) these master calibration files are also stored as 16 bit integers which is not ideal and so downloading the raw calibration files and building masters using PI is probably the best solution.

If I don't want to deal with the raw calibration files I can use the pre-calibrated files also available (these are the lights calibrated with the current set of master calibration frames). My question then relates to how PI handles the PEDESTAL keyword in light frames - the Maxim DL calibration process adds a pedestal of 100 to calibrated light frames. Would it be sufficient to simply subtract the pedestal from the calibrated light frames using pixel math?

Thanks

GRaham

[Juan Conejero]

Hi Graham,

these master calibration files are also stored as 16 bit integers which is not ideal and so downloading the raw calibration files and building masters using PI is probably the best solution.

Sure thing. 16-bit masters are indeed less than ideal—to put it mildly.

how PI handles the PEDESTAL keyword in light frames

Our ImageIntegration tool will simply ignore negative pedestal values, so you can integrate those calibrated light frames without problems. You can then subtract 100/65535 to the integrated image with PixelMath in order to remove the pedestal.

[Juan Conejero]

This is true but not very helpful in case you get masters from a remote telescope, you may only get the masters for calibration (which is a problem IMHO, but there is often no choice).

One solution to this problem would be that the people who operate those remote telescopes offered the option to calibrate the data with PixInsight. For that to happen, of course their users should request it.

In general, we don't change our tools to make them compatible with other applications. This is especially true when the changes involve adapting our tools to bad (IMO) design choices or implementations.

[bitli]

For that to happen, of course their users should request it.

Do you think I wast too shy to requrest it? 

May be you could create a log 'PixInsight compatible' and only allow remote telescope that provide proper file format to use it and, as a reciprocity, list those providers on the PixInsight site ?
Just an idea, no need to reply.
-- bitli

[Warhen]

Hey Guys, I realize this thread is very old. I was helping a client with a -100 pedestal written to the header of an SBIG camera by Bisque's TSX. Juan states 'Negative PEDESTAL keyword values are ignored.' In careful tests w/ both BPP and ImgCal, the result was improper or skipped flat correction, with donuts clearly left behind. Setting the header field to '0' before the script, or forcing '0' via ImgCal's Literal Value filed cured this. So my question- are negative values really being ignored, or are they adversely affecting calibration? As I understand it, most acquisition software doesn't write a value to the header's pedestal field. But if a negative value is there (as old SBIG protocol stated: "Add this ADU count to each pixel value to get to a zero-based ADU.  For example in SBIG images we add 100 ADU to each pixel to stop underflow at Zero ADU from noise. We would set PEDESTAL to -100 in this case.") PI seems not to flat calibrate? Apparently SBIG invented the pedestal extension? Either way, I'd love to hear from Juan on the problem we saw with the negative pedestal. Thanks!!     

[Juan Conejero]

Hi Warren,

In recent versions of PixInsight, PEDESTAL keyword values are applied irrespective of their sign.

Apparently SBIG invented the pedestal extension?

The key word here is invented: the PEDESTAL keyword is not part of the FITS standard format definition. As we interpret it, PEDESTAL should only be present in master calibration frames (typically master bias frames) to prevent negative values after bias or overscan subtraction, and only when it is really necessary. If present, the PEDESTAL keyword (or another custom, user-defined keyword that we also support for the same purpose), should have been written by a previous image calibration task, but it should never be present in uncalibrated raw frames. Since we don't claim to support images calibrated by other applications in our preprocessing tools and pipelines, we implemented the PEDESTAL keyword support as we understand it: a positive integer in DN units that has to be subtracted to get actual pixel sample values referred to zero. When the corresponding keyword is not present, the default PEDESTAL value is zero.

To summarize:

a) If the PEDESTAL keyword is present in raw frames, that is, in pure raw data that has not been previously calibrated, then this is incredibly wrong and we don't support it.

b) If the PEDESTAL keyword is present in calibrated frames or master calibration frames that have been generated by PixInsight, then the corresponding keyword values should be subtracted from pixel data to get unbiased pixel sample values, and there should be (modulo bugs) no problem at all.

c) If the PEDESTAL keyword is present in calibrated frames or master calibrated frames that have been generated by other applications, then it is a nonstandard keyword and we don't support it. We don't claim to support calibrated or calibration data generated by other applications. All master calibration frames must be generated with PixInsight in order to use PixInsight for preprocessing.

So the only practical problem is a): PEDESTAL keywords present in the headers of raw frames. As you have noted, a practical solution to this problem is quite simple: select the "literal value" option for the pedestal mode parameter of ImageCalibration (Pedestal section). Then specify 100 (or the actual PEDESTAL value, known after inspection of FITS headers) as the pedestal value parameter. Of course this assumes that all raw frames have the same value added. Unfortunately, this excludes the BatchPreprocessing script to automate the calibration and registration tasks.

Another possibility—the best one—is writing a script to subtract the PEDESTAL values and keywords from input raw frames automatically, or to fix the wrong keyword values. Such a script is very simple and small; I'll write one as soon as I get a bit of free time.

This is an excellent example of interoperability problems generated by an obsolete, poorly and loosely defined file format.

[Warhen]

Thank you Juan! I'm assuming that this negative value impacted the calibration run. As you intimate, for BPP, I was also able to set the value to zero in batch using ImageContainer and FITSHeader- performing the change to a single image, then dragging IC to FH. Thanks again! 

[AccidentalAstronomer]

Hello Juan,

I thought I'd jump in to add some clarity and context to the history of this issue for people who come across this thread in the future. I completely understand your desire not to pollute PixInsight with a bunch of special case code to accommodate a variety of programs that seem to be doing things incorrectly. I'm sure most vendors feel the same way.

You are correct of course, the PEDESTAL value is not part of the official FITS standard, but it is nearly 15 years old, as part of an SBIG extension to FITS published and adopted by most of the community in 2003:

http://diffractionlimited.com/wp-content/uploads/2016/11/sbfitsext_1r0.pdf

a) If the PEDESTAL keyword is present in raw frames, that is, in pure raw data that has not been previously calibrated, then this is incredibly wrong and we don't support it.

Image acquisition programs written to support SBIG cameras typically would adhere to the SBIG specification (above), so there is a good bit of precedent in place, and the specification is quite clear that the keyword should be added to remove the pedestal (thus the negative numbers). This was the original intent for the use of the keyword by creators of image acquisition software that worked with SBIG cameras (and to be clear, this is really not a Maxim/SkyX issue, it's an SBIG issue - because we are following their specification).

Also to be clear (a common misunderstanding), no image acquisition software is adding this value after the image is read from the camera. As far as I know we are all storing raw data from SBIG cameras as we read it.

In practice I know of no other camera vendor who has ever thought it important to embed the pedestal in the FITS header for raw data, and as far as I know all cameras do add some pedestal value. This all washes away via normal image calibration and will do so for SBIG camera acquired images if this keyword was simply ignored anyway (indeed some users have already figured out to just force the value to 0 and it works). My own opinion on this probably matches yours, which is that there was not really a good reason to include this in the first place as a bias subtract would do just a good a job of removing any arbitrary value added by the camera. Perhaps at the time SBIG thought different cameras might have different values one day in the future, but I can only guess. There are some advanced camera analysis techniques that need to know what this value is, but it could be easily provided by camera manufacturers without having to resort to a new FITS keyword. Well, hindsight is 20x20 isn't it?

It does seem for the sake of end users sanity, that the original PixInsight behavior of ignoring negative values for PEDISTAL would make all this go away though, as 15 years worth of SBIG camera images could then be calibrated with PixInsight without anyone having to find this thread, and see that the keyword is being used two different ways here, and how to work around it. I presume as well with your definition that PEDISTAL would never be negative either.

Thanks,
Richard S. Wright Jr.
Sr. Software Engineer, Software Bisque
Avid PixInsight fan

[llpastro]

Warren, is your procedure mentioned above a candidate for a new tutorial on PI Reloaded?  I would love to see that demonstrated.  I have the same -100 pedestal issue with my raw images coming from TheSkyX Camera Add-on.

Larry

[tdgm]

As the person who Warren mentioned above, I would like to say thanks for all the people who chimed in. Yes I have a solution to my problem and maybe others will need it as well. So again thanks Juan, Warren and Richard for there comments and help.


Steve Herrman Sedona AZ

[Warhen]

Hi Larry, Easy enough- if going the manual track, just set ImageCalibration's Pedestal Mode field to 'Literal Value,' and then Pedestal Value (DN) field to '0' (zero). If you prefer BPP, prior to running the script, pull all files (lights and calibration to play it safe) into the ImageContainer. After having reset the FITS PEDESTAL field to zero to one image once with the FITSHeader process, Drag the ImgContainer New Instance to the process to set all images' fields to zero. If you need help with that last bit, drop me a private line. Thanks! 

[llpastro]

Thanks Warren.  I will see if I can reproduce that workflow.

Larry