Negative PEDESTAL values handling

duncankitchin

Well-known member
Jan 11, 2015
55
0
Beaverton, Oregon, USA
Just wanted to write something up concerning an issue that I've been having (now apparently fixed) and what I think the cause was, and see if anybody else has seen this. In the process of trying to figure out what was going on, I found a lot of (old) threads related to SBIG cameras, negative PEDESTAL values and FITS files generated by TheSkyX. I happen to be using an STL11k and TSX...

The issue first showed up when re-processing some data that I had from last summer's Oregon Star Party. This had previously been processed without issue, but when I reprocessed I found the calibration was very wrong. Specifically, the flats were all messed up and "over correcting", that is, I had white corners in the calibrated lights. This was very odd, since I have a well-debugged PI calibration workflow that I've been using without trouble for years, including on the exact same data. Eventually I worked around the problem temporarily by going back and pulling my master flats that I built last year from the same data.

They worked fine; but my newly generated master flats from the exact same data and created using the exact same process steps were no longer working. I have been trying to figure out what is going on for weeks...

It turned out to be an issue with the new behavior in handling negative PEDESTAL values, in that by default it is accounted for in calibration, when previously it was just ignored.

It happens that just ignoring it in all frames works fine with my calibration flow - it just comes out in the wash as an additional component of bias. It would also have worked fine if it was accounted for in integration as well as calibration, but ImageIntegration seems to still ignore it.

My process is to directly integrate bias and darks to get a master bias and dark, and then calibrate the individual flats. That way, when I do an integration of the flats, they are already zero-based and I can therefore make use of multiplicative normalization and "equalize fluxes" for rejection normalization. When I do light calibration, I just need to be careful to set the "calibrate" checkboxes appropriately to account for the fact that the master dark has not been calibrated but the master flat has.

My process is all working again now - but I need to set the pedestal mode to "literal value" and set the value to 0 when calibrating both flats and lights.

Has anybody else seen this, and does what I'm seeing sound correct?
 

Juan Conejero

PTeam Member
Sep 2, 2004
7,555
96
56
Valencia, Spain
pixinsight.com
In the latest versions of PixInsight, since at least one year or perhaps more (I can't remember the exact numbers now), PEDESTAL keyword values are always evaluated as their absolute values, that is, ignoring signs. This is because—another interoperability problem caused by FITS—some applications write PEDESTAL keywords to be added (negative) and others to be subtracted (positive). Ignoring the sign is a compatibility improvement here, and it's safe because all PEDESTAL values have (or should have) been added when they are present in FITS headers, so we can simply subtract their absolute values to get zero-based pixel data consistently.

It turned out to be an issue with the new behavior in handling negative PEDESTAL values, in that by default it is accounted for in calibration, when previously it was just ignored.
The old behavior, i.e. ignoring negative PEDESTAL keywords, was incorrect. PEDESTAL values, when present, must always be taken into account and their absolute values must be subtracted for calibration.

So the problem you are describing should not happen if you calibrate the whole data set with the current 1.8.8-5 version of PixInsight. If you have deleted the necessary raw files (bad practice!), then you'll have to find a workaround as you have described.

It would also have worked fine if it was accounted for in integration as well as calibration, but ImageIntegration seems to still ignore it.
Which version of PixInsight are you using? Several versions ago I implemented a subtract pedestals option in ImageIntegration, which is enabled by default. So no, ImageIntegration does not ignore PEDESTAL keywords in current versions.
 
  • Like
Reactions: duncankitchin

bulrichl

PTeam Member
Nov 2, 2016
831
58
La Palma, Canary Islands
For the integration of bias frames, dark frames or flat darks to the masters, the 'Subtract pedestal' option must be disabled, thus maintaining the pedestal in the masters. The pedestals will be removed at the beginning of the calibration of flat and light frames respectively.

Bernd
 
  • Like
Reactions: duncankitchin

Juan Conejero

PTeam Member
Sep 2, 2004
7,555
96
56
Valencia, Spain
pixinsight.com
For the integration of bias frames, dark frames or flat darks to the masters, the 'Subtract pedestal' option must be disabled, thus maintaining the pedestal in the masters. The pedestals will be removed at the beginning of the calibration of flat and light frames respectively.
That's correct. For this reason the subtract pedestals option is exposed to the user and hence optional in ImageIntegration. I added this option because there are many users that have no access to uncalibrated light frames, and have to use data already calibrated with applications that generate PEDESTAL keywords systematically.

In the next version this option will be disabled by default.
 

duncankitchin

Well-known member
Jan 11, 2015
55
0
Beaverton, Oregon, USA
Juan,
thanks for the reply - much appreciated

The old behavior, i.e. ignoring negative PEDESTAL keywords, was incorrect. PEDESTAL values, when present, must always be taken into account and their absolute values must be subtracted for calibration.
yes, understood. My workflow was originally developed using a QSI camera for which this never arose, and then when I added the SBIG/TSX generated files with the negative PEDESTAL values, it happened to still work, because the data offset was in the master bias and got correctly subtracted everywhere.

So the problem you are describing should not happen if you calibrate the whole data set with the current 1.8.8-5 version of PixInsight. If you have deleted the necessary raw files (bad practice!), then you'll have to find a workaround as you have described.

Which version of PixInsight are you using? Several versions ago I implemented a subtract pedestals option in ImageIntegration, which is enabled by default. So no, ImageIntegration does not ignore PEDESTAL keywords in current versions.
sounds like I need to do an update - I first saw this over the winter when doing some reprocessing. What I'm seeing currently is that the master bias still contains the pedestal in the data, but not the PEDESTAL keyword, so when I calibrate the flat frames (and lights), the pedestal ends up getting subtracted twice. The problem goes away if I disable pedestal subtraction at calibration, but I'll look for the pedestal subtraction in the image integration once I've updated.

And of course I keep all my raw files :). I have everything carefully archived because I frequently go back and reprocess old data.