Batch Preprocessing Script failing to Calibrate

[savery]

I hesitate to report this as a bug, but I am having trouble getting the Batch Preprocessing Script to calibrate my lights correctly.
The symptom is that the allegedly calibrated output files still display dust motes.  At first, I blamed my flats and tried to make better ones. When this did not help, I tried to calibrate four lights "manually" using the ImageCalibration module.  In this case, using the same master bias, master dark and master flat, the calibration was perfect.  So something seems to be going wrong with the Preprocessing Script.

Has anyone else experienced this problem?  Any ideas what I might be doing wrong?  Is it possible that the Script assumes that the Flat needs to be calibrated (i.e. bias-subtracted)?  My master flat is already calibrated, which I think is the preferred method.

Stephen

[pfile]

the script does assume that master flats are calibrated. so it would not double-calibrate a flat.

is it possible that BPP did not detect that the master flat you loaded belonged to the filter for the lights?

rob

[Juan Conejero]

This cannot be reproduced under normal working conditions. As Rob points out, this is most likely a frame identification issue. The BPP script relies on image metadata, mainly header keywords, to identify frame types, filters, exposure times, etc automatically.

If the required metadata is not available, you have to select files manually by clicking the 'Add Custom' button, or alternatively, add the missing keywords to your master frame (for example, the appropriate FILTER keyword).

[savery]

Thank you for your replies.
I have made further tests.  The FITS header of my flat had the designation "Light Frame" under "ImageTyp".  I therefore edited this to read "FLAT".  However, it does not make any difference!  The Script still fails to calibrate the lights.  What other keywords does the script require in the header?  The filter type is correctly given as "Lum".

Stephen

[savery]

Just to let you know that I have now solved the problem .

It turns out that I need to select "Up-bottom FITS" in order for my lights to be calibrated correctly.  Otherwise, it seems that the flats (and presumably the other calibration frames) are processed upside down!

In this connection, it seems to make no difference whether the metadata identifies the flats as "FLAT" or as "Light Frame" in the "ImageTyp" part of the FITS header.

Strangely, the Script will run apparently without any problem whether the "Up-bottom FITS" box is checked or not.  The diagnosis function does not identify the issue.  Therefore, the correct diagnosis of the problem was not as obvious as might be supposed. (At least, that's my story and I'm sticking to it....)

Stephen

[pfile]

hmm... the thing is as long as all the types of frames are processed with the same fits reader direction, things should work out right calibration-wise (but as you say the images will be upside down if the wrong reader direction is specified)

this is pretty strange.

rob

[savery]

I can only assume that the script takes the fits reader direction for the calibration frames from the "Up-bottom FITS" check box, but not the reader direction for the lights.  I agree it is strange because the origin of coordinates is identified in the FITS header for all the frames.
What other explanation can there be?

Stephen

[pfile]

what keyword is used to indicate the origin? i don't think PI considers it, whatever it is. most FITS keywords are non-standard...

rob

[savery]

In both my flats and my lights the FITS header contains the following keywords:

XORGSUBF = 0
YORGSUBF = 0

Stephen

[pfile]

ok - yeah, those are SBIG's proposed extensions to the FITS header keywords. i don't think PI considers those keywords - no mention of them from grepping thru the strings in the FITS handler module, and no mention in the BatchPreProcessing source. further, it seems that all files in BPP are opened with the same format hints, which contain either up-bottom or bottom-up depending on the switch in the GUI. but that's just from casual inspection of the code. i didn't try to add debug printfs or anything to really check that for certain.

rob

[Juan Conejero]

AFAIK, the XORGSUBF/YORGSUBF pair of (nonstandard) keywords don't define the orientation of the image with respect to the way pixel data are stored in the FITS file. Nothing in these keywords defines how coordinates grow on each axis, e.g. if the vertical coordinate grows from top to bottom (the 'amateur' FITS convention) or from bottom to top (the 'professional' FITS convention).

From the information I have, these keywords are intended to define the location of a subframe within a larger image acquired with a multi-sensor camera, which generates a mosaic of frames.

In the BPP script, the orientation of FITS frames is defined exclusively by the 'Up-bottom FITS' script parameter. There is no standard way to know the coordinate convention used by a particular FITS image. Welcome to the endless story of FITS interoperability issues.