Incompatible image geometry - FITS cropped?

[RadekK]

I have just tried to calibrate my frames collected recently and... I cannot calibrate because PI reports "Incompatible image geometry" even though the files are just fine.
FITS files are 2750 x 2200 px, and FITS header confirms this size. However when I try to calibrate the frames PI reads them as 2749 x 2199 px. 1 pixel missing in both axis!
I inspected the frames and they ARE correct size and this is PixInsight that is wrong. I'm using the latest PixInsight 01.08.06.1457 Ripley (x64) running on Ubuntu 18.04 x64.
See the attached screenshot.

[pfile]

well can you try opening the fits files with fv or fitsliberator or something else to see what happens?

rob

[RadekK]

The files are perfectly fine. I can calibrate them in any other software e.g. siril. When viewed in other software they show proper size and geometry.
Moreover they show proper size and geometry when opened in PixInsight. It seems something wrong is happening in Image Calibration module only (also in BatchPreProcessing).
BTW. I've been using PixInsight for a few years and the issue seems related to the latest version only. I have not observed it before.

[pfile]

well in the end juan wont be able to debug this unless you upload one of these fits files and post a link...

rob

[RadekK]

Here you are https://drive.google.com/drive/folders/1OrEogyA0o5VJh9hAAOxJ0GQz3YBQqGDQ?usp=sharing

[Juan Conejero]

I can use these files with ImageCalibration without any problems. Both images have the same geometry and calibration works as expected. I have just checked this on macOS 10.14 and Linux Mint 19.1.

I cannot find any problems here, neither with the FITS support module nor with the ImageCalibration process.

On your screenshot, the console shows "NGC 2246..." files while the FITSHeader tool has an "NGC2237..." image selected. Are you sure you are trying to calibrate a coherent data set?

[RadekK]

Right, I uploaded wrong light frame. Please redownload the files that allow for reproducing the issue.
Also, attached the screenshot with the exact files uploaded.

BTW. Note the image size 2749 x 2199, which is not correct. When you open the image directly in PixInsight it reads 2750 x 2200 px (as per FITS header).

[RadekK]

I have just noted that all the files processed with the latest PI version report geometry missing 1 px in both axis. Whereas files processed with the older version read proper geometry.
Masterdark and superbias have been processed with older PI version. If I use them with recent lights I cannot calibrate BUT if I recompile them with the latest PI version they can be used in calibration process. However, if I do it all the files' geometry is read as missing 1 px in both axis. Even though the physical geometry is fine.
I think that the latest PI somehow loses this 1 px in both axis when reading FITS. Isn't it 2-dimentional array indexing issue? ;-)

[Juan Conejero]

Hi again,

Here is the FITS header of the 'NGC2246_Light_Ha_600_secs_001.fits' file you have uploaded:


NGC2246_Light_Ha_600_secs_001:
SIMPLE   T                         file does conform to FITS standard
BITPIX   16                        number of bits per data pixel
NAXIS    2                         number of data axes
NAXIS1   2749                      length of data axis 1
NAXIS2   2199                      length of data axis 2
EXTEND   T                         FITS dataset may contain extensions
COMMENT                            FITS (Flexible Image Transport System) format is defined in 'Astronomy
COMMENT                            and Astrophysics', volume 376, page 359; bibcode: 2001A&A...376..359H
BZERO    32768                     offset data range to that of unsigned short
BSCALE   1                         default scaling factor
INSTRUME 'Atik 460ex'              CCD Name
TELESCOP 'EQMod Mount'             Telescope name
OBSERVER 'Unknown '                Observer name
OBJECT   'NGC2246 '                Object name
EXPTIME  6.000000E+02              Total Exposure Time (s)
CCD-TEMP -2.00E+01                 CCD Temperature (Celsius)
PIXSIZE1 4.540000E+00              Pixel Size 1 (microns)
PIXSIZE2 4.540000E+00              Pixel Size 2 (microns)
XBINNING 1                         Binning factor in width
YBINNING 1                         Binning factor in height
XPIXSZ   4.540                     Pixel size, X-axis (um)
YPIXSZ   4.540                     Pixel size, Y-axis (um)
FRAME    'Light   '                Frame Type
FOCALLEN 383.934                   Focal length (mm)
SCALE    2.439068E+00              arcsecs per pixel
SITELAT  5.216610E+01              Latitude of the imaging site in degrees
SITELONG 2.107560E+01              Longitude of the imaging site in degrees
AIRMASS  1.560078E+00              Airmass
OBJCTRA  '6 31 52.784'             Right ascension of the center of the image
OBJCTDEC '+4 57 23.36'             Declination of the center of the image
RA       9.797026E+01              Object J2000 RA in Degrees
DEC      4.956843E+00              Object J2000 DEC in Degrees
RADECSYS 'FK5     '                RADECSYS
SECPIX1  2.4390676878E+00          SECPIX1
SECPIX2  2.4390676878E+00          SECPIX2
DATE-OBS '2019-02-08T21:15:53.694' UTC start date of observation
COMMENT                            Generated by INDI
EQUINOX  2000.0                    Coordinates referred to ICRS / J2000.0
CTYPE1   'RA---TAN'                Axis1 projection: Gnomonic
CTYPE2   'DEC--TAN'                Axis2 projection: Gnomonic
CRPIX1   1374.5                    Axis1 reference pixel
CRPIX2   1099.5                    Axis2 reference pixel
CRVAL1   97.970258634              Axis1 reference value
CRVAL2   4.9568430277              Axis2 reference value
PV1_1    0                         Native longitude of the reference point (deg)
PV1_2    0                         Native latitude of the reference point (deg)
CD1_1    -0.0006769025156961843    Scale matrix (1,1)
CD1_2    2.889137503822072e-05     Scale matrix (1,2)
CD2_1    -2.889137503822072e-05    Scale matrix (2,1)
CD2_2    -0.0006769025156961843    Scale matrix (2,2)
CDELT1   -0.0006775188021800002    Axis1 scale
CDELT2   -0.0006775188021800002    Axis2 scale
CROTA1   2.443999999999983         Axis1 rotation angle (deg)
CROTA2   2.443999999999983         Axis2 rotation angle (deg)


You can verify this with the 'lskwd' command from PixInsight's process console, and also with the FITSHeader tool. So this file actually stores an image with 2749x2199 pixels.

If you don't believe PixInsight, you can upload the file to NASA's FITS File Verifier service. Here is the report:


File name:  NGC2246_Light_Ha_600_secs_001.fits
                                           Run Number 25838
 
              fitsverify 4.18 (CFITSIO V3.440)             
              --------------------------------             
 

1 Header-Data Units in this file.
 
=================== HDU 1: Primary Array ===================
 
   1 | SIMPLE  =                    T / file does conform to FITS standard
   2 | BITPIX  =                   16 / number of bits per data pixel
   3 | NAXIS   =                    2 / number of data axes
   4 | NAXIS1  =                 2749 / length of data axis 1
   5 | NAXIS2  =                 2199 / length of data axis 2
   6 | EXTEND  =                    T / FITS dataset may contain extensions
   7 | COMMENT   FITS (Flexible Image Transport System) format is defined in 'Astronomy
   8 | COMMENT   and Astrophysics', volume 376, page 359; bibcode: 2001A&A...376..359H
   9 | BZERO   =                32768 / offset data range to that of unsigned short
  10 | BSCALE  =                    1 / default scaling factor
  11 | INSTRUME= 'Atik 460ex'         / CCD Name
  12 | TELESCOP= 'EQMod Mount'        / Telescope name
  13 | OBSERVER= 'Unknown '           / Observer name
  14 | OBJECT  = 'NGC2246 '           / Object name
  15 | EXPTIME =         6.000000E+02 / Total Exposure Time (s)
  16 | CCD-TEMP=            -2.00E+01 / CCD Temperature (Celsius)
  17 | PIXSIZE1=         4.540000E+00 / Pixel Size 1 (microns)
  18 | PIXSIZE2=         4.540000E+00 / Pixel Size 2 (microns)
  19 | XBINNING=                    1 / Binning factor in width
  20 | YBINNING=                    1 / Binning factor in height
  21 | XPIXSZ  =         4.540000E+00 / X binned pixel size in microns
  22 | YPIXSZ  =         4.540000E+00 / Y binned pixel size in microns
  23 | FRAME   = 'Light   '           / Frame Type
  24 | FOCALLEN=             3.84E+02 / Focal Length (mm)
  25 | SCALE   =         2.439068E+00 / arcsecs per pixel
  26 | SITELAT =         5.216610E+01 / Latitude of the imaging site in degrees
  27 | SITELONG=         2.107560E+01 / Longitude of the imaging site in degrees
  28 | AIRMASS =         1.560078E+00 / Airmass
  29 | OBJCTRA = ' 6 31 52.86'        / Object J2000 RA in Hours
  30 | OBJCTDEC= ' 4 57 24.63'        / Object J2000 DEC in Degrees
  31 | RA      =         9.797026E+01 / Object J2000 RA in Degrees
  32 | DEC     =         4.956843E+00 / Object J2000 DEC in Degrees
  33 | EQUINOX =                 2000 / Equinox
  34 | CRVAL1  =     9.7970258634E+01 / CRVAL1
  35 | CRVAL2  =     4.9568430277E+00 / CRVAL1
  36 | RADECSYS= 'FK5     '           / RADECSYS
  37 | CTYPE1  = 'RA---TAN'           / CTYPE1
  38 | CTYPE2  = 'DEC--TAN'           / CTYPE2
  39 | CRPIX1  =     1.3745000000E+03 / CRPIX1
  40 | CRPIX2  =     1.0995000000E+03 / CRPIX2
  41 | SECPIX1 =     2.4390676878E+00 / SECPIX1
  42 | SECPIX2 =     2.4390676878E+00 / SECPIX2
  43 | CDELT1  =     6.7751880218E-04 / CDELT1
  44 | CDELT2  =     6.7751880218E-04 / CDELT2
  45 | CROTA1  =     1.8244400000E+02 / CROTA1
  46 | CROTA2  =     1.8244400000E+02 / CROTA2
  47 | DATE-OBS= '2019-02-08T21:15:53.694' / UTC start date of observation
  48 | COMMENT Generated by INDI
  49 | END     
 
 49 header keywords
 
 16-bit integer pixels,  2 axes (2749 x 2199),
 
++++++++++++++++++++++ Error Summary  ++++++++++++++++++++++
 
 HDU#  Name (version)       Type             Warnings  Errors
 1                          Primary Array    0         0     
 
**** Verification found 0 warning(s) and 0 error(s). ****

[Juan Conejero]

I think that the latest PI somehow loses this 1 px in both axis when reading FITS. Isn't it 2-dimentional array indexing issue? ;-)

The 'NGC2246_Light_Ha_600_secs_001.fits' file that you have uploaded has not been written by PixInsight. If it was, it would include the following FITS header keywords:

PROGRAM  'PixInsight 01.08.06.1457' Software that created this HDU
COMMENT                             PixInsight Class Library: PCL 02.01.11.0938
COMMENT                             FITS module version 01.01.05.0433

The file you have uploaded has 2749x2199 pixels, and it has been generated outside PixInsight.

So we have no '2-dimentional array indexing issue' here ;-)

[RadekK]

That's right! It looks like the issue is on image acquisition side. I was mislead with several sets of data collected in various dates.
Thanks a lot for directions. Need to discuss this on INDI forum then

[knro]

Juan, thank you for your response, we're checking this issue. There is another issue, it seems PixInsight cannot open bayered images generated by INDI. I tried to debayer a FITS image and it reports "Missing or invalid CFA pattern description property".

However, if you inspect the FITS header, there is already BAYERPAT property. So what's the problem?

Here is the FITS bayered image: https://www.dropbox.com/s/e3xco2rv7z6aaxw/m42_bayered.fits?dl=0

[RadekK]

It looks like the BAYERPAT property is not read from the header. I have just tested it on the M42 file and manually forcing the CFA pattern (according to BAYERPAT property) debayers properly.

[RadekK]

An update on the issue, so anybody looking for the solution finds it also here.

So the problem is exactly opposite to what I have thought! I have been using this Atik 460EX for over 3 years and it always imaged at 2750px x 2200 px. This geometry was always like that in FITS header and actual FITS data. I haven't used any other software than KStars/Ekos & INDI ever since so I never compared it to any other source. Also PixInsight was my choice for post-processing.
Having said that I would bet Atik 460EX chip resolution was 2750px x 2200 px :silly:
Recently the atik driver maintained by Peter Polakovi? (CloudMakers) has been replaced by Jasem Mutlaq (INDI) and the new driver was included in regular updates as a 3rdparty driver. I have updated my systems and noticed that I cannot calibrate new light frames with dark and bias frames library, which I maintain for myself. I noticed that the resolution of the new images changed to 2749px x 2199 px. First I thought t is PixInsight that crops the images, then I took it as a bug in the new driver BUT the actual issue is that it is the old driver that was wrong with the image geometry!
The native resolution of Atik 460EX is 2749px x 2199 px. Period.
This solves the issue!

[Juan Conejero]

There is another issue, it seems PixInsight cannot open bayered images generated by INDI. I tried to debayer a FITS image and it reports "Missing or invalid CFA pattern description property".

However, if you inspect the FITS header, there is already BAYERPAT property. So what's the problem?

Here is the FITS bayered image: https://www.dropbox.com/s/e3xco2rv7z6aaxw/m42_bayered.fits?dl=0

You can debayer any raw CFA frame without problems. Just select the correct Bayer pattern manually on the Debayer tool (or, equivalently, on the BatchPreprocessing script).

BAYERPAT is not a standard FITS header keyword, so there is no need to support it in a standards compliant FITS decoder. Our current Debayer tool does not support a BAYERPAT keyword.

Having said that, we already support a number of nonstandard FITS keywords for convenience. I'll add support for this one and a few more in the next version of the Debayer tool.

The XISF format, by the way, has standard formal resources to serialize CFA patterns, among many other essential image metadata items. Unlike FITS, XISF is a strictly defined and really extensible format where interoperability problems like this one cannot happen.