Batch Preprocessing script : autodetect image type ?

[xfred53]

Hi,

I'm a new user of PixInsight.

A tutorial
http://harrysastroshed.com/pixinsight/pixinsight%20video%20html/pixinsighthomenewbie.html
showed me that it is possible to have the Batch Preprocessing script automatically detects the image type.

I'm using Prism to get my images and have them saved in fits format.

But when using these fits images with "Add Files", the script is unable to determine the frame type.

Any idea why ?

Thanks in advance and best regards
Francois

[RickS]

I'm using Prism to get my images and have them saved in fits format.

But when using these fits images with "Add Files", the script is unable to determine the frame type.

Any idea why ?

Hi Francois,

BPP looks at the FITS headers first to try to determine the frame type, e.g. if the IMAGETYP header matches "bias frame", "bias" or "master bias" then it will assume it is a bias frame.  If inspecting IMAGETYP fails it will look for hints in the file name, e.g. whether the file name contains "bias", "dark", etc.

I'd guess that Prism is using the FITS headers in some new and unexpected way.  I'd start by taking a dig around in a few example FITS files to see what the headers look like...

Cheers,
Rick.

[rottielover]

Very sorry for digging up such an old thread but I was hoping that someone could help me (and other users) out with this.

I've been hamstrung by this myself and I've finally hit a point of saying to myself "this is stupid, and needs to get fixed."

Here's what I think the root of the issue is.  Prism is not writing out the "IMAGETYP" FITS keyword into the header.  Yet Prism is still able to determine a LIGHT from a DARK etc when I load image sets into it.

If I use the FITSHeader Process and an image container to add the IMAGETYP header of Light into my lights, the BBP script will load the files as Light's, but it still won't correctly determine the filters, even though the FITS headers do in fact contain the Keyword  "FILTERS" and in my case contain either 'LUM '  'B  '  R G  etc.

Here's another kicker for this one,  I tried out the trial of AstroPixelProcessor, and it was able to correctly determine the file type (light, dark, flat) and filter used without me modifying any FITS headers / keywords.

What is it I need to do in order to fix this?


EDIT More data  -   Further if you use the new Weighted BPP script, and you click the "add files" button and feed it frames, then it will correctly determine the Darks from the Lights and Flats, but will not determine the filters on the flats or lights....   

So that's even more confusion at the end user level here, because I'm not modifying any FITS headers between these two scripts.

Further edit for sample file names:

Lights names follow:   M45_R_[date]_#      Target _ Filter Name _ Date/time _ Number of the frame in the sequence.

Darks are named 300s-#     Duration in Seconds - number in sequence

Flats are named :  FLAT_R_[date]_#   
 

[rottielover]

I found a script posted here in the forums that let me export the FITS Headers,  Here is one from one of my FLAT frames

Write Header from FITS File G:/M45/FLAT_SEQ_1__LUM-8(3).fits

Eigenschaften:
   Verzeichnis    /M45
   Laufwerk    G:
   Erstelldatum   Sat Nov 02 2019 15:16:45 GMT-0500 (Central Daylight Time)
   Schreibdatum   Sat Nov 02 2019 07:08:44 GMT-0500 (Central Daylight Time)
   Länge   65563200 Bytes

SIMPLE  =                    T /FITS header                                     
BITPIX  =                  -32 /No.Bits per pixel                               
NAXIS   =                    2 /No.dimensions                                   
NAXIS1  =                 4656 /Length X axis                                   
NAXIS2  =                 3520 /Length Y axis                                   
DATE    = '2019-11-02T12:08:43.978400' /Date of FITS file creation             
SWCREATE= 'PRISM,  Version  10.3.63.500,  10/20/2019' /Software Name and version
ORIGIN  = 'Rottielover's House' /Origin place of FITS image                     
OBSERVER= 'Scott XXXX        ' /Observer name                                   
SITELAT = '+38:XX:XX:00      ' /Latitude observatory                           
SITELONG= '-90:XX:XX:00      ' /Longitude observatory                           
DATE-OBS= '2019-11-02T12:08:42.088800' /UT date of Observation                 
UT      = '12:08:42 PM.08    ' /UT begining of observation                     
EXPOSURE=              0.49300 /Exposure time seconds                           
XPIXELSZ=                 3.80 /X pixel size microns                           
CDELTM1 = 3.79999999999999999E-003 /Size of a pixel (x) in mm                   
XPIXSZ  =                  3.8 /X Binned pixel size as microns                 
YPIXELSZ=                 3.80 /Y pixel size microns                           
CDELTM2 = 3.79999999999999999E-003 /Size of a pixel (y) in mm                   
YPIXSZ  =                  3.8 /X Binned pixel size as microns                 
INSTRUME= 'ZWO ASI1600MM-Cool' /Camera which created data                       
FILTERS = 'Lum               ' /Filters                                         
CVF     =                5.000 /Conversion factor (e-/adu)                     
READOUTT=                 0.01 /Image readout time (sec)                       
X1      =                    1 /X1 image windowing                             
Y1      =                    1 /Y1 image windowing                             
X2      =                 4656 /X2 image windowing                             
Y2      =                 3520 /Y2 image windowing                             
COMPRESS=                    0 /Number of compression                           
TELESCOP= 'ES127             ' /Telescope                                       
FOCAL   =                659.3 /Focal length in mm                             
DIAMETER=                127.0 /Diameter in mm                                 
OBJCTRA = '05 23 41.359      ' /Object Right Ascension (J2000)                 
OBJCTDEC= '+26 14 54.04      ' /Object Declinaison (J2000)                     
RA      = 8.09223304405206250E+001 /Telescope RA                               
DEC     = 2.62483457484666012E+001 /Telescope DEC                               
POSTN-RA= 8.09223304405206250E+001 /Telescope RA                               
POSTN-DE= 2.62483457484666012E+001 /Telescope DEC                               
CRVAL1  = 8.09223304405206250E+001 /approx coord. in RA                         
CRVAL2  = 2.62483457484666012E+001 /approx coord. in DEC                       
CDELT1  = 3.30235040417566929E-004 /ScaleX in deg/pix                           
CDELT2  = 3.30235040417566929E-004 /ScaleY in deg/pix                           
DATAMAX =              3067.00 /Maximum data value                             
DATAMIN =              2685.00 /Minimum data value                             
PIERSIDE= 'EAST              ' /Side of the tube wrt to Pier                   
CCD-TEMP=                -19.8 /CCD temperature as Celsius degrees             
FOCUSPOS=               13.256 /Focuser position as mm                         
FOCUSTMP=                  4.5 /Focuser temperature as Celsius degrees         
BITCAMPX=                   12 /Native camera dynamic in bits per pixel         
STACKNB =                    1 /Amount of stacked images                       
STARCNT =                    0 /Amount of extracted stars                       
GAIN_ELE=                  139 /Camera electronic gain (arbitrary units)       
OFFSET_E=                   40 /Camera electronic offset (arbitrary units)     
TEMPEXT =                  0.0 /External temperature, unit is degrees C         
PRESSURE=                  0.0 /Atm. Pressure, unit is Hpa                     
DEWPOINT=                  0.0 /Dewpoint temperature, unit is degrees C         
HUMIDITY=                  0.0 /Relative humidity, unit is percent             
WINSPEED=                  0.0 /Wind speed, unit is km/h                       
WINDIR  =                  0.0 /Wind direction, North 0deg, East 90d           
HISTORY none                                                                   
END                                                                             
Copyright-Hartmut V. Bornemann


I have munged some personal data with XXXX

[bulrichl]

Strange complaints.

The official FITS keyword for characterizing the used filter is 'FILTER', and not 'FILTERS'. So the right measure would be to change in Prism to the correct FITS keyword. Also the usage of the FITS keyword 'IMAGETYP' in Prism would be appropriate.

It would make much more sense if you address yourself to the programmers of Prism. PixInsight uses the official FITS keywords.

Bernd

[Juan Conejero]

HEASARC - Dictionary of Commonly Used FITS Keywords:
https://heasarc.gsfc.nasa.gov/docs/fcg/common_dict.html

SBFITSEXT Version 1.0:
http://diffractionlimited.com/wp-content/uploads/2016/11/sbfitsext_1r0.pdf

The problem here is, once more, FITS. Both FILTER and IMAGETYP are nonstandard keywords, which were invented by people writing image acquisition applications many years ago, and have been widely accepted as de facto standards by most applications.

If tomorrow a famous application decides to start writing different keywords to identify filters and image types, such as "FOOBARFL" or "FOOBARTP", nothing in the FITS standard would prevent they to do so. Should we add "FOOBARFL" and "FOOBARTP" to the list of recognized keywords in our preprocessing tools and scripts?

Endless interoperability problems caused by an obsolete, poorly designed and loosely defined file format. We created the XISF initiative, among other important reasons, as an attempt to solve these problems once and for all.

[rottielover]

How is that "strange"?    I have two software packages able to read and determine the file types and filters and one that does not, naturally I came to the one that doesn't appear to be working to find out, find the old thread and ask.

I'm more than happy to reach out to the Prism folks now that I know that Pix doesn't like the keyword.

XKCD said it best about standards: https://xkcd.com/927/


So we've determined that Pix doesn't like the keywords, that's fine, it was stated that Pix should then look for hints in the name of the file itself.  So please point me to the section of the PixInsight documentation that explains how I should setup my file naming convention in my image capture software.  Meanwhile I'll certainly reach out to the Prism folks as well and inform them of the "FILTERS" vs "FILTER" thing.

[dave_galera]

XKCD said it best about standards: https://xkcd.com/927/

LOL........

[rottielover]

Funny enough if you search for keywords related to file naming convention for pixinsight batch preprocessing the #1 result is this thread.

I was able to track down some more information in the documentation for the new WBPP script here:  https://www.tommasorubechi.it/2019/11/15/the-new-weighted-batchpreprocessing/

It's roughly 1/2 way down the page under the heading "Frames Grouping and Smart Naming".   According to this document you can name the folder for the subs or the subs themselves.  I'm going to try this next and see if it works.

format appears to follow something like    [Type]-[Filter_<>]-[Binning]-[Exptime_<>]         WHERE

Type:  bias, dark, flat, light, masterDark, masterFlat, masterBias, masterLight
Filter:   FILTER_Lum, FILTER_Ha, FILTER_R, FILTER_Red   etc
Binning: BINNING_1, BINNING_2, etc.
Exptime:  EXPTIME_300, EXPTIME_60, etc.   

So an example file or folder name might be:    light-FILTER_Red-BINNING_1-EXPTIME_300    (Place all your 5min Red light subs in this folder)

I'll edit this and report back if it's working




EDIT:   Not working quite the way it was documented (you might still have to choose the button for Type instead of the "files" button, explained below)

I copied a sample set of data to use the new names.

with a file named "masterFlat-FILTER_Blue.xsif" for example (and using the "Add Files" button:   BPP was able to determine it to be a flat file and even listed them all under Binning 1, but the newer WBPP script errors "Unable to determine frame type".  Note these master flats were created in Pi and are XISF files.  I then tried using the Add Flats button and these were then put into the correct Filter groups and I was able to use the "Use master flat" checkbox to have it recognize each as a master for that filter.

WBPP recognized the files if I added them one group at a time under each folder, and picked up the info from the folder names.  It even was able to read the EXPTIME from the FITS header (I assume) because I named the folder with "EXPTIME_300" but the lights (correctly) showed up as 60s .   

For all you who find this thread with the same problem I guess the way to handle this for the moment is to set your image capture software file naming convention to use the convention I've outlined above (WATCH OUT for the Hyphen (-) vs Underscore (_) in the naming! )

This way you should at least be able to load your frames into the WBPP script slightly easier and with less mistakes.  You may still need to use the individual buttons for "Load Flats" or "Load Lights" but at least you won't have to spend time in the custom dialog window re-specifying the filter name, exposure time etc, which was a time consuming process for data that already existed.

Best of luck out there and clear skies!!