Author Topic: *** Error: Parsing SITELAT FITS keyword during BPP  (Read 792 times)

Offline paolobar

  • Newcomer
  • Posts: 3
    • View Profile
*** Error: Parsing SITELAT FITS keyword during BPP
« on: 2019 June 06 14:45:43 »
Hi,
I got this error during BPP with new version 1.8.06.1475. The strange things is that I got this error when try to quickly integrate a batch of FITS created from SGP beta 3.1.0.198.
I strongly suspect that the culprit is the new Beta SGP, but to be honest that happen only when PI is trying to parse the .XISF file already calibrated and registeredwithout any problem.
I checked the hex of the two strings, both in the original FIT and in the _c_r.XISF and both looks to me legit, without the spurious characters in the strings that you can see in the log. The same exact error is reportedd also for the keyword SITELONG with the same spurios chars inserted.
Note that, despite being marked as ERROR, the integration process is continuing and producing a reasonable result.
Log and hex dump of FIT and XISF attached. I'm available to investigate more and to make available all files.

Thanks
Paolo

https://www.dropbox.com/s/6z61bcvxkt940vc/Screenshot%202019-06-06%2014.38.44.png?dl=0

Offline pfile

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 4389
    • View Profile

Offline paolobar

  • Newcomer
  • Posts: 3
    • View Profile
Re: *** Error: Parsing SITELAT FITS keyword during BPP
« Reply #2 on: 2019 June 07 11:16:00 »
Sorry, my bad.
I searched, but not enough evidently.
I competely agree with Juan to not to write a single line of code to condone or help any deviation from the standards.

That confirm also my feeling about the real source of the problem, and that gives me also a good base for asking SGP for a modification.
Thanks again and sorry for the waste of time.
Paolo

Offline pfile

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 4389
    • View Profile
Re: *** Error: Parsing SITELAT FITS keyword during BPP
« Reply #3 on: 2019 June 07 11:28:02 »
i don't think its a waste of time, don't worry about it.

while i agree that the problem is with the capture software, with millions of frames out there with bad fits data it's not unreasonable to have to "clean up" after SGP as now the only way to solve the problem is on the processing side...

rob

Offline tchitty

  • Newcomer
  • Posts: 10
  • PixInsight Convert
    • View Profile
    • My Astrobin Page
Re: *** Error: Parsing SITELAT FITS keyword during BPP
« Reply #4 on: 2019 June 12 13:10:58 »
It's not just SGP, but TheSkyX captures have the same issue. I talked with one of the developers of TSX, and since they helped write the standard, feel like this is a PI issue.

I'm not trying to start a flame war, just pointing something out.

TomC

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 6782
    • View Profile
    • http://pixinsight.com/
Re: *** Error: Parsing SITELAT FITS keyword during BPP
« Reply #5 on: 2019 June 12 15:07:46 »
No, this is not a 'PI issue'. We follow strictly the original SBIG document describing the SITELAT and SITELONG extended keywords.

It is important to point out that the SITELONG and SITELAT, as well as LONG-OBS, LAT-OBS and ALT-OBS, which we also support, are all of them nonstandard keywords, which have been invented to overcome important lacks in the FITS format standard. Finally, the FITS standard 4.0 defines the OBSGEO-B, OBSGEO-L and OBSGEO-H reserved keywords to represent spherical geodetic coordinates, as well as OBSGEO-X, OBSGEO-Y and OBSGEO-Z for rectangular geodetic coordinates, as part of the 'WCS convention'. The FITS standard 3.0 defines similar keywords, only for rectangular geodetic coordinates, without any specification of a terrestrial reference system. For the sake of standards compliance, future versions of our tools and scripts will support these FITS v.4 standard keywords.

By the way, see this post, where a principal TSX developer agrees that they follow the original SBIG definitions.

Once again, I have to point out the fact that the actual problem here is FITS: a poorly defined, obsolete, inefficient format, which is a source of endless interoperability problems.
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline AccidentalAstronomer

  • Newcomer
  • Posts: 4
    • View Profile
    • The Evening Show
Re: *** Error: Parsing SITELAT FITS keyword during BPP
« Reply #6 on: 2019 June 13 09:47:21 »
FITS... it gives us all fits... ;-)

I've always said I wish most scientist would not write code... FITS is exhibit A. I doubt Juan would disagree.

(Not to derail this, but I also think .xisf might be a better solution for video than .SER files. Anyone else think so?)

The link Juan provided is to a summary of the original document, with at least one error in it. The page has a link to the original document, but the link is broken, so in my earlier message I attached a copy of the original document for reference.

Customers have us between a rock and a hard place. We follow the spec. Pi follows the spec. They still get warnings.

Is there anything I can do in TheSkyX to make the warning go away? We've followed the same spec for 16 years. I can provide sample .fits files if needed as well.

When I can catch my breath, I'm going to start adding support for .xisf directly to TheSkyX. Make no mistake though, FITS is not going away, there's too many in the scientific community and our institutional customers that rely on it.

Richard
richard@bisque.com
Richard S. Wright Jr.
Sr. Software Engineer, Software Bisque
PixInsight fan & Astrophotographer at Large

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 6782
    • View Profile
    • http://pixinsight.com/
Re: *** Error: Parsing SITELAT FITS keyword during BPP
« Reply #7 on: 2019 June 14 03:27:25 »
Hi Richard,

Quote
I've always said I wish most scientist would not write code... FITS is exhibit A. I doubt Juan would disagree.

Wholeheartedly agreeing! :)

Quote
Is there anything I can do in TheSkyX to make the warning go away? We've followed the same spec for 16 years. I can provide sample .fits files if needed as well.

If you are writing these SITELONG and SITELAT keywords following the format described by SBIG in their FITS extension document, then there will be no problems.

To be 100% precise, what we expect is a sequence of one, two or three floating point numbers separated by single spaces or colon (':') characters. These numbers represent degrees, minutes and seconds of arc, with minutes and seconds in the [0,60) range. Longitude should be either in the [-180,+180] range or in the [0,360) range, latitude in [-90,+90]. Longitude is positive east of the reference meridian, latitude is positive north of the equator. So for example, the following keyword values will be correctly parsed without errors:

'120 34 75.493'
'2.123456'
'-23 45.67'
'+44:25:12'

Anyway, we prefer the LONG-OBS, LAT-OBS and ALT-OBS nonstandard keywords for geodetic longitude, latitude and height in meters, mainly because they are 'coherent' with the standard DATE-OBS keyword name. The expected formats are the same.

Finally, should we use the OBSGEO-B, OBSGEO-L and OBSGEO-H keywords defined in the FITS standard 4.0 document instead (or, better, along with) these ones?

Quote
(Not to derail this, but I also think .xisf might be a better solution for video than .SER files. Anyone else think so?)

Very interesting idea. An extension to the XISF format for serialization of video sequences should be relatively easy to formalize. By compressing the entire video sequence using byte shuffling and a fast nondestructive codec, such as LZ4HC (which is as fast as pure LZ4 for decompression and almost as efficient as zlib for compression), we could achieve high compression ratios and real-time decompression with zero degradation. This is because the difference between two consecutive video frames is typically very small, so there should a lot of redundancy in the data stream. We have all the required resources to do this already defined and implemented in XISF.

Quote
When I can catch my breath, I'm going to start adding support for .xisf directly to TheSkyX. Make no mistake though, FITS is not going away, there's too many in the scientific community and our institutional customers that rely on it.

That would be very nice of you. I say this not because you are supporting a format that we have created, but because that would be a strong step forward to achieve the levels of modernization, standardization, efficiency and interoperability that we urgently need in the astronomy community. I started the XISF project exactly with this purpose, but unfortunately—although not surprisingly—it has been received more as a threat than as an attempt to improve the way we work with our data. One would expect more capacity of innovation and mental plasticity, and less resistance to change, from people doing science.
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline AccidentalAstronomer

  • Newcomer
  • Posts: 4
    • View Profile
    • The Evening Show
Re: *** Error: Parsing SITELAT FITS keyword during BPP
« Reply #8 on: 2019 July 05 11:31:52 »
>Finally, should we use the OBSGEO-B, OBSGEO-L and OBSGEO-H keywords defined in the FITS standard 4.0 document instead (or, better, along with) these ones?

I would like to do this. I've already added this (internally) and tested it. If these keywords are present along side the legacy SBIG extensions, will you defer to these first?

Richard
Richard S. Wright Jr.
Sr. Software Engineer, Software Bisque
PixInsight fan & Astrophotographer at Large