calibration of lightbuckets.com images?

[Juan Conejero]

What is really becoming evident here, in my opinion, is that we are suffering the consequences of important weaknesses in the FITS format.

In my opinion the FITS format, despite its name, is neither an image transport system —I think it is an arbitrary data transport system— nor a flexible system —I think it is an undefined system—.

FITS was defined in the 70's. It pertains to the magnetic tape era. We are now exiting the blue ray era. We need something more contemporary. The main flaws that require urgent addressing are:

- No way to define the actual numerical range to which the data are referred to. In the case of images, this means that we have no standard way to know which values correspond to the "no signal" state (usually black) and "full signal" state (usually white). For unsigned integer images, we assume that 0=black and 2^nbits-1=white. For floating point data, there is no way to know anything and each software package does what their authors think is more appropriate. This leads to serious incompatibility problems.

- No native support for unsigned integer data. The BZERO and BSCALE (or TZERO and TSCALE) tricks are picturesque but obviously should not be necessary.

- Severe limitations to metadata representation and storage. Header keywords must be 80 bytes wide (which was the size of a magnetic tape record when FITS was conceived) and can only contain 7-bit ASCII characters. Keyword names can only have 8 characters. In the 21th century one would expect XML metadata and full Unicode support as UTF8 or UTF16. This is an absolute necessity.

- No standardized way to store key items such as ICC profiles, resolution metadata, working color spaces, etc. Obviously these elements go beyond astronomical image processing but are absolutely necessary to integrate an image format in the normal workflow of any serious imaging platform. PixInsight is, as far as I know, the only application able to embed an ICC profile in a FITS file (as an extension HDU), but it uses a private convention because there is no standard.

- No standardized, efficient lossless image compression system.

Just my opinions. Actually, we have been thinking on the draft of an XFITS format since some time back.

[Niall Saunders]

From bitli's last post:

The master images cause the problem. They have been processed by MAximDL (that's why they are master images), and it makes perfect sense to save them in 32 bits floating point

Not to me it doesn't. I do not believe that there is ANYTHING to be gained by having ANY image data in a format offering more than 16-bit resolution until AFTER all image calibration stages have been completed. It is almost justifiable to claim that any results of the 'division by the MasterFlat' could benefit by being held in 32-bit float format. However that immediately implies some means of 'standardizing' the to range of such images. And that does not (yet) exist (as far as I am aware).

but as there is no way to read a (32-bit Float) image without external knoweldege, there is indeed a problem

Yep, the nail has been hit squarely on it's head !! PixInsight sets it's 'own' standard, by keeping the 32-bit Float data in the [0.0, 1.0] range - which is actually perfectly valid, for any other application capable of reading 32-bit Float data correctly. AIP4WIN, for example, at one time did not 'understand' the concept of all data being in the [0.0, 1.0] range, and required you to "multiply the image by 65535" to bring the data back into an AIP4WIN-usable format. That seems to have been rectified in later releases.

Anyhow, it is unlikely that maximdl modifies their product so that we can easily use PI instead of their product...

They don't have to - the conversion process is easily accomplished in PI. But what IS needed is a clear definition of WHAT the 'strategy' actually is so that a user can deal with the data accordingly. But, that needs to be clearly explained - in exactly the way that it WASN'T in the Meade Envisage software!!




From pfile's last post:

when it comes to maximDL they seem to draw a line in the sand

That is all very well assuming that MaximDL is generating files in a UNIVERSAL STANDARD FORMAT. Otherwise, presumably, you (as their 'customer') are going to have to invest in some appropriate software that can read these non-standard files correctly.

they went back and calibrated the files for me by hand with maximDL

At least they do provide good customer service - and that has to be commended

but the files were saved as compressed fits

But, they are still expecting you to deal with a problem that was of THEIR making, because (as far as I am aware, but would be happy to be corrected) 'compressed fits' is another 'non-standard' image format

"these open okay in maximDL, so, uh...?"

Sorry, not good enough - unless somewhere in your contractual obligation with Lightbuckets it states that the customer will HAVE to own a copy of MaximDL. You are purchasing a commodity, and it should meet certain well-defined standards (even if THEY are going to define those standards - which is actually quite acceptable, but they HAVE to make you aware of that). And, if the 'standard' is to be 'FITS' then the data should pass a FITSverify inspection and should be viewable in FITSViewer - in my opinion anyway.

i'm really just guessing here about what maxim is doing with CBLACK and CWHITE

That's part of the problem - so is everyone else !!


And, from Juan's last post :

we have been thinking on the draft of an XFITS format since some time back

Fine Juan, but who is 'behind' the XFITS format? Is this already in some embrionic stage? Or are these opening thoughts, suggestions and proposals purely coming out of the PixInsight stable?

My concern there is that the "always has been" community (that forms the backbone of the astronomy world) may find it difficult to accept proposals from that 'upstart PixInsight community' - after all, "they already want us to give up our airbrush tools, so they obviously CANNOT be trusted"

There certainly is NOT a 'right' or 'wrong' way to do things here - but at least sticking with such FITS guidelines as already exist must at least help. And perhaps that means that the 32-bit Float format needs to be seriously re-evaluated, so that the notion of a [black, white] range IS implicit. Either that, or the use of the 32-bit Float format is avoided until some stage in processing where the lack of a universally accepted range no longer matters.

we are suffering the consequences

That is for sure

Cheers,

[bitli]

Back to trying to calibrate the images (they may not be the right format, but it is what I have):

Looking at the FITS, the calibrated images are on 15 bits while the lights are on 16 bits. After rescaling them with the proper value I get stuff that look useable, the bias is high. The dark is higher. The flat is right around 0.5.  I saved all the rescaled files in 32 bits PI from 0.0 to 1.0. The dark is for a much longer exposure, but my understanding is that PI does not care.

If I try to do the calibration with PI, I get the message "Zero or insignificant scaling factor for flat frame channel# 0. (empty master flat frame image?)". And no, my master flat does not look empty. In fact it is probably the one which is the most accurate +/- the bias.

Any idea of what may cause this error message?

-- bitli

[pfile]

do you get a dark scaling factor of 0.000 when it tries to subtract the dark from the light frame?

this might be specific to the recent driver updates on LB-0003, but the darks i currently have are shifted up by 10 pixels with respect to the lights. this causes pixinsight to fail to find a correlation between the light and the dark, and so it can not scale the dark.

i do have some older data from LB-0002, but i never tried to calibrate it with PI. i'll have to give that a try and see what i get.

i am still having problems with pixinsight's handling of the flat frame. can i upload one light and the master flat somewhere and maybe juan or vincent can look at it? (did vincent write the ImageCalibration tool?)

[bitli]

1)
It was with LB 002 data, so not related to the issue of flat on LB003.
Anyhow that problem disappeared after a restart of the computer...
It was able to do the calibration, the result looks like an image, but was much worse than the auto calibrated one,
The log is
Calibrating master flat frame ...
Optimizing master dark frame:
Bracketing...
Optimizing...
k0 = 0.013
Computing master flat scaling factors ...
s0 = 0.450142
...
Loading target frame:
..Reading
FITS: 16-bit integers, 1 channel(s), 3326x2504 pixels: 100%
Writing output file: ..
Dark scaling factors:
k0 = 0.073

I need to dig in the videos and forums to find out if the number helps figuring out the problem.


2)
I tried another route.  I got my files loaded and MAximDL and saved as FITS Integer 16 without auto rescaling.  I though this should solve my problems.

False hopes. The result is:

PCL FITS Format Support: Unable to open FITS file:
.../calibrationtest/Flat_FilterLuminance_Bin1x1_PosAng95.2_1_I16.fit
CFITSIO error message stack:
01 : Extension doesn't start with SIMPLE or XTENSION keyword. (ffrhdu)
02 : SIMPLE?=?T²BITPIX?=?16/8unsignedint,16&32int,-32&-64real?NAXIS?=?2/n
03 : ffopen could not interpret primary array header of file:
04 : ...\calib
05 : rationtest\Flat_FilterLuminance_Bin1x1_PosAng95.2_1_I16.fit
06 : This does not look like a FITS file.

Your desesperate...
... bitli

[pfile]

on #2 that looks like what happened to me when the file was saved with compression in maximDL.

[Niall Saunders]

For those who may be interested, there is an 'official' treatise on compressed FITS files here : http://heasarc.nasa.gov/docs/software/fitsio/compression.html

I have no idea whether this is how other software tackles the problem, but I did wonder, Juan, whether PixInsight would consider at least conforming to this 'standard' as well?

Cheers,

[Juan Conejero]

Hi Niall,

we have been thinking on the draft of an XFITS format since some time back

Fine Juan, but who is 'behind' the XFITS format? Is this already in some embrionic stage? Or are these opening thoughts, suggestions and proposals purely coming out of the PixInsight stable?

Actually this is something that I've been thinking about for years ... shortly after reading the FITS standard for the first time Besides me, Vicent and some people at the OAUV nobody else is behind this idea AFAIK. Indeed, it is embryonic.

My concern there is that the "always has been" community (that forms the backbone of the astronomy world) may find it difficult to accept proposals from that 'upstart PixInsight community' - after all, "they already want us to give up our airbrush tools, so they obviously CANNOT be trusted"

Yeah, that's true. But if I had felt conditioned by that now PI would be just a forgotten little experiment. Nothing in the "astronomy world" has helped us until very recently.

More seriously now, FITS is an anachronistic format that must evolve as soon as possible. Somebody has to push in this direction and as you know we are always ready to push things

XFITS (or whatever it would be called) is not a priority for us right now but we know it's something in our way, unless somebody else starts a similar project —and starts it well—, which is unlikely to happen, IMO.

There certainly is NOT a 'right' or 'wrong' way to do things here - but at least sticking with such FITS guidelines as already exist must at least help. And perhaps that means that the 32-bit Float format needs to be seriously re-evaluated, so that the notion of a [black, white] range IS implicit. Either that, or the use of the 32-bit Float format is avoided until some stage in processing where the lack of a universally accepted range no longer matters.

That would be nice, but I wouldn't have too much hope that it will ever happen. Everybody likes the way their applications do the trick. For example, I think the [0,1] range is the best way to store pixel data as real numbers, but if you ask the developers of other applications, they will tell you that I am wrong. The same if you ask me about how they do it. I won't change and they won't change. Great.

This is the problem when something important is not standardized. The FITS format is not being flexible here; it is just undefined. And an undefined format is useless by definition.

[Juan Conejero]

Hi again Niall,

I have no idea whether this is how other software tackles the problem, but I did wonder, Juan, whether PixInsight would consider at least conforming to this 'standard' as well?

No, thanks. Excerpting directly from the page you have linked above:

The 3 supported image compression algorithms are all 'loss-less' when applied to integer FITS images; the pixel values are preserved exactly with no loss of information during the compression and uncompression process. Floating point FITS images (which have BITPIX = -32 or -64) are first quantized into scaled integer pixel values before being compressed.

Besides this "limitation" (being purposely euphemistic here), there are more problems. FITSIO's implementation only works on a tiled image compression scheme. Although FITSIO does all the tricks behind the scenes, this is inefficient because it essentially prevents incremental reading/writing of FITS images, which is one of its best features. For example, a tiled scheme wouldn't work with our ImageIntegration tool. There should be a row-based compression scheme such as the one that exists in the TIFF format for example.

On the other hand, this is no standard at all. I would not recommend anybody to store his/her data compressed in this way. A FITS file compressed using FITSIO's method is anything but a FITS image. The compressed data are stored using special extension HDUs and exotic header keywords in the main HDU, since the FITS standard does not provide any support to implement compression.

Finally, the "External File Compression" method is already supported by PixInsight (for reading only). However, unfortunately CFITSIO does not support GZIP compression on Windows, because it refuses to compile with GNU code under Windows. On Linux and Mac OS X, you can open an image such as "test.gz.fit" without problems; it is decompressed on the fly. This is also supported by PJSR; for example you can enter this command from the console:

j ImageWindow.open( "test.gz.fit" )[0].show();

and it will open and show a "test.gz.fit" image. Unfortunately this is not possible on Windows, due to CFITSIO's licensing policy.

[Niall Saunders]

Hi Juan,

Thanks again for taking the time to explain the 'background' (or 'reality') behind some of the concepts that you have to deal with whilst building PI. The more I learn, the more I realise how big some of the obstacles put in your way have actually been!!

So, what had 'appeared' to be a 'FITS standard' actually isn't, and even where implemented it cannot be used on all OS platforms. Well, that certainly would make me consider it 'unsuitable' for inclusion with PI, although I admire the fact that you allow PI to use the facilities provided within CFITSIO, even though they are not fully cross-platform compatible

Never mind, perhaps - eventually - common sense will prevail

Cheers,

[Niall Saunders]

Never mind, perhaps - eventually - common sense will prevail

In fact, on the way home tonight, I realised that the following quote by astronomer J. Donald Fernie is actually quite appropriate for a lot of the ground-breaking stuff promoted by PixInsight some 50 years later

The definitive study of the herd instincts of astronomers has yet to be wriitten, but there are times when we resemble nothing so much as a herd of antelope, heads down in tight formation, thundering with firm determination in a particular direction across the plain. At a given signal from the leader we whirl about, and, with equally firm determination, thunder off in a quite different direction, still in tight parallel formation.

^{Publication of the Astronomical Society of the Pacific 81 (1969):707}

Cheers,

[pfile]

can someone advise me regarding this?



the top row of images are : the raw science frame, the master flat frame and the resultant calibrated frame. the statistics window is set to the master flat frame.
the bottom row of images are: the flat frame, scaled with the histogram tool as shown in the histogram settings. the calibrated frame is on the right. it looks pretty reasonable.

is it to be expected that i'd have to manipulate the flat this way in order to get correct division in ImageCalibration? clearly the topmost calibrated frame is completely screwed up. my problem with this is that what i've done to the flat to make the calibration right is completely ad-hoc.

[Carlos Milovic]

No, this is not "normal" behavior. After bias substracting, flats should be ready to be applied (of course, they needs to be normalized, but this is done in the division operation, multiplying by a constant number, usually the flat's median).

[pfile]

when i get home tonight i can post the fits files somewhere. maybe someone can take a look at this, perhaps it is a bug?

of course in this example i have not calibrated the master flat frame (nor have i set bias or darks when calibrating the science frame) but i don't think it should matter - i get the same result if i put the master bias and master dark into ImageCalibration as well.

[pfile]

http://pixinsight.com/forum/index.php?topic=1951.0