New tutorial: Master Calibration Frames: Acquisition and Processing

[NKV]

AA4 supports up to 96 bits

RGB 3*32bit=96bit 
So PixInsight support 192bit 

[Juan Conejero]

Hi Tasos,

I must make an important correction here. Your bias frame is correct and it is PixInsight that is not interpreting it correctly.

Your bias image has been stored as a signed 16-bit FITS file. Hence, it stores pixel values in the range [-32768, ..., 0, ..., +32767]. Although this is conceptually incorrect in my opinion, it is indeed correct as per the FITS standard, and indeed no BZERO and BSCALE keywords are necessary in this case (that is, they would be 0 and 1, respectively) because the file stores physical pixel values.

The conceptual error, in my opinion, is in the fact that a CCD does not produce negative values in the range [-32768,0[, so storing physical pixel values as signed integers does not make any sense. However, as I've said the FITS standard allows it, and that's the only fact that counts here.

PixInsight is making the wrong assumption that the file stores valid pixel values in the whole 16-bit signed range, while the file actually stores physical values, which can only be positive, and hence the negative half of the numeric range is unused. So the problem is all ours and the file is definitely valid. This is actually a design error that I'll try to fix as soon as possible.

My sincere apologies to the software and to the persons who have written it. I hope nobody is too offended.

[Niall Saunders]

Hi Juan,

Just trying to 'keep on top' of this wonderful world of 'FITS Standardisation' !!

So, given a 'standard' FITS 16-bit Signed Integer file, how does PI interpret this? Do you currently set the PI 'internal' [0.0, 1.0]_int range to equate to the 16-bit SI range of [-32768, 32767]_ext - or is this where you now perceive an 'error'?

In which case, is your 'error' such that you 'expect' that a FITS 16-bit SI image would only actually provide PI with the 'positive half' of the range (because a CCD, itself, cannot produce a 'negative' electron count)? In which case, are you making the 15-bit translation such that [0, 32767]_ext is mapped to [0.0, 1.0]_int?

Or do you just consider the image to be 16-bit, and therefore impose a translation of [0, 65535]_ext to [0.0, 1.0]_int?

I hope you don't mind us asking you to 'air your dirty laundry in public' - but, if we at least know what went wrong, and how it will be corrected, then we can help you deal with these issues as they arise.

Cheers,

[Harry page]

Hi Juan

Fabio is not offended    , particularly when He is right 

Let there be peace in imaging land

Harry

[Juan Conejero]

Hi Niall,

So, given a 'standard' FITS 16-bit Signed Integer file, how does PI interpret this? Do you currently set the PI 'internal' [0.0, 1.0]int range to equate to the 16-bit SI range of [-32768, 32767]ext

Yes. And here is the problem. A signed integer image can be interpreted in two ways (assuming a 16-bit signed image):

(1) The [-32768,+32767] range represents the whole dynamic range to which the image is referred. Then we simply remap that range to [0,1] assuming that -32768=black and +32767=white. This is what PixInsight does right now.

(2) The [0,+32767] range stores physical pixel values, and the negative range [-32768,0[ is not used. Some applications are doing this, and it is absolutely correct because the FITS standard allows it. Unfortunately, PixInsight does not support this interpretation, but it will do in the next version. The reason why we have not faced this problem before is because we didn't have calibration in PixInsight.

The solution is, as in the case of floating point data, to let the user define how signed integer images must be interpreted. What I am not sure is which option should be the default one. (2) seems more convenient because many applications are using signed integer data in that way. (1) would guarantee backward compatibility out-of-the-box for signed integer images written with previous versions of PixInsight, but I think that almost nobody has been writing signed FITS images with PI, so a possible compatibility problem would be of marginal importance, if any. Suggestions welcome!

I hope you don't mind us asking you to 'air your dirty laundry in public'

No dirty laundry in PixInsight --just good things, bad things and some mistakes; just as they are present anywhere. When I am wrong or when I make mistakes, I have absolutely no problem to recognize it. Sometimes I am too vehement and that causes (and has always caused) me problems. But since I'm older than 19 there's little hope I'll ever change

[Niall Saunders]

Hi Juan,

The reason why we have not faced this problem before is because we didn't have calibration in PixInsight

Yes - I have to agree there! After all, you are only too familiar with the long struggles that I have been having with Meade's Envisage, struggles that I thought were over forever when I started using PI in earnest, but struggles that returned to trouble me when I started using ImageIntegration and PixMath to 'manually' calibrate my images in PI 1.5.9

I think that you could choose (2) as your 'default' method - and then just wait and see what the feedback is. Some folks will never 'see' the problem, some will 'adapt' and reconfigure PI to suit their needs, and others will need to be helped. Same old same old

And, as far as I am concerned, I think I stopped changing when I was about 14 - I have been an obstinate 'pain in the behind' ever since

Cheers,

[bitli]

Hi,
I think (2) would be more convenient too, it is better to have least surprised when one transfer data from another software. In case of problem it is not clear where to look if two programs are involved.

If we have a problem when reading back PI data in PI, it is always possible to refer to the documentation :-).

This could help:

• Having an integer range of 15 bits in the readout to match the FITS file
• Having the options (for fits, raw,..) currently in the file format used as a default, but editable in the file open dialog box (or may be a button to reach the settings from the open box).  If possible these options should be settable by Javascript when calling ImageWindow.open (may be they are, in fact).
  

-- bitli

[vicent_peris]

(2) The [0,+32767] range stores physical pixel values, and the negative range [-32768,0[ is not used. Some applications are doing this, and it is absolutely correct because the FITS standard allows it. Unfortunately, PixInsight does not support this interpretation, but it will do in the next version. The reason why we have not faced this problem before is because we didn't have calibration in PixInsight.

So Astroart does calculations in 15 bit, as does Iris??

Vicent.

[georg.viehoever]

(1) The [-32768,+32767] range represents the whole dynamic range to which the image is referred. Then we simply remap that range to [0,1] assuming that -32768=black and +32767=white. This is what PixInsight does right now.

(2) The [0,+32767] range stores physical pixel values, and the negative range [-32768,0[ is not used. Some applications are doing this, and it is absolutely correct because the FITS standard allows it. Unfortunately, PixInsight does not support this interpretation, but it will do in the next version. The reason why we have not faced this problem before is because we didn't have calibration in PixInsight.

How about deciding on the interpretation by looking at the values in the FITS file. If there are values in the negative range, we have case (1), otherwise case (2).

Georg

[Niall Saunders]

Hi Georg,

I am not that 'comfortable' with your suggestion - because now we are talking about PI starting to have 'dedicated code' to allow it to interpret the myriad of 'non-conforming' data methods that people are using.

I still feel that the PI approach to date is the most 'robust' - and that even means that I have to 'deal' with my own incorrectly constructed FITS files (thanks to Meade). But at least the solution is relatively simple, and I have no problems implementing it, and I have taken the time to 'understand it' - all of which helps ME remain in control.

Yes, shielding a total novice from the problem is one possible solution - but they don't 'learn' from that approach. After all, if someone actually realises that there is a problem, and can perhaps eliminate that problem 'at source', then wouldn't life would be easier in the long-term? (Not in my particular case - Envisage does NOT have the flexibility to change, and Meade have steadfastly ignored the software bugs for YEARS now)

Cheers,

[harist]

The conceptual error, in my opinion, is in the fact that a CCD does not produce negative values in the range [-32768,0[, so storing physical pixel values as signed integers does not make any sense.

Indeed, and AA4 reports values in the range of 0 to 891 for the bias frame. It is beyond me why it decided to save flats, darks, and lights as 16bit unsigned and bias as signed 16bit.

So Astroart does calculations in 15 bit, as does Iris??

No, it just has a peculiar way of saving data.

BR
Tasos

[georg.viehoever]

Niall,

...
I am not that 'comfortable' with your suggestion - because now we are talking about PI starting to have 'dedicated code' to allow it to interpret the myriad of 'non-conforming' data methods that people are using.
...

I think one aspect of user friendly software design means do as much as possible "behind the scenes". Frankly, I would not mind if PI had some fewer dials and knobs. Even as a long time PI user I am often unsure on what setting is usually best.

How about letting the user decide? PI could offer 3 options: Automatic (what I think should be the default), and your two manual options.

Georg

[Cheyenne]

The "number of dials and switches" isn't  a bad thing.

There is a scientific / mathematical reason for each (or most).  For those who are in the "know" about such things, it would be reasonable (and desirable) to have that level of control.

What would make sense would be to have reasonable defaults and suggestions on which knobs one should tune, for those of us that don't really know (or care) what the connection phi, psi, and filter banks and Ingrid Daubechies could ever be.

[cfranks]

All these problems will go away with PI 1.7.  That's the release with ImageAcquisition right?? 

Charles

[Niall Saunders]

I have been assured that release 2.1.4 (due out at the beginning of April next year) will allow you to download a really nice picture, straight off the internet