PixInsight Forum (historical)
PixInsight => General => Off-topic => Topic started by: Martin on 2010 February 21 14:11:40
-
Hello
recently purchased PixInsight and have found the video tutorials very useful indeed (thanks Harry and others).
But I cannot really proceed beyond what is probably something easy to fix.
I use Maxim DL. In that software, they are saved as 16bit FITS and saved as "unsigned format'.
I then calibrate in Maxim DL, and save them as 32bit integers.
Image alignment appears to work fine with no errors recorded, but when PixInsight then saves those images, it destroys the histogram i.e there is no data left. Indeed, PixInsight reports 'insufficient data found' when trying to do its own stacking.
I just cant find the software setting that is doing this.
Could someone point me in the right direction?
thanks
Martin
-
Hi
Inspect one of your imported images from maxim, if it appears totally bland ( usually grey ) you will need to rescale them ( under intensity transformation )
This will need to be done before you use " image alignment "
let us know how you get on
Harry
-
Hi Harry
not sure what you mean - the issue occurs after PixInsight has done the alignment and resaves the aligned image. Up to that point, the alignment process works fine, finds plenty of stars, and successfully completes the alignment - it does something after that when saving.
When I reopen those images in Pixinsight, the images are totally black, and when I open them in Maxim, there is no data left to work with.
Martin
-
Hi
Are you saying before registrstion you can see stars , please check a single image :D ( ie check in pixinsight )
Harry
-
I cannot see stars in the calibrated image when opening in PI (I can in all other software). It is just black.
Are you are saying I have to scale 30+ images individually before I start the alignment? Why would I want to stretch my data even before I have a combined image?
thanks
Martin
-
Its been a long time since I tried Maxim for calibration but seem to remember I had some problems with the calibrated images once I got them into PixInsight. I switched to CCDStack for cal and all is well. Do you have the option of trying a different program to calibrate your images (CCDStack, DSS, etc)
I know the Maxim format is convenient and quick, but as I said the output did not work for me.
-
Hi
Ok next step have you tried to apply the STF to this one image ? what result do you get ?
Harry
by the way rescaling does not stretch a image , just translates it into a way pixinsight likes it to be ( usually in a range of 0 to one )
Harry
-
Hi
Just had a quick go in maxim and see what you mean the image is black ???
But trust me apply the rescale to a image and then the STF it will work ;D
If you want to avoid this rescaling try saving in 16 bit in maxim
Harry
-
Jack - just calibrated a whole data set in CCDSTACK - still black on opening in PI.
Harry - you say rescale and then STF?
I opened up a black image and applied the STF - but when aligning - I can just select files, not open images, so this will not work unless I open 30+ images, and apply STF to all of them?
thanks
Martin
-
Hi
quote "I opened up a black image and applied the STF "? what was the result of this , can you see a image with just STF applied ?
Please remember that PI opens up a image in liear form ie there is no automatic screen stretch , unless you apply it so images will appear very dark
Harry
-
Hi Martin,
I then calibrate in Maxim DL, and save them as 32bit integers.
This is your problem.
I have been struggling with EXACTLY this problem, but because I was using data saved by Meade's "Envisage" capture program, for the DSI range of cameras. The problem is that Envisage saves in 32-bit Float format - which is similar to the way you are saving in Maxim.
With your 32-bit data, considering that your CCD is most likely only digitising to a 16-bit (unsigned integer) range, PI is 'trying' to interpret the data as best as it can. PI does not 'understand' that you actually only have 16-bit data, but that it is stored in a 32-bit environment. It just assumes (incorrectly, as it happens) that your data is all just concentrated in a VERY SMALL part of the MASSIVE integer 32-bit range. In fact the 16-bit range is actually only 1/65536th of the 32-bit range and, in reality, your data is most likely to be concentrated in the bottom 25% of that range.
So, when PI tries to scale your data it just 'assumes' that the data was 'all black' - because that is what the incoming data values are telling it.
However, because you are using Maxim, you will have (hopefully) more flexibility than I did with Envisage. Just ask Maxim to save the images in 16-bit Unsigned Integer FITS format. And, if that option is not available, then I am sorry, but that would be the final evidence that I, personally, would need to tell me that Maxim just isn't worth any part of it's purchase price.
But, even if that IS the case, all is not lost. PixInsight does actually have a simple, if convoluted workaround, that - once you understand it - is actually very easy to implement.
You should take the time to read this thread http://pixinsight.com/forum/index.php?topic=1669.0 (http://pixinsight.com/forum/index.php?topic=1669.0) - especially the replies around 39 to 45
And then you want to read through this thread http://pixinsight.com/forum/index.php?topic=1696.0 (http://pixinsight.com/forum/index.php?topic=1696.0) - especially around reply 11 (from Juan)
You have to understand that PI will be working on your data (internally) in a [0.0, 1.0] range - either as 32-bit floating-point data, or sometimes even as 64-bit floating-point data. Because the FITS format IS so 'Flexible' (hence its name!!), PI does NOT try to 'double-guess' what the original data acquisition program 'intended' to achieve when it created the FITS file that you are working with, it simply reads the format AS IT HAS BEEN STORED.
But, PI does give you the opportunity to 'make ammends' for crappy use of the FITS format in the first place (such as writing 16-bit unsigned integer data in a 32-bit floating point format).
I have just been able to take 30-odd each subs (Lights, Darks, Flats, FlatDarks of Lu data), all stored as 32-bit float, and perform a FULL CALIBRATION in PixInsight - ENTIRELY in PixInsight, and I can honestly say that I have NEVER seen such clean data before. EVER.
Yes, I had to jump through a couple of hoops to get the data into a format that PI was happier to be working with, but the hoops were quite large, not too high off the ground, and they weren't on fire and the landing was nice and soft on the other side.
Have a look at those two threads, think about your data acquisition process again (can it be modified, for example), and realise that you will (or should) still be able to recover any data stored in the 'wrong' (32-bit Float) format.
You will be bound to have further questions - that is what we are here for.
If I can find some time, I will (try to) make a video tutorial of the hoop-jumping procedure - but, right now, freshly laundered bed linen is beckoning . . . . ::)
Cheers,
-
Ok Here is my latest experiment. Took some images which I had previously taken with Maxim camera control (my usual) and then set calibration in Maxim for Dark and Flat calibration. Calibrated the images and selected Save As. In the Save as window on the bottom the Trype is Fits and SIze Format is IEEE Float and it works fine for me today. Stars seen in the frames before calibration, improvement in the image after calibration and stars present in PixInsight with good dark background and a histogram with a nice spike with no black clipping.
-
Hi Harry,
If you want to avoid this rescaling try saving in 16 bit in maxim
If that is an option, the problem should go away. As Juan says, unless you have a 32-bit imager (which I would doubt VERY much!!), why are you saving in 32-bit mode? (unless you are a poor cousin, like me, with only a DSI and Envisage to work with).
And, if you are saving in 16-bit mode (because your raw ADU data was 16-bit digitised, into a [0, 65535] integer range) then why save in a 'signed' or 'float' format?
Save your data as 16-bit Unsigned Integer format whenever and wherever you can - until, that is, you finally have a MASTER image ready for PixInsight to work it's magic on.
Only once you are past the Calibration stage should you be considering then using the 'full power' of PixInsight, and would then be saving everything thereafter in 32-Bit Floating-Point format, scaled to [0.0, 1.0].
Cheers,
-
Thanks all
I have resaved everything as 16 bit and image alignment and integration now works in PI.
The only thing odd now, is that 16 bit unsigned FITS files from Maxim, do not open in CCDSTACK (a BZERO problem somewhere) - but that's CCDSTACK/Maxim issue.
I can understand now how/why PI treats the data and hence the black image.
I will have to go and recreate all of my master calibration frames, as they were saved as 32bit files - so when you calibrate a 16bit image with a 32bit calibration file, the image is now in the 32bit space - this is particularly true in CCDSTACK and when you try to save that image, it tells you that you are going to compress the dataspace from 32bit back down to 16bit unless you choose a 32bit option.
So as you can see - the FITS standard is loosely interpretated by the different softwares.
thanks for the help
Martin
-
Ok Here is my latest experiment. Took some images which I had previously taken with Maxim camera control (my usual) and then set calibration in Maxim for Dark and Flat calibration. Calibrated the images and selected Save As. In the Save as window on the bottom the Trype is Fits and SIze Format is IEEE Float and it works fine for me today. Stars seen in the frames before calibration, improvement in the image after calibration and stars present in PixInsight with good dark background and a histogram with a nice spike with no black clipping.
Hi Jack,
I would be interested to see the FITS headers for the images that you acquire through Maxim. It may well be that there are unique FITS keywords in these files, which later allow the 'authoring' program, Maxim, to 'understand' how the actual data should be 'interpreted'.
PixInsight will never attempt to 'interpret' FITS data files - I agree with Juan on that simple concept. To do so would require Juan to write, and maintain, a library of 'interpretation' routines for every camera manufacturer or acqusition program that chose to 'ignore' or 'adapt' the CFITSIO standard to suit their own needs.
Instead, Juan has adopted the 'most sensible' attitude that, whatever the FITS format used, then the lowest possible value in the range selected to save the data in will be considered to be 'Black Level', and will subsequently become [0.0] in PixInsight's internal model. Similarly, the highest possible value will become 'White Level', and will become [1.0] in the internal model.
Where most of us have failed to recognise this in action - myself included - is when we just 'open' one of our FITS images. We have probably all just seen the image open 'as normal', with the data concentrated down at the 'Black' end, requiring a decent STF, or opening Histo stretch, for us to see anything useful on-screen.
But, this has only 'worked' (even for my 32-bit Float data) because PixInsight simply looked at the incoming data, decided that it was outwith the 'internal' format of [0.0, 1.0], and immediately applied a ReScale process call to it. It does this, because that is what the 'default rule' states that it should do (as defined in the <View> Format Explorer> <Fits> panel). Most of us have probably never had to look at this panel, because our images have 'seemed' to be 'OK'.
However, the moment you try to open something like a Dark frame - which will have data in the range of [P, Q], where P is not '0' and where Q is not '65535' - then the ReScale call will result in a 'false' histogram. Typically, what you will see is your data rescaled such the the 'median' of the dataset is now somewhere around 0.5 in the [0.0, 1.0] world. And, obviously, your 'original' Dark frame subs would NEVER (I hope!!) have a median up at the 50% ADU value (i.e. around 32767)!!!
So, whilst this may NOT be what Martin is seeing, I think that all PI users should start asking themselves how they would like their raw image data to be 'captured and stored'. Apart from the obvious adavntage in disk storage space (even though that is cheap enough nowadays), why complicate matters needlessly?
If your capture software allows you to save in 16-bit Unsigned Integer format, then use this method - because that is how the data is arriving off your CCD chip. There is simply NO advantage, whatsoever, in saving your data in ANY other format. And, if your capture software does NOT allow you to do this, then you have to SERIOUSLY question the reasoning (or ethics?) of programmers who would have you perhaps believe that 'they know better', or that 'more bits are better'.
Cheers,
-
Thanks Niall for the additional explanation.
So, when a 16bit unsigned FIT created in Maxim does not open correctly in CCDSTACK (saturated areas are black e.g star cores), then is this likely to be how Maxim writes the files or how CCDSTACK reads it?
thanks
Martin
-
The only thing odd now, is that 16 bit unsigned FITS files from Maxim, do not open in CCDSTACK (a BZERO problem somewhere) - but that's CCDSTACK/Maxim issue.
And, there we see the nail being firmly hit on the head, twice !!
Several software packages obviously have failed to understand how to implement 16-bit Unsigned Integer data using the FITS specification - and yet it is NOT difficult to do (even without CFITSIO) if you sit down and take your time (it took me one evening, three large coffees, and about three sheets of paper to note all the differences down). I knew that Envisage had made this mistake (quite amusing if not so tragically sad, Meade's OWN software could not even read images that THEY created!!!) - and despite having pointed this out to them, the error STILL exists - after several YEARS now. It seems now that the 'Lord High and Mighty' of acquisition programs may also be capable of the same error (I can't confirm this as, for as long as I can possibly do so, I will NOT invest the kind of money they are asking for their software, even more so if they have 'bugs' like this).
And your final statement,
So as you can see - the FITS standard is loosely interpretated by the different softwares.
is the most damming. The FITS standard IS 'flexible', bith by name and by nature. But, therein lies its 'power'. It is only when programmers do not 'understand' this flexibility that problems arise.
Fortunately, for us, Juan DOES appreciate the 'flexibility' issues, which is why he cannot allow PixInsight to 'bend' to accommodate them. PixInsight adheres, strictly, to the FITS standard - and ONLY uses the data available in the 'compulsory' FITS header keywords, and only uses that data in the manner originally intended.
If everybody else 'worked to rule' then we wouldn't have these problems.
Cheers,
(sound of soap-box being pushed back into cupboard, and 'click' of megaphone being switched off. Rant over :footinmouth:)
-
Thanks Niall for the additional explanation.
So, when a 16bit unsigned FIT created in Maxim does not open correctly in CCDSTACK (saturated areas are black e.g star cores), then is this likely to be how Maxim writes the files or how CCDSTACK reads it?
Hi Martin,
That would be dependent on how the header and data sections of the Fits file have been created in Maxim.
Now, as I refer to my 'FITS Bible' (i.e. the notes I gathered on my long learning session, back on 21/10/07, Chapter One, Verse One):
There are 5 possible FITS Data Unit types
8-bit Unsigned Integers, where BITPIX=8
16-Bit Signed Integers, where BITPIX=16
32-Bit Signed Integers, where BITPIX=32
32-Bit Single Precision Floating-Point Real Numbers, where BITPIX=-32
64-Bit Double Precision Floating-Point Real Numbers, where BITPIX=-64
There is also a proposal to include the following (but I do not believe that it has, as yet, been ratified)
64-Bit Signed Integers, where BITPIX (would be)=64
As you can see , there is NO 'standard' for 16-Bit Unsigned Integers - which is unfortunate, because this is EXACTLY what current CCD imagers need.
However, the 'standard' can be 'expanded' to allow for 16-Bit Unsigned and 32-Bit Unsigned Integers as follows:-
Subtract 2^(BITPIX-1) from every data value, before saving
Then save the data, using BITPIX=16 (or BITPIX=32, for 32-bit data),
but ALSO incorporate two extra FITS Header keywords; BSCALE=1 and BZERO=(2^(BITPIX-1))
Now, when the image is to be opened, if the BSCALE=1 parameter is present,
Add 2^(BITPIX-1) back to every stored data value as it is passed to the parent program
For 16-bit data, 2^(BITPIX-1) = 32768 and for 32-bit data 2^(BITPIX-1) = 2 147 483 648
Usually, what happens is that this crucial stage is either NOT implemented, or is implemented incorrectly (such as failure to include BOTH of the extra keywords, or failure to 'subtract before saving'). Alternatively, images might be saved correctly, but applications attempting to read the format have no idea how to process the incoming data.
In any case, however the failure occurs, it is just shabby programming, made worse by the failure to acknowledge the fact, and then worse still by the failure to do anything about the problem.
Hope this background helps.
Cheers,
-
Howdy Niall
a raw frame downloaded from Maxim says in the fits: BitPix 16/8 unsigned int, 16 & 32 int, -32 & 64 real
-
Sorry Jack,
I could not clearly interpret your layout of the BITPIX summary.
Can you compare it with my 'understanding', and let me know where the differences are?
Cheers,
-
This why Juan need to release the calibration module so we can work on the stack from the start of the process. Hopefully soon.
Max
-
In the "Fits Header" in Maxim the BIT Pix entry is the one we are interested in. That is how it is printed out and I suspect Doug George the Maxim guru would have to give you what you ask for - sorry:-((
And I too are eagerly awaiting the ability to easily calibrate in PixInsight.
And finally (for me) Welcome aboard Martin!
-
Hi Jack,
the ability to easily calibrate in PixInsight
Well, the 'ease' will be dependent on the calibration data being correctly interpreted by PI - so this 'FITS' discussion is probably going to affect more and more people.
I would love to see the actual header from a 'typical' Maxim sub-frame. Feel free to email me one.
You could also Google for <FITS verifier online> - there is an online utility that should test your FITS file for correct use of 'the standard'.
Cheers,
-
And of course many of us create our calibration frames differently, so this will be a discussion also<G>. The Fits Header from Maxim is sent to your email.
-
Thanks for the Maxim header Jack, I saw it pop up on my iPhone, but I need to look at it in more detail once I am back home.
Cheers,
-
Good thats that all sorted then 8)
Harry
-
I'm so glad I only work with CR2 and Tiff. :P :D