New user question

[Martin]

Hello
recently purchased PixInsight and have found the video tutorials very useful indeed (thanks Harry and others).
But I cannot really proceed beyond what is probably something easy to fix.
I use Maxim DL. In that software, they are saved as 16bit FITS and saved as "unsigned format'. 
I then calibrate in Maxim DL, and save them as 32bit integers.
Image alignment appears to work fine with no errors recorded, but when PixInsight then saves those images, it destroys the histogram i.e there is no data left.  Indeed, PixInsight reports 'insufficient data found' when trying to do its own stacking.

I just cant find the software setting that is doing this.

Could someone point me in the right direction?
thanks
Martin

[Harry page]

Hi

Inspect one of your imported images from maxim, if it appears totally bland ( usually grey ) you will need to rescale them ( under intensity transformation )

This will need to be done before you use " image alignment "

let us know how you get on

Harry

[Martin]

Hi Harry
not sure what you mean - the issue occurs after PixInsight has done the alignment and resaves the aligned image. Up to that point, the alignment process works fine, finds plenty of stars, and successfully completes the alignment - it does something after that when saving.

When I reopen those images in Pixinsight, the images are totally black, and when I open them in Maxim, there is no data left to work with.

Martin

[Harry page]

Hi

Are you saying before registrstion you can see stars , please check a single image    ( ie check in pixinsight )

Harry

[Martin]

I cannot see stars in the calibrated image when opening in PI (I can in all other software).  It is just black.

Are you are saying I have to scale 30+ images individually before I start the alignment?  Why would I want to stretch my data even before I have a combined image?
thanks
Martin

[Jack Harvey]

Its been a long time since I tried Maxim for calibration but seem to remember I had some problems with the calibrated images once I got them into PixInsight.  I switched to CCDStack for cal and all is well.  Do you have the option of trying a different program to calibrate your images (CCDStack, DSS, etc)

I know the Maxim format is convenient and quick, but as I said the output did not work for me.

[Harry page]

Hi

Ok next step have you tried to apply the STF to this one image ?   what result do you get ?

Harry

by the way rescaling does not stretch a image , just translates it into a way pixinsight likes it to be ( usually in a range of 0 to one )

Harry

[Harry page]

Hi

Just had a quick go in maxim and see what you mean the image is black 

But trust me apply the rescale to a image and then the STF it will work 

If you want to avoid this rescaling try saving in 16 bit in maxim

Harry

[Martin]

Jack - just calibrated a whole data set in CCDSTACK - still black on opening in PI.
Harry - you say rescale and then STF?

I opened up a black image and applied the STF - but when aligning - I can just select files, not open images, so this will not work unless I open 30+ images, and apply STF to all of them?

thanks
Martin

[Harry page]

Hi


quote "I opened up a black image and applied the STF "?    what was the result of this , can you see a image with just STF applied ?

Please remember that PI opens up a image in liear form ie there is no automatic screen stretch , unless you apply it so images will appear very dark



Harry

[Niall Saunders]

Hi Martin,

I then calibrate in Maxim DL, and save them as 32bit integers.

This is your problem.

I have been struggling with EXACTLY this problem, but because I was using data saved by Meade's "Envisage" capture program, for the DSI range of cameras. The problem is that Envisage saves in 32-bit Float format - which is similar to the way you are saving in Maxim.

With your 32-bit data, considering that your CCD is most likely only digitising to a 16-bit (unsigned integer) range, PI is 'trying' to interpret the data as best as it can. PI does not 'understand' that you actually only have 16-bit data, but that it is stored in a 32-bit environment. It just assumes (incorrectly, as it happens) that your data is all just concentrated in a VERY SMALL part of the MASSIVE integer 32-bit range. In fact the 16-bit range is actually only 1/65536th of the 32-bit range and, in reality, your data is most likely to be concentrated in the bottom 25% of that range.

So, when PI tries to scale your data it just 'assumes' that the data was 'all black' - because that is what the incoming data values are telling it.

However, because you are using Maxim, you will have (hopefully) more flexibility than I did with Envisage. Just ask Maxim to save the images in 16-bit Unsigned Integer FITS format. And, if that option is not available, then I am sorry, but that would be the final evidence that I, personally, would need to tell me that Maxim just isn't worth any part of it's purchase price.

But, even if that IS the case, all is not lost. PixInsight does actually have a simple, if convoluted workaround, that - once you understand it - is actually very easy to implement.

You should take the time to read this thread http://pixinsight.com/forum/index.php?topic=1669.0 - especially the replies around 39 to 45

And then you want to read through this thread http://pixinsight.com/forum/index.php?topic=1696.0 - especially around reply 11 (from Juan)

You have to understand that PI will be working on your data (internally) in a [0.0, 1.0] range - either as 32-bit floating-point data, or sometimes even as 64-bit floating-point data. Because the FITS format IS so 'Flexible' (hence its name!!), PI does NOT try to 'double-guess' what the original data acquisition program 'intended' to achieve when it created the FITS file that you are working with, it simply reads the format AS IT HAS BEEN STORED.

But, PI does give you the opportunity to 'make ammends' for crappy use of the FITS format in the first place (such as writing 16-bit unsigned integer data in a 32-bit floating point format).

I have just been able to take 30-odd each subs (Lights, Darks, Flats, FlatDarks of Lu data), all stored as 32-bit float, and perform a FULL CALIBRATION in PixInsight - ENTIRELY in PixInsight, and I can honestly say that I have NEVER seen such clean data before. EVER.

Yes, I had to jump through a couple of hoops to get the data into a format that PI was happier to be working with, but the hoops were quite large, not too high off the ground, and they weren't on fire and the landing was nice and soft on the other side.

Have a look at those two threads, think about your data acquisition process again (can it be modified, for example), and realise that you will (or should) still be able to recover any data stored in the 'wrong' (32-bit Float) format.

You will be bound to have further questions - that is what we are here for.

If I can find some time, I will (try to) make a video tutorial of the hoop-jumping procedure - but, right now, freshly laundered bed linen is beckoning . . . .

Cheers,

[Jack Harvey]

Ok  Here is my latest experiment.  Took some images which I had previously taken with Maxim camera control (my usual) and then set calibration in Maxim for Dark and Flat calibration.  Calibrated the images and selected Save As.  In the Save as window on the bottom the Trype is Fits and SIze Format is IEEE Float and it works fine for me today.  Stars seen in the frames before calibration, improvement in the image after calibration and stars present in PixInsight with good dark background and a histogram with a nice spike with no black clipping.

[Niall Saunders]

Hi Harry,

If you want to avoid this rescaling try saving in 16 bit in maxim

If that is an option, the problem should go away. As Juan says, unless you have a 32-bit imager (which I would doubt VERY much!!), why are you saving in 32-bit mode? (unless you are a poor cousin, like me, with only a DSI and Envisage to work with).

And, if you are saving in 16-bit mode (because your raw ADU data was 16-bit digitised, into a [0, 65535] integer range) then why save in a 'signed' or 'float' format?

Save your data as 16-bit Unsigned Integer format whenever and wherever you can - until, that is, you finally have a MASTER image ready for PixInsight to work it's magic on.

Only once you are past the Calibration stage should you be considering then using the 'full power' of PixInsight, and would then be saving everything thereafter in 32-Bit Floating-Point format, scaled to [0.0, 1.0].

Cheers,

[Martin]

Thanks all

I have resaved everything as 16 bit and image alignment and integration now works in PI.
The only thing odd now, is that 16 bit unsigned FITS files from Maxim, do not open in CCDSTACK (a BZERO problem somewhere) - but that's CCDSTACK/Maxim issue.

I can understand now how/why PI treats the data and hence the black image. 
I will have to go and recreate all of my master calibration frames, as they were saved as 32bit files - so when you calibrate a 16bit image with a 32bit calibration file, the image is now in the 32bit space  - this is particularly true in CCDSTACK and when you try to save that image, it tells you that you are going to compress the dataspace from 32bit back down to 16bit unless you choose a 32bit option.

So as you can see - the FITS standard is loosely interpretated by the different softwares.

thanks for the help
Martin

[Niall Saunders]

Ok  Here is my latest experiment.  Took some images which I had previously taken with Maxim camera control (my usual) and then set calibration in Maxim for Dark and Flat calibration.  Calibrated the images and selected Save As.  In the Save as window on the bottom the Trype is Fits and SIze Format is IEEE Float and it works fine for me today.  Stars seen in the frames before calibration, improvement in the image after calibration and stars present in PixInsight with good dark background and a histogram with a nice spike with no black clipping.

Hi Jack,

I would be interested to see the FITS headers for the images that you acquire through Maxim. It may well be that there are unique FITS keywords in these files, which later allow the 'authoring' program, Maxim, to 'understand' how the actual data should be 'interpreted'.

PixInsight will never attempt to 'interpret' FITS data files - I agree with Juan on that simple concept. To do so would require Juan to write, and maintain, a library of 'interpretation' routines for every camera manufacturer or acqusition program that chose to 'ignore' or 'adapt' the CFITSIO standard to suit their own needs.

Instead, Juan has adopted the 'most sensible' attitude that, whatever the FITS format used, then the lowest possible value in the range selected to save the data in will be considered to be 'Black Level', and will subsequently become [0.0] in PixInsight's internal model. Similarly, the highest possible value will become 'White Level', and will become [1.0] in the internal model.

Where most of us have failed to recognise this in action - myself included - is when we just 'open' one of our FITS images. We have probably all just seen the image open 'as normal', with the data concentrated down at the 'Black' end, requiring a decent STF, or opening Histo stretch, for us to see anything useful on-screen.

But, this has only 'worked' (even for my 32-bit Float data) because PixInsight simply looked at the incoming data, decided that it was outwith the 'internal' format of [0.0, 1.0], and immediately applied a ReScale process call to it. It does this, because that is what the 'default rule' states that it should do (as defined in the <View> Format Explorer> <Fits> panel). Most of us have probably never had to look at this panel, because our images have 'seemed' to be 'OK'.

However, the moment you try to open something like a Dark frame - which will have data in the range of [P, Q], where P is not '0' and where Q is not '65535' - then the ReScale call will result in a 'false' histogram. Typically, what you will see is your data rescaled such the the 'median' of the dataset is now somewhere around 0.5 in the [0.0, 1.0] world. And, obviously, your 'original' Dark frame subs would NEVER (I hope!!) have a median up at the 50% ADU value (i.e. around 32767)!!!

So, whilst this may NOT be what Martin is seeing, I think that all PI users should start asking themselves how they would like their raw image data to be 'captured and stored'. Apart from the obvious adavntage in disk storage space (even though that is cheap enough nowadays), why complicate matters needlessly?

If your capture software allows you to save in 16-bit Unsigned Integer format, then use this method - because that is how the data is arriving off your CCD chip. There is simply NO advantage, whatsoever, in saving your data in ANY other format. And, if your capture software does NOT allow you to do this, then you have to SERIOUSLY question the reasoning (or ethics?) of programmers who would have you perhaps believe that 'they know better', or that 'more bits are better'.

Cheers,