Starshoot Raw Frames

[Paul Mahoney]

Hi,

As a new user of PixInsight, I have been practising with some old images whilst waiting for the weather here in UK to change for the better.

I was working on a pretty poor set of images of M81 and noticed that the brighter of the stars had dark pixels in the centres. When I checked on a processed image from Maxim DL V5.8 there were no pixels in the centres.

To cut a long story short, I went back to just one Raw frame without any processing at all and opened it in Maxim and PixInsight. The Maxim veiw showed no central pixels but Pixinsight did. Being that PI is the most recent software I have used, I immediately (naughty, I know) thought what is happening in PI. However, before I jumped in with both (Big) feet, I opened the image in Photoshop (using Fits Liberator) and low and behold there were the pixels in the centre.

Now I am wondering what is going on in Maxim. Also Maxim view is flipped in comparison to both PI and PS. Not the end of the world but I still wonder why!

Can anyone throw some light on this for me (I will mention it in the Maxim Group as well). The reason I ask here is that I obviously want to do my processing in PI but how would I get rid of these unwanted central pixels if I am lumbered eith them?

Sorry for the long post and kind regards to all,

Paul
London, UK

[Carlos Milovic]

Hi Paul

I assume that you are working with FITS files, right? If so, the vertical mirroring is caused by the selected coordinate system for that format. By default, bottom-left is used, while other formats use top-left. You may change that in the Format Explorer, editing the FITS preferences.

Could you upload a small sample of your images? Please, do it with the same bit depth and format as the raw images (or, upload a raw image if you can).

[Paul Mahoney]

Hi Carlos and thank you for your reply. I have attached a cropped portion of the Raw image which shows the central star pixels.

I appreciate your comments.

Kind regards,

Paul
London, UK

[Niall Saunders]

Hi Paul,

My very first suspicion is that this is because the data has been saved in 16-bit FITS Unsigned Integer format, but that Maxim is NOT correctly implementing the BSCALE and BZERO parameters.

This is exactly the scenario that is encountered using the Meade Envisage software to capture DSI camera data in 16-bit FITSINT format.

Basically, 16-Bit Unsigned Integer format is not a 'standard' FITS format, the standard 16-bit format is 16-bit Signed Integer format (-32768 through 0 to +32767). Correct implementation requires CORRECT implementation of the BSCALE and BZERO parameters in the FITS HEADER, as well as 'pre-processing' of the raw ADU data PRIOR to saving it to disk. Do this sequence of steps incorrectly and the 'brighest data' (where the MSB of the 16-bit word is 'set high') triggers the application into believing (quite correctly) that this MSB being 'set' actually means that the data is NEGATIVELY SIGNED. The end result is that 'bright' data (like star cores, etc.) actually gets interpreted as 'dim' data instead.

I would be happy to look at one of your RAW fits images if you want to email it to me. At least that way I could tell you whether my hypothesis is applicable in your case.

However, if you KNOW which 'save mode' you are using in MAXIM, that might also help as well (I don't use MAXIM - expensive and overkill for me, sorry)

Cheers,

[Carlos Milovic]

Hi Paul

I agree with Niall. There is a problem between signed and unsigned data. When the image is loaded in PI, out of range values are interpreted as "overflow" data. So, everything above 0.5 is thrown to the other side.
So, what you see in the image: 0.5 is in fact what 0 should be. Your bright features now begin from 0 to 0.5, while dim features go from 0.5 to 1. You have to "flip" the data. One way is to use a PixelMath instance (I'm not that good with it's syntaxis... I'll try to give you the command later). Other workaround involves creating two temporary images, and a mask, so it is very inconvenient for a large set of images.

Well, I'll build that PixelMath instance and let you know how to fix it.


PS: I'm not sure where is the problem... but it won't surprise me if it were the capture software who messed it up.

[Carlos Milovic]

Ok, here is the PixelMath command:

iif($T<0.5,$T+0.5,$T-0.5)

Disable rescaling, and replace target.

If you have a set of images, use a ImageContainer, and drop the PixelMath instance into it to fix them all.

[Paul Mahoney]

Hi again Carlos and Niall,

Wow!!

First, I think I need to explain that I am  very new to Astro Imaging and certainly to the complexities of PixInsight so please bear with me if I appear dumb.

Second, Carlos, the PixelMath formula worked perfectly and I can't thank you enough for working that out for me. I will work out how to use an ImageContainer to batch process future sets of images.

Third, Carlos and Niall, I get a bit confused. Prior to purchasing PI, I sent an email asking them if there was any problem in using their software when capturing my images in Maxim DL V 5.08. (The reason I asked is because I had read posts by Niall who was not best pleased with Maxim and, I believe, Envisage software). The reply I got was 'You'll have no problems at all if you save your raw images as unsigned 16-bit integer FITS files in Maxim DL.' I thought I was doing just this in Maxim. I do not understand the difference between signed and unsigned and I dont get the option to choose between them in Maxim.

Fourth, Should I be kicking 'Butts' at Maxim for the way they save their data or should I dump their software after spending a small fortune on it. If it is not conforming why are all of the Maxim community not 'Up in Arms' with this problem....and finally - what capture software should I be using?

Hope this doesn't sound to rambled, I'm trying to capture my thoughts and piece together my way ahead. Like so often in life, at the moment it seems like one step forward and two back! I want to improve on my equipment in time but feel I've got to learn the basics first with what I've got.

Thanks for your help and kind regards,

Paul
London, UK

[Niall Saunders]

Hi Paul,

Harry has a video tutorial on using the ImageContainer process - worth watching. (I also posted a follow-up, but I need to sort out the links to my updated video, which were scrambled during a change of internet servers).

Enjoy PixInsight - and especially enjoy the benefits you will get here on the Forum. Just remember that there is no question 'too simple' to ask (only questions that are too difficult to answer )

Cheers,

[Niall Saunders]

Sorry Paul - I meant to refer to your other points as well, concerning Maxim.

I am not really sure how to advise you. Like I said, I don't have Maxim - mostly because I balk at their pricing structure, but also because Maxim, as a package, has nothing to offer that I currently require.

I really don't know how Maxim saves its 16-bit data. I was working through the problem with Jack Harvey, but I think I got side-tracked (or there was suddenly a clear night and I selfishly ignored his pleas for assistance in favour of capturing data of my own - sorry, Jack)

What you can do for us is to let us know what the 'Save As' options are when you are capturing data in Maxim. A simple line-by-line text reply is all we need. You could also email me a single FITS image, or perhaps even just a screenshot of the FITS HEADER for the various file types.

For the quality of program that Maxim purports to be, I would really hope that the writers HAVE conformed to 'the FITS standard' (such as it is) - and that 16-bit data IS more 'useable' than that produced in Envisage. I have certainly verified that Nebulosity (even v1.x.x) DOES understand the need to 'correctly' encode its 16-bit images. If Maxim cannot do the same, I would be intrigued to know why not!

As to why this is NOT an issue elsewhere in the image-processing community, it all comes down to 'how' the data is interpreted. Many, many programs delve into the FITS HEADER information and use what they find to try and establish what camera and software was used to create the actual FITS file in the first place, and then they invoke an internal 'workaround' to 'massage' the data back into a usable state.

PixInsight does NOT do this - and I wholeheartedly agree with that approach.

Instead, PI has the strength to allow an individual user to create a workaround themselves (perhaps with the help of others, like St. Carlos in your case ) and this is what I have had to do, because NEITHER of the methods provided by Envisage will work with the new ImageCalibration process now available in PI. So, my workflow now has a simple 'convert all to 16-bit' step, where I invoke an ImageContainer and the SampleFormatConversion process to modify EVERY captured (32-bit Float) image from a given session (to 16-bit Unsigned Integer). The reality is that the conversion process is so rapid that I cannot conceive of 'something else to do' whilst I wait for it to complete.

It is simple, painless, reliable and efficient - just like most things in PI, once you figure them out

Cheers,

[Juan Conejero]

Hi Paul,

Be sure that if there's some problem we'll work to solve it.

The FITS file that you have uploaded has been saved with PixInsight. Could you please upload it as it has been generated originally? That's the only way we can know what's happening.

Niall, Paul and Carlos,

We have detected a problem with signed 16-bit FITS files that store physical pixel values. The problem is in PixInsight and will be corrected soon. See this post. The FITS standard allows storing physical pixel values (which can only be positive) as signed integers. This is conceptually an error (IMO) but legal, and we must support it. Perhaps we have a similar case here?

[Jack Harvey]

I have been using both Maxim and PixInsight since August of 2005 and both have worked seamlessly for me.  The fits header of one of the images I collected with Maxim last night is BITPIX     = 16 /8 unsigned int, 16 & 32 int, -32 & -64 real.  Niall will correct me if I am wrong<G> but I think this means it was collected in 16 bit unsigned integer.  Once the raw frames enter Pixinsight they are changed to 32 bit IEEE floating point, and that is how they stay until the final image is produced at the end.  I do then change  a copy to jpeg for placing on a website and those pesky APOD guys demand a jpeg also<G>.  

Niall's recent treatise on the subject is message #7 in this thread  http://pixinsight.com/forum/index.php?topic=1783.msg10673#msg10673

[Niall Saunders]

Hi Jack,

I am trying to remember our original discussion but, fundamentally, if you are 'expecting' your 16-bit ADU data to be processed as if it was in the range [0, 65535] then this is "16-bit Unsigned Integer" format as far as the FITS specification is concerned.

However, and this is a CRUCIAL 'however', two ESSENTIAL steps MUST be performed by the software that will 'save' this data :-

1.) immediately prior to being transferred to the FITS 'data block', every ADU must first have the value (2^{(16 - 1)}) - i.e. 2¹⁵, or 32768 - subtracted from it
2.) the FITS header block MUST have two essential keywords present, and they should be set correctly - namely "BSCALE = 1" and "BZERO = 32768"

It is then the onus of the 'next software package' (PixInsight in our case) to examine the FITS header block, and observe that "BITPIX = 16"; and that "BSCALE" and "BZERO" are present (they can be present in ALL formats in any case), but that - essentially - the software must detect that "BZERO" is NOT equal to "0". When it does detect this condition, the software MUST then 'add' the value of BZERO back to every stored ADU value, prior to presenting the 'decoded' data for use by the remainder of the software.

So, your data files MUST contain the extra FITS keywords as in (2) above, otherwise I think that you must expect to see all sorts of problems when using the likes of PI

Cheers,

[Jack Harvey]

Agreed, and below are those lines from the same FITs header.  In fact this is the default setting for Maxim I believe as I have never changed it in Maxim 4 or 5.

BSCALE     = 1.0000000000000000
BZERO      = 32768.000000000000

[Paul Mahoney]

Hi again and thank you to all who are contributing to this topic, I am trying very hard to keep up with you all.

Juan, I am sorry I misunderstood what Carlos asked for and I am attaching a cropped portion of the original RAW frame from Maxim and screen shots of the FITS Header. Niall, you will probably be interested in these as well.

I could change to a different  capture program but I like Maxim for syncing my scope using Pinpoint. If there is no other answer to the problem, I am quite happy to use the PixelMath expression that Carlos sent me and I will look up Harry's tutorial to learn how to use the ProcessContainer.

Please let me know if I can supply any further information and again, my thanks to all.

Kind regards,

Paul
London, UK

PS. Nearly forgot....the Save As options in Maxim DL V5.08 are 8-bit Int / 16-bit Int / 32-bit Int / IEEE Float, all with the choice of being compressed or Uncompressed.

[Paul Mahoney]

Sorry, Had to do another post to sent the screen captures of the FITS Header due to limitation on size.

Thanks,

Paul
London, UK