calibration of lightbuckets.com images?

[pfile]

has anyone else successfully calibrated a lightbuckets image with pixinsight?

right now i've got 3 problems. my lights are 4096 by 4096, but the darks and bias frames are 4098x4098. this appears to be some kind of bug and they are regenerating their master dark and bias frames tonight.

the 2nd problem is with the master bias and dark frames... first off, i'm not sure if they have been calibrated. i think they probably have not as the darks seem to exhibit amp glow. secondly, they are saved as f32 format fits files out of maxim. in another thread Juan mentioned that maxim saves f32 as [0.0 - 65535.0], in other words, same as i16 but with floating point numbers. since pixinsight interprets f32 images as [0.0-1.0] by default, you need to tell the fits reader to use [0.0-65535.0] as it's input range. however, i see that the pixel values in the bias frame start at 1388.800048828125 and go up to 2120.828613281250. if i read this fits with range = [0-65535] the bias frame is way too bright with respect to the light. if i tell pixinsight to interpret the input as [1388-65535] then the bias frame is very reasonable and can be subtracted from the light without destroying the light, but i have no way of knowing if 1388 should actually equal 0. the raw light frames are provided as i16 and so i don't think there's any ambiguity about the pixel values in the light.

i have the same problem with the dark frame. i scaled it in a similar fashion but pixinsight gives it a weight of 0 while calibrating. however, this could be because i had to chop off 2 columns and 2 rows to match the lights, and i did it wrong, in which case the dark current pixels in the light and the dark don't actually line up on each other, and so PI finds no correlation.

finally, PI seems to be oversubtracting the flat frame. does the flat need to be normalized in some fashion?

thanks for any advice.

[bitli]

Hi,
I had some problem, but finally as I used LB002 I was realized that the processed image are calibrated. So I only do registration, integration and LRGB combination. It seems that for other scope you have to use the raw images or at least apply the flat to the calibrated (for other readers, LighBucket provide the raw lighht FITS, the half or full cooked light fits, and a master dark, bias and flat for each binning).

I just checked the master flat, darks and bias of my pictures, and they have the proper size (both for binned  x1 and x2). This is for the masters delivered with the LB002 images.  Which imager did you use? Do you get master black, darks,... or raw ones ?

You may want to look at the fits header with the script of Juan.  For a processed image, here is an extract:

Code: [Select]

HISTORY,,Bias Subtraction (Bias 1, 3326 x 2504, Bin1 x 1, Temp -30C,
HISTORY,,Exp Time 0ms)
HISTORY,,Dark Subtraction (Dark 1, 3326 x 2504, Bin1 x 1, Temp -30C,
HISTORY,,Exp Time 1800s)
HISTORY,,Dark-Bias(Bias 1,3326 x 2504,Bin1 x 1,Temp -30C,Exp Time 0ms)
HISTORY,,Flat Field (Flat Luminance 1, Luminance, 3326 x 2504, Bin1 x 1,
HISTORY,,Exp Time 2s)
HISTORY,,Flat-Bias(Bias 1,3326 x 2504,Bin1 x 1,Temp -30C,Exp Time 0ms)
HISTORY,,Flat-Dark (Dark 1,3326 x 2504,Bin1 x 1,Temp -30C,Exp Time 1800s)
HISTORY,,File was processed by PinPoint 5.0.13 at 2010-05-04T03:03:01
HISTORY,,File was processed by PinPoint 5.0.13 at 2010-05-04T03:03:13
HISTORY,,WCS added by PinPoint 5.0.13 at 2010-05-04T03:03:16
HISTORY,,Matched 263 stars from the Gray GSC-ACT Catalog
HISTORY,,Average residual was 0.49 arc-seconds
IMAGETYP,'Light Frame',Type of image
For the master dark the full fits keywords are:

Code: [Select]

C:/Users/jmlugrin/Documents/Astronomie/Photos/2010-05-03 Galaxies NGC 5371/BiasDarkFlats/Dark_Bin1x1_Temp-30C_ExpTime1800s_1266752095.fit
BITPIX,-32,8 unsigned int, 16  32 int, -32  -64 real
BSCALE,1.0000000000000000,physical = BZERO + BSCALE*array_value
BZERO,0.00000000000000000,physical = BZERO + BSCALE*array_value
CBLACK,1369,Initial display black level in ADUs
CCD-TEMP,-30.000001000000001,CCD temperature at start of exposure in C
CSTRETCH,'Medium  ',Initial display stretch mode
CWHITE,1655,Initial display white level in ADUs
EXPOSURE,1800.0000000000000,Exposure time in seconds
EXPTIME,1800.0000000000000,Exposure time in seconds
HISTORY,,Cal Master Dark, 15 inputs
NAXIS,2,number of axes
NAXIS1,3326,fastest changing axis
NAXIS2,2504,next to fastest changing axis
PEDESTAL,-100,Correction to add for zero-based ADU
SIMPLE,T,
SWOWNER,'Stephen G. Cullen',Licensed owner of software
XBINNING,1,Binning factor in width
YBINNING,1,Binning factor in height
Maybe you can compare with yours to see if there was any difference in processing.

I am interested in extending my script to calibrated the LighBucket images, I could apply stuff like 'Juan's bizarre Fits Correction' automatically (this is the reason I started writing the script). I will try with my LB 002 images in the next few day (PI days ).  But if the images do not have the right size, this should probably be corrected at LigthBucket side.

-- bitli

[pfile]

thanks for the reply.

this was on LB-0003. i find that more often than not, lightbuckets' automatic calibration of images fails. and as you say, if your position angle is not 0.0deg, the flat is not used in lightbuckets' calibration anyway. so it's probably better to calibrate on your own.

in this case, the lightbuckets calibration failed, so i got the raw lights. they do not provide the subs for the bias, darks and flats.

i think the biggest problem to solve here is mapping the pixel values in the calibrated dark and bias frames properly. to be honest i don't understand why they are so high to begin with. perhaps maxim understands these files okay for calibration, but they don't seem very portable.

[Jack Harvey]

The issue with the 4096 light frames vs the 4098 cal frames could be from the Apogee firmware (assuming they have an Apogee camera on the scope you used).  I had this problem with my U 16 camera here too when I used a old set of cal frames (4098) and after a obs crash and needing to re install camera firmware ended up with 4096 light frames.  After some queries I found out, Apogee decided to change the camera frame size to 4096 to get rid of some edge effects; and in the process forgot to tell their customers about the change in the new firmware edition.  About par for them sometimes.

Vicent told me to go to Geometry and select crop (not dynamic  crop) and readjust the frame size for the 4096 light frames by adding a row of pixels on each edge.  So add 1 on each of the four sides and then re save your 4096 images, which will now be 4098 and should calibrate (mine did).

Of course the answer is to be sure to get the new 4096 cal frames being generated by them when they do.

[pfile]

thanks jack. i had deleted 2 pixels on the right and bottom from the calibration masters, somewhat at random. i'll delete one all the way around and try again.

can you comment about the levels in the bias and darks? when you bring these into pixinsight, what range values do you use in the FITS reader?

[bitli]

Here is the illustration of a part of the problem, with an image of LB 002.

Top row:  Raw L, L calibrated by LightBucket, L calibrated with PI (problems are obvious),
Second row: the master bias, flat and dark from LightBucket.
Below them,  the corresponding statistics.

All images Autostretched with STF defaults (so quite aggressively).

Now I need to understand the explanations of Juan and find if the problem comes from the FITS format or because dark/bias were already applied to the flat or something else.  There seem to be a bad column too (it is not visible in the reduced image below for the dark and bias, but is present for them too).

-- bitli

[pfile]

that flat gray calibrated image is what happens if you read the bias frame in with the default fits range of [0.0-1.0] or even juan's suggestion for maximDL f32 fits files ([0.0.-65535.0]). if you tell the fits reader to "always ask" for the range, it will tell you the minimum value in the file and the maximum value in the file. in my case the minimum was 1388.800048828125, so i told the reader to use [1388.0-65535.0]. when i subtracted that bias file manually it was reasonably successful - the amp glow in the corners had been subtracted out.

while this works, it's not exactly scientific. i'm not sure if the lowest pixel value in the bias frame is really pure black.

in your example above, the fits header does have this stuff:

CBLACK,1369,Initial display black level in ADUs
CSTRETCH,'Medium  ',Initial display stretch mode
CWHITE,1655,Initial display white level in ADUs

i guess i thought this was just there for maxim to record what it's "screen stretch" values were. but maybe we can take this to mean that black = 1369 in your file. try reading the bias frame with 1369-65535 in the fits reader and see if you can subtract it from the light frame using pixel math.

finally, is it legitimate to apply the same technique to the dark frame?

and regardless of all of that, somehow my flats are being over-subtracted in the ImageCalibration tool. that's why i wondered if they have to be normalized somehow.

[pfile]

i am talking to myself here, but as far as LB-0003 goes, i think i'd better wait for them to deliver the new calibration masters.

for fun i read in the dark as i described above, and tried to calibrate with it, choosing optimization and bias subtraction. PI found no correlation. so then i manually subtracted the bias frame from the dark, and then scaled the dark with pixel math. i then calibrated a light frame with the bias frame and the manually calibrated dark; here is a 9:1 zoom on part of the image:



as you can see the dark is shifted up about 10 pixels. i guess the apogee firmware update must have done more than just change the size of the file...

[bitli]

From http://www.cyanogen.com/help/maximdl/FITS_File_Header_Definitions.htm

CBLACK – indicates the black point used when displaying the image (screen stretch).
CWHITE – indicates the white point used when displaying the image (screen stretch).

So these values do not help getting mathematically correct black/white.

I did set the FITS reader mode to ask if values are out of range, but when I get out of the field of upper value after entering 65536, it changes itself to -9223.372036854776500
    

-- bitli

[Niall Saunders]

but when I get out of the field of upper value after entering 65536, it changes itself to -9223.372036854776500

I have replied to this in the 'Bug Reports' section.

I have exactly the same problem. Juan is aware of it, but believes that the cause is using a wireless keyboard. There is a 'kludgy' workaround - which does work.

However, I am NOT convinced that a 'wireless keyboard', that otherwise performs flawlessly, is the root of this 'bug' - but I am willing to wait and see if v1.6.1 makes any difference (I only became aware of the issue somewhere back in v1.5.x)

Cheers,

[pfile]

From http://www.cyanogen.com/help/maximdl/FITS_File_Header_Definitions.htm

CBLACK – indicates the black point used when displaying the image (screen stretch).
CWHITE – indicates the white point used when displaying the image (screen stretch).

So these values do not help getting mathematically correct black/white.

I did set the FITS reader mode to ask if values are out of range, but when I get out of the field of upper value after entering 65536, it changes itself to -9223.372036854776500
    

-- bitli

well, you saw steve's reply on lb.com. he seems to feel that if the images work with maxim and ccdstack then it's our problem.

i'm starting to think that even though those keywords are for display only that both maxim and ccdstack might be using them to determine the blackpoint in the file. and i'll bet that if you save the stacked bias master as i16 maxim would put the same levels in the file, so that's probably not a solution.

it does not make sense to me for bias frames of a CCD cooled to -30C to have such high values. maxim has this goofy "auto stretch" setting when saving files, and i'll bet that's got something to do with the pixel values in these calibration frames.

i'm still having problems with the flat 'subtraction' when running ImageCalibration (obviously it's division...) i manually divided the flat into the light and found by trial and error that i had to compress the flat histogram into the 0.990 - 1.0 range or the result was wrong. and if i used the modified flat in ImageCalibration, the flat got scaled somehow and the results were the same.

[Niall Saunders]

well, you saw steve's reply on lb.com. he seems to feel that if the images work with maxim and ccdstack then it's our problem

Well, whoever 'steve' is, he is entitled to his opinion.

That does not make him 'right', and it further demonstrates that Maxim and CCD Stack must be pieces of software that are quite happy to 'kludge' their way to a solution. And, if that is the case, it also demonstrates that they are probably, more than likely, at the 'root' of the problem as well.

It took me a little time to appreciate Juan's stance on this - and it really is quite simple at the end of the day. Until such time as our CCD cameras have analogue-to-digital converters in them that provide an output with MORE than 16 bits of resolution (and that would have to be more than 16 bits of USEFUL resolution) then there is - in my opinion, of course - absolutely NO justifiable reason to save that image data in anything OTHER than 16-bit format.

If someone DOES HAVE a reason for doing this - with RAW CCD DATA - then I would like to hear it, in order that I might stand corrected.

And, if a piece of software is then going to use a 16-bit FITS format, then the programmer had better spend some time understanding how to use the BSCALE and BZERO (standard) FITS keywords. (Heck, it took me less than an evening to figure it out for myself, and I had NO IDEA what the FITS system even was at the time)

And, finally, the CBLACK and CWHITE keywords are TOTALLY NON-STANDARD. They could just as easily have been called MICKEY and MOUSE, and they would have done EXACTLY the same job, but ONLY for those few 'kludged' software packages that agreed to use them.

And it is for these reasons that I cannot ever see myself voluntarily using 'kludgeware' such as CCDStack or Maxim - after all, if they are willing to include this kind of 'work around', what other 'horrors' could be lurking in the depths of their code?

Juan has stuck, with quite rightful determination, to the 'core' of the FITS 'standard'. And, if everyone else did so as well, life would be MUCH simpler for astro-imagers. The alternative would be an ever changing library of code that would be required for every CCD in the market place (discontinued, old, new, and prototype), which would be out of date as soon as someone changed a CCD driver (as appears to have been the case with the cameras in this particular case).

I guarantee that there would be a h3ll of a lot more complaints if THAT was how we had to get our calibration data into PixInsight.

</sound of high-horse being returned to cupboard>

Cheers,

[Juan Conejero]

seems to feel that if the images work with maxim and ccdstack then it's our problem.

Welcome to my world. I've had to live with that way of treating us since 2004. With PixInsight, you at least can always know what is black and what is white <G>.

[bitli]

Just to clarify the issue (I hope): the raw images of LB are in 16 bits and are perfectly readable. They do not provide raw dark,flat,bias.

The master images cause the problem. They have been processed by MAximDL (that's why they are master images), and it makes perfect sense to save them in 32 bits floating point (you would do the same in PI).
What makes NO sense is that they are saved as 32 bit floating AND have a range (probably) from 0 o 2^16-1 AND there is no hint or  value in the FITS keywords to guess the white level. As to state if FITS allows or does not allow floating point values outside a range of 0 to 1.0 I do not know neither really care, but as there is no way to read an image without external knoweldege, there is indeed a problem.

Apparently MaximDL and CCDStack do not care too much of the white level or assume that it is 2^16-1.  There are some explanation on how to rescale a floating point image so that it is between 0 and 1 in the documentation of CCDStack, so it seems to be a task for the user.

Anyhow, it is unlikely that maximdl modifies their product so that we can easily use PI instead of their product...

As for CBLACK and so on, the documentation of Maxim-DL indicates clearly that it is one of their extensions. I can live with that.

-- bitli

[pfile]

well, you saw steve's reply on lb.com. he seems to feel that if the images work with maxim and ccdstack then it's our problem

Well, whoever 'steve' is, he is entitled to his opinion.

That does not make him 'right', and it further demonstrates that Maxim and CCD Stack must be pieces of software that are quite happy to 'kludge' their way to a solution. And, if that is the case, it also demonstrates that they are probably, more than likely, at the 'root' of the problem as well.

steve owns lightbuckets, owns the telescopes, etc. etc. they use maximDL as their main capturing/processing tool.

i may be putting words in his mouth here. he has always been very accommodating when things go wrong and they are pretty customer-oriented for what they are doing, but when it comes to maximDL they seem to draw a line in the sand. at some point their auto-calibration failed, and i could not calibrate my own files. at the time i did not know why - it turns out to be the problem i am discussing now. but anyway they went back and calibrated the files for me by hand with maximDL, but the files were saved as compressed fits. pixinsight could not open these files; i thought they were corrupted. i encountered a slight bit of resistance but they eventually resaved them as uncompressed fits files for me. at the time though the reaction was sort of like "these open okay in maximDL, so, uh...?"

It took me a little time to appreciate Juan's stance on this - and it really is quite simple at the end of the day. Until such time as our CCD cameras have analogue-to-digital converters in them that provide an output with MORE than 16 bits of resolution (and that would have to be more than 16 bits of USEFUL resolution) then there is - in my opinion, of course - absolutely NO justifiable reason to save that image data in anything OTHER than 16-bit format.

If someone DOES HAVE a reason for doing this - with RAW CCD DATA - then I would like to hear it, in order that I might stand corrected.

And, if a piece of software is then going to use a 16-bit FITS format, then the programmer had better spend some time understanding how to use the BSCALE and BZERO (standard) FITS keywords. (Heck, it took me less than an evening to figure it out for myself, and I had NO IDEA what the FITS system even was at the time)

And, finally, the CBLACK and CWHITE keywords are TOTALLY NON-STANDARD. They could just as easily have been called MICKEY and MOUSE, and they would have done EXACTLY the same job, but ONLY for those few 'kludged' software packages that agreed to use them.

And it is for these reasons that I cannot ever see myself voluntarily using 'kludgeware' such as CCDStack or Maxim - after all, if they are willing to include this kind of 'work around', what other 'horrors' could be lurking in the depths of their code?

i'm really just guessing here about what maxim is doing with CBLACK and CWHITE. i have to assume though that maxim is doing something before using these files as calibration masters... the levels are just way too high. they are actually being written out as f32, but apparently with a range of [0.0-65535.0].