Show Posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.


Messages - pfile

Pages: 1 ... 263 264 [265] 266 267
3961
Bug Reports / Re: ImageCalibration problem
« on: 2010 May 22 10:52:19 »
i can't explain this yet, but i have tried what you are suggesting and i get the same result, whether i divide the flat or i use ImageCalibration. the science frame is still way blown out. i guess i still don't understand the basics here - if you divide any part of that image by 0.65, won't that part just be ridiculously bright?

are you also using pixinsight 1.6 ?

as for the raw-ness of the science frame, there are definitely donuts there, but they are hard to see. they are more obvious when i've stacked the 20 uncalibrated science frames i have and also applied curves. in the attachment i've also given a crazy STF so they can be seen. i did ABE on the image to try to correct the vignetting.

the cause of the junk at the bottom end of the histogram seems to be a stacking artifact. if you look at the bottom of the flat there are 10 rows of dark pixels. it's almost like maximDL has tried to register the flat subexposures before stacking them. in fact my darks look the same way. this would be a fatal error as far as i'm concerned. EDIT: i think this stuff is actually an artifact in the old apogee driver. i can see that my raw science frames from last year look the same.

also, NKV: all the calibration checkboxes were unticked.


3963
Bug Reports / ImageCalibration problem
« on: 2010 May 21 22:23:28 »
as seen near the end of this thread: http://pixinsight.com/forum/index.php?topic=1930.msg12207#msg12207



the flat frame started life as an f32 fits file, as integrated by MaximDL. i have converted it to i16 to match the raw science frame.

if i use the flat frame as-is in ImageCalibration, the result is as in the top right image in the screenshot above: the flat has pretty much destroyed the science frame.
if i use the histogram tool to compress the data in the flat all the way to the right and use that new flat in ImageIntegration, the results are satisfactory.

it seems like ImageCalibration is not scaling the flat properly? or is this user error?

here are the science and flat files in question:

http://public.me.com/ix/pfile/pixinsight_1.6_flat_problem.zip?disposition=download+1274505738440

thanks.


3964
General / Re: calibration of lightbuckets.com images?
« on: 2010 May 21 13:49:58 »
when i get home tonight i can post the fits files somewhere. maybe someone can take a look at this, perhaps it is a bug?

of course in this example i have not calibrated the master flat frame (nor have i set bias or darks when calibrating the science frame) but i don't think it should matter - i get the same result if i put the master bias and master dark into ImageCalibration as well.

3965
General / Re: calibration of lightbuckets.com images?
« on: 2010 May 21 13:17:30 »
can someone advise me regarding this?



the top row of images are : the raw science frame, the master flat frame and the resultant calibrated frame. the statistics window is set to the master flat frame.
the bottom row of images are: the flat frame, scaled with the histogram tool as shown in the histogram settings. the calibrated frame is on the right. it looks pretty reasonable.

is it to be expected that i'd have to manipulate the flat this way in order to get correct division in ImageCalibration? clearly the topmost calibrated frame is completely screwed up. my problem with this is that what i've done to the flat to make the calibration right is completely ad-hoc.

3966
General / Re: calibration of lightbuckets.com images?
« on: 2010 May 19 23:33:22 »
on #2 that looks like what happened to me when the file was saved with compression in maximDL.

3967
General / Re: calibration of lightbuckets.com images?
« on: 2010 May 19 13:40:21 »
do you get a dark scaling factor of 0.000 when it tries to subtract the dark from the light frame?

this might be specific to the recent driver updates on LB-0003, but the darks i currently have are shifted up by 10 pixels with respect to the lights. this causes pixinsight to fail to find a correlation between the light and the dark, and so it can not scale the dark.

i do have some older data from LB-0002, but i never tried to calibrate it with PI. i'll have to give that a try and see what i get.

i am still having problems with pixinsight's handling of the flat frame. can i upload one light and the master flat somewhere and maybe juan or vincent can look at it? (did vincent write the ImageCalibration tool?)

3968
Announcements / Re: Version 1.6.1 — Delayed
« on: 2010 May 19 11:43:53 »
thanks for the heads-up. it will be worth the wait for the 64-bit macosx version, for me anyway. i've made the leap to snow leopard on my main computer.

3969
General / Re: calibration of lightbuckets.com images?
« on: 2010 May 19 11:20:46 »
Quote
well, you saw steve's reply on lb.com. he seems to feel that if the images work with maxim and ccdstack then it's our problem

Well, whoever 'steve' is, he is entitled to his opinion.

That does not make him 'right', and it further demonstrates that Maxim and CCD Stack must be pieces of software that are quite happy to 'kludge' their way to a solution. And, if that is the case, it also demonstrates that they are probably, more than likely, at the 'root' of the problem as well.

steve owns lightbuckets, owns the telescopes, etc. etc. they use maximDL as their main capturing/processing tool.

i may be putting words in his mouth here. he has always been very accommodating when things go wrong and they are pretty customer-oriented for what they are doing, but when it comes to maximDL they seem to draw a line in the sand. at some point their auto-calibration failed, and i could not calibrate my own files. at the time i did not know why - it turns out to be the problem i am discussing now. but anyway they went back and calibrated the files for me by hand with maximDL, but the files were saved as compressed fits. pixinsight could not open these files; i thought they were corrupted. i encountered a slight bit of resistance but they eventually resaved them as uncompressed fits files for me. at the time though the reaction was sort of like "these open okay in maximDL, so, uh...?"

It took me a little time to appreciate Juan's stance on this - and it really is quite simple at the end of the day. Until such time as our CCD cameras have analogue-to-digital converters in them that provide an output with MORE than 16 bits of resolution (and that would have to be more than 16 bits of USEFUL resolution) then there is - in my opinion, of course - absolutely NO justifiable reason to save that image data in anything OTHER than 16-bit format.

If someone DOES HAVE a reason for doing this - with RAW CCD DATA - then I would like to hear it, in order that I might stand corrected.

And, if a piece of software is then going to use a 16-bit FITS format, then the programmer had better spend some time understanding how to use the BSCALE and BZERO (standard) FITS keywords. (Heck, it took me less than an evening to figure it out for myself, and I had NO IDEA what the FITS system even was at the time)

And, finally, the CBLACK and CWHITE keywords are TOTALLY NON-STANDARD. They could just as easily have been called MICKEY and MOUSE, and they would have done EXACTLY the same job, but ONLY for those few 'kludged' software packages that agreed to use them.

And it is for these reasons that I cannot ever see myself voluntarily using 'kludgeware' such as CCDStack or Maxim - after all, if they are willing to include this kind of 'work around', what other 'horrors' could be lurking in the depths of their code?


i'm really just guessing here about what maxim is doing with CBLACK and CWHITE. i have to assume though that maxim is doing something before using these files as calibration masters... the levels are just way too high. they are actually being written out as f32, but apparently with a range of [0.0-65535.0].


3970
General / Re: calibration of lightbuckets.com images?
« on: 2010 May 19 09:29:33 »
From http://www.cyanogen.com/help/maximdl/FITS_File_Header_Definitions.htm

CBLACK – indicates the black point used when displaying the image (screen stretch).
CWHITE – indicates the white point used when displaying the image (screen stretch).

So these values do not help getting mathematically correct black/white.

I did set the FITS reader mode to ask if values are out of range, but when I get out of the field of upper value after entering 65536, it changes itself to -9223.372036854776500
 ???  ???  ???

-- bitli


well, you saw steve's reply on lb.com. he seems to feel that if the images work with maxim and ccdstack then it's our problem.

i'm starting to think that even though those keywords are for display only that both maxim and ccdstack might be using them to determine the blackpoint in the file. and i'll bet that if you save the stacked bias master as i16 maxim would put the same levels in the file, so that's probably not a solution.

it does not make sense to me for bias frames of a CCD cooled to -30C to have such high values. maxim has this goofy "auto stretch" setting when saving files, and i'll bet that's got something to do with the pixel values in these calibration frames.

i'm still having problems with the flat 'subtraction' when running ImageCalibration (obviously it's division...) i manually divided the flat into the light and found by trial and error that i had to compress the flat histogram into the 0.990 - 1.0 range or the result was wrong. and if i used the modified flat in ImageCalibration, the flat got scaled somehow and the results were the same.




3971
General / Re: calibration of lightbuckets.com images?
« on: 2010 May 18 15:38:18 »
i am talking to myself here, but as far as LB-0003 goes, i think i'd better wait for them to deliver the new calibration masters.

for fun i read in the dark as i described above, and tried to calibrate with it, choosing optimization and bias subtraction. PI found no correlation. so then i manually subtracted the bias frame from the dark, and then scaled the dark with pixel math. i then calibrated a light frame with the bias frame and the manually calibrated dark; here is a 9:1 zoom on part of the image:



as you can see the dark is shifted up about 10 pixels. i guess the apogee firmware update must have done more than just change the size of the file...

3972
General / Re: calibration of lightbuckets.com images?
« on: 2010 May 18 13:32:23 »
that flat gray calibrated image is what happens if you read the bias frame in with the default fits range of [0.0-1.0] or even juan's suggestion for maximDL f32 fits files ([0.0.-65535.0]). if you tell the fits reader to "always ask" for the range, it will tell you the minimum value in the file and the maximum value in the file. in my case the minimum was 1388.800048828125, so i told the reader to use [1388.0-65535.0]. when i subtracted that bias file manually it was reasonably successful - the amp glow in the corners had been subtracted out.

while this works, it's not exactly scientific. i'm not sure if the lowest pixel value in the bias frame is really pure black.

in your example above, the fits header does have this stuff:

CBLACK,1369,Initial display black level in ADUs
CSTRETCH,'Medium  ',Initial display stretch mode
CWHITE,1655,Initial display white level in ADUs

i guess i thought this was just there for maxim to record what it's "screen stretch" values were. but maybe we can take this to mean that black = 1369 in your file. try reading the bias frame with 1369-65535 in the fits reader and see if you can subtract it from the light frame using pixel math.

finally, is it legitimate to apply the same technique to the dark frame?

and regardless of all of that, somehow my flats are being over-subtracted in the ImageCalibration tool. that's why i wondered if they have to be normalized somehow.

3973
General / Re: calibration of lightbuckets.com images?
« on: 2010 May 18 08:15:53 »
thanks jack. i had deleted 2 pixels on the right and bottom from the calibration masters, somewhat at random. i'll delete one all the way around and try again.

can you comment about the levels in the bias and darks? when you bring these into pixinsight, what range values do you use in the FITS reader?

3974
General / Re: calibration of lightbuckets.com images?
« on: 2010 May 17 22:58:07 »
thanks for the reply.

this was on LB-0003. i find that more often than not, lightbuckets' automatic calibration of images fails. and as you say, if your position angle is not 0.0deg, the flat is not used in lightbuckets' calibration anyway. so it's probably better to calibrate on your own.

in this case, the lightbuckets calibration failed, so i got the raw lights. they do not provide the subs for the bias, darks and flats.

i think the biggest problem to solve here is mapping the pixel values in the calibrated dark and bias frames properly. to be honest i don't understand why they are so high to begin with. perhaps maxim understands these files okay for calibration, but they don't seem very portable.

3975
General / calibration of lightbuckets.com images?
« on: 2010 May 17 21:34:48 »
has anyone else successfully calibrated a lightbuckets image with pixinsight?

right now i've got 3 problems. my lights are 4096 by 4096, but the darks and bias frames are 4098x4098. this appears to be some kind of bug and they are regenerating their master dark and bias frames tonight.

the 2nd problem is with the master bias and dark frames... first off, i'm not sure if they have been calibrated. i think they probably have not as the darks seem to exhibit amp glow. secondly, they are saved as f32 format fits files out of maxim. in another thread Juan mentioned that maxim saves f32 as [0.0 - 65535.0], in other words, same as i16 but with floating point numbers. since pixinsight interprets f32 images as [0.0-1.0] by default, you need to tell the fits reader to use [0.0-65535.0] as it's input range. however, i see that the pixel values in the bias frame start at 1388.800048828125 and go up to 2120.828613281250. if i read this fits with range = [0-65535] the bias frame is way too bright with respect to the light. if i tell pixinsight to interpret the input as [1388-65535] then the bias frame is very reasonable and can be subtracted from the light without destroying the light, but i have no way of knowing if 1388 should actually equal 0. the raw light frames are provided as i16 and so i don't think there's any ambiguity about the pixel values in the light.

i have the same problem with the dark frame. i scaled it in a similar fashion but pixinsight gives it a weight of 0 while calibrating. however, this could be because i had to chop off 2 columns and 2 rows to match the lights, and i did it wrong, in which case the dark current pixels in the light and the dark don't actually line up on each other, and so PI finds no correlation.

finally, PI seems to be oversubtracting the flat frame. does the flat need to be normalized in some fashion?

thanks for any advice.

Pages: 1 ... 263 264 [265] 266 267