Calibration Problem: `No correlation' and `binarized-looking' results

[marekc]

Hi Everyone,

I just did a search on the phrase `no correlation', to see if anyone has been having the same problem that I have. This has been a pretty big problem for me during the last few weeks.

I'm trying to calibrate monochrome CCD images that I shot with an Orion Parsec 8300M camera. I shot lights, flats, darks, and biases. I have prepared the master bias, dark, and flat frames according to the standard Pixinsight tutorial:

http://pixinsight.com/tutorials/master-frames/en.html

After making these master frames, I've taken a quick look at each of them with an auto STF, and they look like images I've seen in books about CCD imaging, showing what the master frames `should' look like (e.g. Berry and Burnell, Wodaski).

I am applying these master frames to my light frames, using the Image Calibration module (The light frames also look normal with an auto STF).

However, when I try to calibrate my light frames, two strange things happen:

1) The processing console tells me that there is `no correlation' between my dark frame and my target frame.

2) I get a `calibrated' light frame that looks very strange... when I do an auto STF on it, it is perfectly gray, except for the brighter pixels, which are clipped to white. I call it a `binarized-looking' appearance. There is no useful data in the image. It's hardly even an image any more.

In the Image Calibration module, I've tried various combinations of checking and unchecking the `calibrate' and `optimize' boxes, in all of the places where they appear, without any luck.

I'm wondering if I'm just making a mistake, or if this might possibly be a bug? My darks, flats, biases, and lights were shot with the same camera, at the same temperature, and the same binning. They were saved at the same bit depth. (I get this problem both when I acquire everything at 16-bit and when I acquire everything at 32-bit depth).

There have been a couple of  recent threads that have mentioned this issue:

http://pixinsight.com/forum/index.php?topic=2528.0

http://pixinsight.com/forum/index.php?topic=2460.0


I'm using 1.06.09.0652 eng (x86), running under Mac OS X 10.6 (Snow Leopard) on a MacBook Pro.

Thanks, hopefully this is just some simple mistake that I'm making. I hope to be able to calibrate frames again sometime soon.

- Marek Cichanski

[marekc]

A quick follow-up:

My darks were all shot at a longer exposure time than any of my lights or flats, so that the darks could be scaled to match the lights and flats.

- Marek

[vicent_peris]

Hi Marek,

At the time of calibrating your light frames, the "Calibrate" checkbox of the dark section must be enabled because the bias frame must be subtracted from the dark frame.

The problem you report is the typical case applying the dark frame without subtracting the bias. This causes a double subtraction of the bias, so the result is a black (or grey) image where you have low signal.


Regards,
Vicent.

[marekc]

Hi Vincent,

Thank you very much for your reply. Unfortunately, I'm afraid that your suggestion didn't work in my case. After I read your message, I ran Image Calibration on my light frames, both with and without the `calibrate' box checked (in the Master Dark section). Either way, I get the same `binarized-looking' result.

I wonder... is it possible that somehow PI isn't getting my `calibrate' command? In other words, maybe when I check the box, the Image Calibration module doesn't register that I've checked it?

-Marek Cichanski

[vicent_peris]

Yes, it could be... could you upload your master calibration frames and a light frame so that I can check myself?

Regards,
Vicent.

[jeffweiss9]

Vicent-
  This sounds very similar to the problem I reported at

http://pixinsight.com/forum/index.php?topic=2846.0

  Per your request, I sent you a few days ago 2 calibrated lights and the master dark and had similar grey calibrated subs apparently caused by a large range of negative values using up the available range.

   I did not have calibrate checked in the darks section when calibrating my lights, so I reprocessed the Ha and Blue subs that gave the gray calibrated frames as Marek described.  That did fix that problem of the gray calibrated lights in that they no longer had the large negative range that I think was the cause of that.  However, when I register and stack the new calibrated lights, I still obtain an integrated result that appears to have its information compressed into a narrow range and is, consequently, clearly inferior to the registered stacks that I got by independently calibrating entirely in AIP4WIN.  So something is still incorrect in my translation of your tutorial.

  I attach here (psm file) an updated version of the process icons that embodies my interpretation and wonder if you could quickly review them because clearly something is not correct to get the poor results that I am getting.  I remind you (from the other post) that the calibration appeared to work correctly for L, R and G subs for the same data collection and the same PI calibration procedures and those were used, along with the AIP4WIN calibrated frames for Ha and B to create the following image (all lights, darks, flats were at -20 deg C, camera regulated):

http://www.astrophotogallery.org/member-galleries/p7052-ngc-2264-cone-nebula.html

  Thanks very much for your help. I didn't intend to hijack Marek's post but this does appear to be a closely related, if not the same, problem and I'm still trying to figure out the solution.
-Jeff

[marekc]

Hi Vincent,

Thanks very much for your help with this issue. I've uploaded a light frame, as well as master dark, flat, and bias files to my Dropbox public folder. I hope that these links work correctly:

http://dl.dropbox.com/u/11055667/light_frame.fit

http://dl.dropbox.com/u/11055667/master_bias.fit

http://dl.dropbox.com/u/11055667/master_dark.fit

http://dl.dropbox.com/u/11055667/master_flat.fit

If anyone else wants to try calibrating these frames, that's fine with me, too.

Thanks,

- Marek Cichanski

[vicent_peris]

Hi, Marek and Jeff,

I found the solution in Marek's data. The problem is the bias frame: it is 500 times higher than it should be. This is bizarre IMO... it can be an acquisition problem. Which software do you use to acquire your data? Which numeric format do you use to save your calibration frames? They must be 16-bit unsigned integer. Jeff: could you check this also?


Regards,
Vicent.

[marekc]

Hi Vincent,

Thank you very much for figuring this out!

Looking at my raw calibration frames in PI, here's what I see when I look at a bias frame: The `BITPIX' item in the FITS header says that the number of bits per pixel is -32. I assume this means that this raw bias frame is in 32-bit floating-point format.

Here's why it's that way: Basically, I saw an option for acquiring my frames in 32-bit float format, and I assumed that `more is better'. I'm using an Orion Parsec 8300M monochrome CCD camera, and I'm capturing my data using the camera's ASCOM driver and Craig Stark's `Nebulosity' software. In that program, there's an option to capture data in 32-bit float format, instead of the default 16-bit integer format. I figured `hey, sounds better!'... mostly because I don't know much about the file formats. `More' sounded like `better'.

It sounds like I should restrict my captures to the default 16-bit integer format, and all will probably be well.

I wonder if I could convert my 32-bit float frames to 16-bit integer format? I'll look around in PI (and the other software I have) and see if I can batch-convert like this. If so, I'll try converting the frames and then try again to calibrate. In the future, I'll just shoot at the default 16-bit depth.

Oh, I have a question about some settings in the `master calibration frames' tutorial, but I'll make that a new thread...

Thanks again!

- Marek Cichanski

[jeffweiss9]

Hi, Vicent-
 My bias frames were 16 bit unsigned (collected by CCDSoft v5) but your first suggestion about clicking the "calibrate" checkbox of the enabled dark section looks like that was at least part of my problem.  A more careful study of the Ha, B stacks I was now able to make with PI against those I made previously for those filters with AIP4WIN (using dark-flats) suggests they are very close.  When I remade the L stack this way and compared it with a reference AIP4WIN-calibrated L stack, (being careful to disable rescaling in PixelMath to compare them), they also seem to be very close.  However, since the "calibrate lights" section of the tutorial is just one brief paragraph at the end of the "Master Flat (II)" section which didn't mention checking calibrate in the dark section and there is no figure showing the actual settings in the ImageCalibration tool, I don't have a lot of confidence that I'm still doing that step of the PI calibration correctly.  As shown in the psm file I attached in my previous message, I have

Master Bias section checked; calibrate UNchecked; not mentioned in the tutorial
Master Dark section checked; calibrate CHECKED, optimize UNchecked; not mentioned in the tutorial
Master Flat section checked; calibrate UNchecked, as mentioned in the tutorial

Are those settings correct?  If I'm understanding the logic, they say you are going calibrate the MasterDark by subtracting the MasterBias and then use both the bias, the bias-subtracted MasterDark and the MasterFlat to calibrate the light frames (but not change the already-calibrated MasterFlat).  I'm most uncertain about whether the optimize box should be unchecked or checked.
Thanks very much.
-Jeff

[marekc]

Hi to Vincent and anyone else who's knowledgeable about correlation,

I'll second Jeff's request for some more guidance on how to use the Image Calibration tool. I have found the tutorial on `Master Calibration Frames: Acquisition and Processing' to be extremely helpful, but I could use more guidance on how to calibrate the light frames. As Jeff said, an image showing which items should be checked and which should not be checked would be great.

Also, in that tutorial, I'm wondering about something in Figure 1: Should `clip low range' and `clip high range' be checked? I was trying to combine some darks recently, and I found that my hot pixels were absent from part of the frame. What was strange about this was that the cut-off between the `with hots' and `without hots' was a straight horizontal line. Odd. When I unchecked `clip low range' and `clip high range' at the Figure 1 stage, the problem went away.

At any rate, I'm very grateful for the `Master...Calibration...' tutorial. If it were possible to extend it a bit to cover the calibration of light frames, or perhaps to write another tutorial about that process, I think it would be very helpful.

Thanks,

- Marek Cichanski

[marekc]

Many thanks again to Vincent for looking into the calibration issues that I've been having.

Now I've found something really weird in my camera... as Vincent noted, it has a giant bias value!

I just checked many of the bias frames I've shot during the time I've had the camera, and they all have pixel values around 3000 ADU (on a 16-bit unsigned integer scale, i.e. from 0 to approx. 65,000). Strange! That camera uses up about 5% of its full-well depth on the BIAS value. That is really weird.

A friend of mine just got a QSI monochrome camera, also based on the Kodak 8300 chip. His master bias has pixel values around 200 (on a 16-bit unsigned integer scale). Clearly, my camera (a `Parsec 8300M', from Orion Telescopes) has a really strange bias value.

I'll check with Orion and see if they think there's anything wrong with it. If not, then... that is a really strange way to build a CCD camera, it seems to me. Using up 5% of the dynamic range on the bias value, when a similar camera from a different company (but with the same chip) has a bias value 1/15th as big... not so good.

The big problem this presents for me is that I can't calibrate with Pixinsight. I really like the way PI allows me to just shoot a bunch of long-duration darks, which will cover any lights and flats that I shoot, as long as the lights and flats are the same temp and shorter exposure times thank my master dark. That's a really nice PI feature. But... with this crazy-big bias value, which is essentially the same as almost any dark frame I'm likely to shoot... that's messed-up. It seems to me that if I'm going to calibrate data from this camera, I have to shoot darks and `flatdarks' whose exposure times match my lights and flats. That's not impossible, of course - I can build a library of darks, but it's rather sub-optimal. Seems unnecessarily awkward. I'm feeling rather disappointed.

Thanks, Vincent, for helping me uncover this strange behavior. This is something for me to investigate further. Grr, argh...

- Marek Cichanski

[vicent_peris]

Hi Marek,

3000 counts for a bias frames is high, but not so high... I have a FLI Proline 16803 camera and it has about 2100 ADU bias level. On the other hand, the CCD camera of the 1.23 meter scope at Calar Alto has a bias level of about 80 ADUs.

I think your problem could be from the acquisition software... your bias level is so high only in your bias frames, but not in all the other frames. Give it a try: Acquire a normal dark frame, but only of 0.1 sec exposure time; this is much like a bias. Save it as a dark and then use it as a bias in your calibration workflow in PI. First, open this "biark/bias" in PI and see the its median level.


Regards,
Vicent.

[Juan Conejero]

... your bias level is so high only in your bias frames, but not in all the other frames.

I think this is the key of the problem —and it seems to point directly to the acquisition software as the origin of the problem. Please upload a single raw frame for each calibration category, just as they are generated by your acquisition software: one bias, one dark, one flat and one light. Only then we'll be able to know why is this happening exactly.

[Ioannis Ioannou]

I had similar problems with bias from MaximDL.
Apparently some software (drivers maybe? ) transparently add a high offset to bias (I guess in order to keep it in the linear area of the sensor ? )
Maxim seemed to know how to handle it, but with Pix I had to use a bit of pixelmath on them, otherwise I was getting gray images.