ImageCalibration gives highly quantised results

[rfrancis]

I have encountered an annoying oddity in the ImageCalibration function. I have used this function many times in the past but now it's getting weird!

Previously I had encountered similar effects if the images (say) were 16 bit integer and unmatched to the calibration frames. Now all files are 32-bit floating point.

I attach an screen-shot of the effect (sorry for the poor quality to met the size restrictions). All frames have the screen transfer function auto-stretch. You can see the original frame from the camera at top left. The master bias at bottom left, master dark at top right and the master flat at bottom right. On the right is the control window with its parameters (but I have tried all combinations. The only one which is benign is applying the flat only).

In the middle is the result. It's the same with everything I've tried all evening.

The master cal frames were all made exactly as in the tutorial (I tried that in comparison to my more sloppy ones earlier, but it did not change the situation).

All I can get is 3 or 4 distinct pixel values, all close to 0.0.

What am I doing wrong???

thanks,
Richard

PS: MacOS 10.8.2, PixInsight Version 01.07.06.0793 Starbuck (eng x86_64). No updates available.
PPS: I checked it on two different Macs, both with the same OS version.

[lucchett]

Hi richard,
It happened to me choosing both bias and dark calibration, but skipping the dark calibration ation flag.
Problem solved either avoiding bias or flagging dark for this.
Both ways gave similar results with my temp matched dark.
Hope it helps,
Andrea

[rfrancis]

I have some more information.

First, though, I now attach the missing ImageCalibration control window which was missing from the previous post.

As you'll see, I did select the action flags (thanks Andrea!). The only one of the calibration files which does not produce this effect is flat calibration. Both bias and dark calibration, in any combination, do it.

The new information is that the lights are very low signal (it was the fist time out with a new camera lens combination -- 250 mm f5.6 mm lens with a QSI 583wsg). The signal levels are really low, with an average of 2*10^-3. In 16 nit scaling, the peak values are only around 600 vs a background level of 250-300.

I think this is the problem.

The next problem is how to increase the levels -- this was already with 10 minute exposures. I don't fancy exposures of more than 30 minutes and that won't improve things so much.

The filters were 3nm Astrodon, which certainly helped with background signal but would not have diminished the nebula signal.

I think I need to do more experiments.

cheers,
Richard

[lucchett]

Richard,
I think you should be able to calibrate your light frame in any case.
Just to confirm, have you tried to only load the dark, check the part, and uncheck the two flags( dark bias calibration and optimize)?

[rfrancis]

Yes, I have just tried that (ie uncheck bias and flat, leave dark checked, and uncheck the 2 options).

Same result.

I have checked that using PixelMath, subtracting the bias from the light gives the expected (normal) results when I check the box for rescaling the result between 0.0 and 1.0.

But,using ImageCalibration with only the (same) bias frame, and the Calibrate option unchecked, gives the strange quantization appearance.

Anyway, this discussion is useful as it's triggering me to try some further investigations. And I think I've found another clue to the problem.

Most of the pixels are value 0.0 and this is upsetting the auto-stretch function. If I use another approach, like histogram modification or Digital Development, I can see grey levels. BUT, due to the calibration step, only the brightest parts of the image remain, and most of the signal has been replaced by zeros.

I will remake a series of bias frames, and then, again, go through the process of generating master calibration frames.

Hopefully there was an error in my process which is exaggerated by the low signal in my lights.

thanks,
Richard

[pfile]

what is the source of the original subframes you are using? meaning, what program has been used for camera control? it could be that the issue is incompatibilities with the pixel representations. some programs use floating point numbers to store integers. PI expects any f32 fits file to have pixel values in [0.0..1.0]. in the FITS reader you can adjust how PI scales the values it finds in the file to the range [0.0..1.0]

i think the best thing to do is just use i16 to save the subframes coming out of the camera.

[rfrancis]

I'm using Nebulosity on a Mac. I have successfully used it before, and changed from i16 to float-32 some time ago to avoid similar issues I was having then.

Anyway, I'll have a closer look.

thanks,
Richard

[georg.viehoever]

The "strange quantization appearance" may also be an artefact of a very extreme STF. Try using Histogram Transform to do a stretch-it does not have this issue.
Georg

[rfrancis]

Two years on and I still have this problem.

I'm now using TheSkyX as well as Nebulosity for image acquisition. Both are providing 16 bit images.

Uncalibrated images give reasonable-looking resuts using STF. So do all my calibration frames (32-bit floats).

If I calibrate for the Flat only I get very similar images to the raw ones. This is true if I select Calibrate or not.

If I choose Master Bias and/or Master Dark I get un unusable result which is quantised in very few values. This is true for all combinations of flags.

A problem this basic should have generated many reports, so clearly I'm doing something wrong.

Any ideas anybody?

thanks,
Richard

[pfile]

sometimes, and i don't know why this happens, a bias frame can have a bias signal that's way bigger than the bias signal that's in your dark or light frames. i think some cameras/drivers might be rescaling the bias frames based on some optical black areas, but i am not really sure about that.

the telltale sign is that after you've calibrated your darks with a master bias, you have a whole bunch of "0" pixel values in the calibrated dark frames. this should never happen, as the dark signal should always exceed the bias signal. but when it does happen, there are negative values in the calibrated dark and they are all clipped to 0.

the solution is to add an output pedestal to the dark subs during calibration. this pedestal value should be set high enough that you avoid any 0 values.

i don't know how to do this with BPP, but if you are manually calibrating your images you can find the output pedestal in the "output files" section of the ImageCalibration tool.

this adds a new field to the fits header of the calibrated frames that remembers the pedestal value you put in. when you integrate a master dark from those dark subs, the pedestal value is copied to the integrated master dark. then when you calibrate a light with that master dark, the last step is to subtract the pedestal. the idea is that the values in your light frame should far exceed this pedestal, so you don't end up with clipped pixels in the calibrated lights. for narrowband images this might not be true, so beware of that.

rob

[jkmorse]

You may be running into some FITS incompatibility issues.  I did a lot of research recently on FITS incompatibilities and this issue has been addressed time and again by PI.  Here is the results of my research into old posts on the subject:

Fixing FITS Incompatibilities
If you experience these types of incompatibilities, try the following until you hit on a solution:

First
•   Open the Format Explorer (access under “View > Explorer Windows”)
•   Double click on “FITS”
•   Make sure the top two boxes are checked
       o   “Write scaling keywords . . .”
       o   “Signed Integer images . . .”

Second
•   Where possible, particularly in the Image Calibration tool for Lights, but also with ImageIntegration when creating Master Darks and Master Bias, add the following to the “input hints” line under “Format Hints”:
       o   lower-range 0 upper-range 65535

Third
•   If the above does not work, insert the following in the “input hints” line:
       o   Signed-is-physical

Stop using input hints AFTER you have successfully saved an image after applying a PI tool; i.e, do not use input hints in ImageIntegration after successfully calibrating Light images in PI with the ImageCalibration tool (after first PI application, images are saved in proper PI format (0,1) and no longer need input hints)

For an excellent explanation of what is happening to cause this problem, check out the following threads from Juan Conejero:

http://pixinsight.com/forum/index.php?topic=3118.msg21348
http://pixinsight.com/forum/index.php?topic=3474,msg23973
http://pixinsight.com/forum/index.php?topic=3800.msg26086

Hope that helps,

Jim

[jkmorse]

Richard,

One of the keys to making what I said in my previous post work is that you need to go back and remake your master dark and bias frames using these hints.  I have had particular success with the "lower-range 0 upper-range 65535" input hint when making master darks and bias frames in ImageIntegration.

One other thing I run into is that I sometimes have to convert my raw lights to 32 bit floating point images, then use the "lower-range 0 upper-range 65535" input hint in ImageCalibration to get things to work out right.

Hope that helps,

Jim

[rfrancis]

Thanks Jim, I'm working on regenerating those masters now. Hopefully this will solve it. I'm off on a trip tomorow morning though so I may have nothing to report until next week.

thanks,
Richard

[rfrancis]

OK, that solved the problem for bias and darks. But then flats had an issue: zero or insignifiacnt scaling factor for flat frame channel 0 (empty master flat frame image?).

It's not empty. But then using your second hint, Signed-is-physical, solved that one too!

Great, and thanks for getting me this far!

cheers,
Richard

[lucchett]

Guys,
after reading the conversation, I now have a few doubts.

I went through all the discussion but I am now very confused...

In my case:
acquisition: Skyx
calibration files: in Maxim dl (because Skyx doesn't allow delays in series for flats, darks, bias)

Calibration: manually carried out in PI
Integration: manually carried out in PI

The lights seems to calibrate well, but how can I be sure everything is properly handled?
what I need to check in the fits header of lights and cal frames?

many thanks!
Andrea