Deconvolution & Bias Frames

[dpaul]

I've been doing basic image processing whilst gradually learning PI. I've always used master dark frames but only just started taking bias frames.
I've experienced something unexpected and want to know why?

Without calibrating the pre-integrated light frames with bias frames - I can use deconvolution on the linear files and it works (to varying degrees) with deranging and no-deranging, although I usually check the box.

Now when calibrating the light frames with bias frames and then applying deconvolution, I get a weird mosaic effect across the whole image (so its unusable). It only happens when I choose deringing and without it, all is fine except for the dark rings (which I don't want).

Can anyone suggest what's going on?

I have one final question.
I took 30 bias frames then integrated them with Image inetegrator and fails to work when I use the defaults Iterative K-sigma setting for the scale estimator
I got it to work only with Percentage bend variance and average absolute deviation from the median.
Is this expected (Its a CMOS chip not CCD).

Thanks

David

[pfile]

check the mean of the background in your lights vs. the mean of the bias frames. the calibrated frames probably have a bunch of pixels clamped at 0 (because the bias value is exceeding the light frame value). you can also use the pixelmath expression iif($T==0,1,0) on your calibrated frame to see which/how many pixels are 0.

rob

[dpaul]

Hi Rob,

I don't know how to check the mean of a frames?

I tried the Pixelmath expression on one of the individual bias frames (before integrating lots of them together) and got the following:

[b]PixelMath: Processing view: _3
Writing swap files...
361.992 MiB/s
Executing PixelMath expression: combined RGB/K channels: iif($T==0,1,0): done
Truncating to [0,1]: done
139.786 ms

Then I tried on one of the light frames (before any calibration) and got the following:

PixelMath: Processing view: M1_6X15_SECS_GAIN_7_5_c
Writing swap files...
1112.305 MiB/s
Executing PixelMath expression: combined RGB/K channels: iif($T==0,1,0): done
Truncating to [0,1]: done
137.501 ms

No idea what these mean?

My actual problem is that I can't integrate the individual bias frames together into a single master, this is the error I get:

C:/Users/Dave/Documents/MALLINCAMSKY/NEW BIAS FRAMES/1.fit
* Retrieved data from file cache.
*** Error: C:/Users/Dave/Documents/MALLINCAMSKY/NEW BIAS FRAMES/1.fit: Zero or insignificant signal detected (empty image?)
<* failed *>

I tried redoing the integration with the scale estimator set to Percentage bend variance and average absolute deviation from the median (instead of the itertaive-K sigma default) and it worked - the end result after integrating 40 bias frames was as follows:

MRS noise evaluation: done
Computing noise scaling factors: done

Gaussian noise estimates  : 3.2578e-04 3.7177e-04 2.9596e-04
Scale estimates           : 5.6437e-04 1.3869e-03 6.3475e-04
Location estimates        : 3.1943e-02 2.9755e-02 3.9211e-02
SNR estimates             : 9.6253e+03 6.2007e+03 1.7424e+04
Reference noise reduction : inf inf inf
Median noise reduction    : inf inf inf

03:14.87


It all seems to work, is this ok?


Thanks

David

[dpaul]

Sorry Rob,

I think you were referring to my first question not the second:

I ran Pixelmath (with suggested expression) and got this on one of the calibrated light frames:

PixelMath: Processing view: M51_7X8_SECS_GAIN_13_c
Writing swap files...
1056.625 MiB/s
Executing PixelMath expression: combined RGB/K channels: iif($T==0,1,0): done
Truncating to [0,1]: done
146.103 ms


On the integrated master bias file, Pixelmath gave me this:

PixelMath: Processing view: Master_Bias_40_frames_5_deg_C_
Writing swap files...
1107.919 MiB/s
Executing PixelMath expression: combined RGB/K channels: iif($T==0,1,0): done
Truncating to [0,1]: done
134.328 ms

Does thsi help explain?

David

[dpaul]

Rob,

the deconvolution problem didn't seem to repeat when de-ringing on the M51 integrated image (the previous was M1 where it did) - I'm sure I was doing everything the same?

David

[pfile]

i was not suggesting that the pixelmath is a fix for your problem, rather, it is a diagnostic. the console output is not too relevant, what is relevant is if you get a frame speckled with loads of white pixels.

to find the mean of a frame use the Statistics process.

the use of a single bias frame could be your problem. also, when integrating bias frames, you don't want any weighting at all. you can read this tutorial to understand how the different processes should be configured for each type of integration (master bias, master dark, master flat, master light...)

http://www.lightvortexastronomy.com/tutorial-pre-processing-calibrating-and-stacking-images-in-pixinsight.html

rob

[dpaul]

Thanks Rob,

I'll read the tutorial as suggested -

Regarding the bias, I integrated 40 separate bias frames into one Master. Isn't that what one is supposed to do? For example during image calibration, when you check the box for the master file, can it pick up one or multiple?  It lets me click on multiple but I can only see one?

Another issue is compatibility of the dark frames, I'm capturing them in a similar fashion to bias but much longer?

Thanks

David

[dpaul]

Hi Rob,

Great Tutorial,

I've learnt a lot about calibration - the bias results are good although the integration of bias files to get a master bias still only works when the scale factor is set at Percentage bend variance, but results seem good - the final super bias was virtually noise free.

I tried similar approach to the darks and achieved what looked like a good master dark (after calibrating each dark with the superbias). But when I tried to calibrate the light frames, it works for the bias but the dark is rejected, saying ''no correlation between master dark and target frames (channel 0) - no idea why I can't apply darks?

Regards

David

[pfile]

well this is where the statistics process comes in... if the values in the superbias exceed the values in the dark, then you can get pixels clamped at 0. it is no doubt an unusual situation for that to happen, unless you are looking at one of these sony sensors that have very low dark current. anyway the stats process should be able to tell you something - check the mean in the integrated bias vs the mean in the darks...

no correlation essentially means that the ImageCalibration process can not find a scaling factor which minimizes the noise in the calibrated frame. it usually means something is wrong with the dark (for instance, clipped pixels).

what kind of sensor are we talking about? a DSLR or one of these newfangled CMOS OSCs?

rob

[bulrichl]

David,

if that what Rob described in his last post is the reason for the failing calibration, consider the following:

As I wrote before, https://pixinsight.com/forum/index.php?topic=11806.0 (reply #4 and #6), a precalibration of the Master Dark may result in severe clipping (in the MasterDark). I observed this with images from a Canon DSLR - maybe other cameras behave similarly.

You can easily check whether there is clipping in your current pre-calibrated MasterDark: inspect it with HistogramTransformation. A peak in the histogram that is severely cut-off at the left shows that your MasterDark is corrupt. I am curious what your current pre-calibrated MasterDark looks like, perhaps you can show it here?

Please try this approach in order to avoid clipping in the MasterDark:
1) Don't precalibrate the MasterDark with the MasterBias (or SuperBias),
2) set the options 'Calibrate' and 'Optimize' (or 'Calibrate' only) in the MasterDark section when calibrating the light frames.

Bernd

[dpaul]

Rob,

The three mean values (using statistics) for the Superbias master is lower than the Master Dark.
Please see attached the histogram of the Master Dark. This was created by integrating 10 separate Dark frames into a single Master. It would appear there is indeed a sudden clip on the left. The sensor is indeed a CMOS (Panasonic v Maicovicon).

I seem to be ok now with the BIAS and I tried FLATS today - its just this incompatibility between the dark master and the light frames I'm trying to calibrate.

Any thoughts on what could be done to get darks that work?

Regards David

(Bernd - this reply also addresses your question about the clipped histogram to the left. I also get the problem still regardless of whether I use the bias on the darks or not).

[bulrichl]

Hi David,

obviously there are some important informations missing. It would have been helpful if you stated at the beginning which type of camera (monochrome or One Shot Color (OSC) camera) you are using.

The first point that catches my eye is: your MasterDark is a RGB image, and it shouldn't. You want to perform the entire calibration process with monochrome images. You didn't tell us, but I suppose that you use a OSC camera. In this case the calibration has to be performed with monochrome Color Filter Array (CFA) images. Only after completed calibration the calibrated light frames are debayered (to yield RGB images), then registered and finally integrated.

I guess the bias, dark or light frames are stored as monochrome FITS files by the acquisition software you are using. So how are you getting RGB images at the calibration stage? Answering this question will enable us to help you further. There seems to be a fundamental mistake in your workflow.

Bernd

[dpaul]

Hi Bernd,

It is a OSC and I'm aware that non-monochrome captured data should appear monochrome until debayered. Currently all data which I thought should be 'Raw' is coming out as RGB including the bias frames so I need to look into that. I had thought it was 'desirable' to have pre-debayered data during calibration but I'll now look into this.  Strangely the bias frames integrated into master bias and superbias 'do' seem to make a big different to the final integrated image even though they were RGB.

I'll now look into this.

Thanks

David