How to use overscan correctly?

[VoidPointer]

Hi,

I've started to use a QHY168C  cooled color CMOS camera. The images from the camera have an overscan area at the top and at the right hand side and I am unsure how to correctly configure that in PI (and whether it even makes sense to do anything with the overscan data or if it should just be cropped)
The FIT file 5040 x 3346 pixels
The effective sensor area is specified as 4952 x 3288.
So there are 58 overscan pixels at the top and 88 overscan pixels at the right-hand side.
The sensor is a Sony IMX071 and I found some information on the overscan areas at https://landingfield.wordpress.com/2014/06/07/imx071-characteristics-2/
The areas called "virtual dummy" and "horizontal blanking region" don't seem to be included in the FIT file from the camera.
So remaining are the "dummy pixels" (pixels without a photodiode) and the "optical black" pixels (pixels that don't receive any light).

I understand that if I only specify the "Image region" in the overscan parameters (0, 58, 4952, 3288), that will just lead to cropping to the effective pixels.
Does it make sense to set up one or two overscan areas with the "dummy pixel" or "optical black" pixels as the source region and the image region as target region?
If I do that, does it change anything about how I would be using bias and darks?
Currently, I use a super-bias and a master-dark that include the overscan area. When I'm calibrating manually, I use dark frame optimazation for calibrating flats and lights with the same master-dark (and I understand that BatchPreprocessing is doing the same)

Thanks for your help...

Lars

[Niall Saunders]

Hi Lars,

Here is my understanding on OverScan pixels:
1.) They do not receive any photons
2.) Apart from (1), they are 'real' photosensors
3.) The ADU value that can be expected of these pixels is very much the same as that of a Bias frame pixel
4.) As with Bias pixels, the ADU is based on the Offset/Gain settings - modified only by the random effect of Readout Noise
5.) Every Light and Dark frame has (statistically) the 'same' level of Bias and OverScan so, if a (Master)Dark is subtracted from a Light, then the PreCalLights that result contain (statistically) only the ADU values that were created by incoming photons
6.) The same argument applies to Flats and FlatDarks - Bias/OverScan and DarkCurrent are (statistically) eliminated from all CalFlats. This means that PreCalLights can then be divided by an average-integrated MasterCalFlat to generate a final collection of CalLights, ready for the final stages of the calibration process.

So, IMHO, just as the requirement for collecting Bias frames has a justifiable argument against it, there is also an arguable case against ever needing to look at the OverScan area.

In all the years that I have been acquiring astro-images, I have never bothered with OverScan areas or Bias frames - other than purely out of curiosity, to see if there have been any changes on the ADU values of these image areas that might suggest some sirt of 'ageing' is taking place (evidence for which I have never yet seen).

So, I hope my thoughts and experiences might help you understand what you feel you want to do as far as image acquisition and pre-processing is concerned.

[RickS]

Overscan calibration is only useful if your camera exhibits bias drift.  If the camera bias is stable then you'll suffer extra pain for no benefit.  The dummy pixels are the ones of interest since they provide an estimate of the actual bias in each frame uncontaminated by dark current or detected photons.

Cheers,
Rick.

[VoidPointer]

Overscan calibration is only useful if your camera exhibits bias drift.  If the camera bias is stable then you'll suffer extra pain for no benefit.  The dummy pixels are the ones of interest since they provide an estimate of the actual bias in each frame uncontaminated by dark current or detected photons.

Thanks Rick, that seems very helpful...

I guess I could look at the dummy pixels in the overscan area of a series of light frames and determine whether there is some kind of deviation over time. If that's not the case then there's no further need to bother with the overscan calibration. Right?

If there is a drift and I wanted to set up the dummy pixels as a source for overscan calibration, would I use that instead of my master (super) bias or in addition to that?

Cheers!
Lars

[RickS]

Thanks Rick, that seems very helpful...

I guess I could look at the dummy pixels in the overscan area of a series of light frames and determine whether there is some kind of deviation over time. If that's not the case then there's no further need to bother with the overscan calibration. Right?

If there is a drift and I wanted to set up the dummy pixels as a source for overscan calibration, would I use that instead of my master (super) bias or in addition to that?

Cheers!
Lars

Hi Lars,

I do overscan calibration on my Apogee U16M camera because I noticed that the bias values can change significantly over time.  I suspect it is caused by temperature sensitivity of the external camera circuitry.  I have a couple of other cameras that are more stable so I don't bother for them.  Just take a bunch of bias frames over a couple of days and see if the average value changes much.

If you decide to do overscan calibration then you need to take all your calibration and light frames with overscan enabled.  The overscan bias is used to adjust the calibration frames when you build the master bias, master dark and master flats as well as the lights.

Cheers,
Rick.

[VoidPointer]

If you decide to do overscan calibration then you need to take all your calibration and light frames with overscan enabled.  The overscan bias is used to adjust the calibration frames when you build the master bias, master dark and master flats as well as the lights.

Got it. Thanks!

[VoidPointer]

Well, this almost all works now... I got some more specifics from QHY regarding the overscan regions (part of the dummy pixel region) to use. That's good.

What's a bit strange: When I use batch pre-processing with the overscan set up and also use a cosmetic correction template that is based on my master_dark (which includes the overscan), the cosmetic-correction step will fail because at the time where it is applied, the calibrated light frames no longer have the overscan and so the master_dark's geometry no longer matches.
I worked around this by creating a cropped version of the master_dark.
However, I feel like the correct solution would be for the cosmetic-correction process to be able to specify an overscan region on the master_dark too so that it can be used with a master-dark including overscan on lights that had their overscan removed during calibration...

[1DegreeN]

Hi Lars,

Did you notice that there is an option on the ASCOM driver settings to remove the overscan area? Sounds like you are making life awfully complicated for yourself by leaving it in. As many users of these new cooled CMOS cameras recommend, I don't use bias frames to calibrate but only temperature and duration matched master darks and flats which have been calibrated with matched darks (i.e. short exposure to match duration of the flats and at same temp that the flats were shot at).

[VoidPointer]

Well, I guess you are right and I probably went a bit beyond the point of diminishing returns with this :-)
Thanks for the reality check :-)