Problem with Overscan Inconsistent Geometry

[smccully]

I upgraded the Drivers for my QHYCCD Camera this week, and the default was to leave overscan on which i had previously turned off. When doing image calibration i setup Overscan and it appears to be working. But I am seeing a few images with this error "Inconsistent overscan geometry". Not sure this is something with PixInsight, the Camera?

It also seems to be the first image being taken, so i suspect this may be something to do with Camera Drivers?

https://www.dropbox.com/s/tclnez12rs6l9d1/M84-0092-B.fit?dl=0

Code: [Select]

open "file:///home/smccully/workspace/Processing/M84/SixtySeconds/Calibrated/Additional/M84-0088-B_c.xisf"
Reading 1 file(s):
/home/smccully/workspace/Processing/M84/SixtySeconds/Calibrated/Additional/M84-0088-B_c.xisf
Loading image: w=9576 h=6388 n=1 Gray Float32
73 FITS keyword(s) extracted.

ImageCalibration: Global context

Loading master calibration frames:
Loading calibration frame image:
/home/smccully/workspace/Dropbox/Calibration/QHY600/Masters/15c/SuperBias.xisf
Loading image: w=9576 h=6388 n=1 Gray Float32
1246 FITS keyword(s) extracted.
Loading calibration frame image:
/home/smccully/workspace/Dropbox/Calibration/QHY600/Masters/10c/Dark60.xisf
Loading image: w=9576 h=6388 n=1 Gray Float32
16 image properties
258 FITS keyword(s) extracted.
Loading calibration frame image:
/home/smccully/workspace/Dropbox/Calibration/QHY600/Masters/Rasa11/MasterFlat-Blue60.xisf
Loading image: w=9576 h=6388 n=1 Gray Float32
16 image properties
325 FITS keyword(s) extracted.

Dark frame optimization thresholds:
Td0 = 0.00004021 (30585749 px = 50.000%)
Computing master flat scaling factors ...
s0 = 0.650183

Calibration of 4 target frames:
* Using 4 worker threads

Calibrating target frame 1 of 4
Loading target frame:
/home/smccully/workspace/Dropbox/MaximDL/M84-R/2020-02-13/M84-0092-B.fit
Reading FITS image: 16-bit integers, 1 channel(s), 9600x6422 pixels: done
*** Error: Inconsistent overscan geometry

* Applying error policy: Continue on error.

Calibrating target frame 2 of 4
Loading target frame:
/home/smccully/workspace/Dropbox/MaximDL/M84-R/2020-02-13/M84-0093-B.fit
Reading FITS image: 16-bit integers, 1 channel(s), 9600x6422 pixels: done
47 FITS keywords extracted.

Calibrating target frame 3 of 4
Loading target frame:
/home/smccully/workspace/Dropbox/MaximDL/M84-R/2020-02-13/M84-0094-B.fit
Reading FITS image: 16-bit integers, 1 channel(s), 9600x6422 pixels: done
47 FITS keywords extracted.

Calibrating target frame 4 of 4
Loading target frame:
/home/smccully/workspace/Dropbox/MaximDL/M84-R/2020-02-13/M84-0095-B.fit
Reading FITS image: 16-bit integers, 1 channel(s), 9600x6422 pixels: done
47 FITS keywords extracted.

Writing output file: /home/smccully/workspace/Processing/M84/SixtySeconds/Calibrated/Additional/M84-0093-B_c.xisf
Dark scaling factors:
k0 = 0.596
Gaussian noise estimates:
s0 = 2.457e-03, n0 = 0.969 (MRS)
Writing image: w=9576 h=6388 n=1 Gray Float32
73 FITS keyword(s) embedded.

* Waiting for running tasks to terminate ...

Writing output file: /home/smccully/workspace/Processing/M84/SixtySeconds/Calibrated/Additional/M84-0094-B_c.xisf
Dark scaling factors:
k0 = 0.597
Gaussian noise estimates:
s0 = 2.670e-03, n0 = 0.969 (MRS)
Writing image: w=9576 h=6388 n=1 Gray Float32
73 FITS keyword(s) embedded.

Writing output file: /home/smccully/workspace/Processing/M84/SixtySeconds/Calibrated/Additional/M84-0095-B_c.xisf
Dark scaling factors:
k0 = 0.592
Gaussian noise estimates:
s0 = 2.928e-03, n0 = 0.968 (MRS)
Writing image: w=9576 h=6388 n=1 Gray Float32
73 FITS keyword(s) embedded.

===== ImageCalibration: 3 succeeded, 1 failed, 0 skipped =====
17.366 s
[code]

[smccully]

I am seeing some inconsistencies with the overscan features throwing errors on images "Incompatible Geometry" when reloading ImageCalibration from recent history. After resetting settings and rerunning errors went away.

[bulrichl]

I am unclear whether your issues are solved or not.

I took a look at your uploaded light frame. This image has a width of 9600 and a height of 6422 pixels. In STF, I set the parameters to Shadows Clipping = 0.008 and Midtones Balance = 0.010. Then one realizes black borders at the left side (width 24 pixels) and at the lower side (height 34 pixels). In Image calibration, the rectangles are defined by the coordinates x, y of the upper left point, and width and height of the rectangle.

So image and target region are:
x = 24, y = 0, w = 9576, h = 6388

I tried the overscan calibration with two different Source region settings:
1. x = 24, y = 6388, w = 9576, h =   34 (i.e. the lower region)
2. x =  0, y =    0, w =   24, h = 6388 (i.e. the left region)

After overscan calibration, the image has a width of 9576 and a height of 6388 pixels since the black borders are cropped. It made no difference which Source region was used for the overscan calibration, the statisitcs of the calibrated images were almost identical.

However, in the FITS headers of the calibrated images, the rectangles used by Overscan calibration are defined differently: by the upper left and the lower right point coordinates:

Image and target region:
x1 = 24, y1 = 0, x2 = 9600, y2 = 6388

Source region:
1. x1 = 24, y1 = 6388, x2 = 9600, y2 = 6422
2. x1 =  0, y1 =    0, x2 =   24, y2 = 6388

This might have been somewhat confusing.

Bernd

[smccully]

I used these overscan settings,

Image: x = 24, y = 0, w = 9576, h = 6388

Source 1: x1 = 0, y1 = 0, x2 = 24, y2 = 6422

Source 2: x1 = 24, y1 = 6388, x2 = 9600, y2 = 24

It worked except for two images, which were the first images in each set that. Not sure why these images were failing, but you said it worked fine for you?

Also, i am wondering if Overscan and Master BIAS and Master Dark are set will the overscan be used in conjunction with Master Bias or will the Overscan be used over the Master bias? Dark?

[smccully]

Ive tried again using new set of images, and the below overscan settings and it still fails on the first image but the rest are fine.


Image: x = 24, y = 0, w = 9576, h = 6388

Source 1: x1 = 0, y1 = 0, x2 = 24, y2 = 6388

Source 2: x1 = 24, y1 = 6388, x2 = 9600, y2 = 6422

[bulrichl]

I am confused by the provided information about the overscan settings that you used for the source regions. As I stated above, in ImageCalibration, ALL of the rectangles are defined by the coordinates x, y of the upper left point and width and height of the rectangle. This does not only apply to the image and target region but also applies to the source region!

In my post above I just tried to figure out which settings could work based on the single raw light frame, "M84-0092-B.fit", that was uploaded by you. However, my conclusion might be wrong. The necessary information about how to correctly set up the overscan calibration should be contained in the documentation of your camera. I am, however, unclear whether it is reasonable at all to apply an Overscan correction for this camera model which seems to be a QHY600 (with a CMOS sensor).

I want to explain the Overscan correction in PixInsight in more detail.

Here is an excerpt of the mouse-over text of 'Overscan #1' in section 'Overscan':

... A source overscan region is used to compute an overscan correction value as the median of all source region pixels that will be subtracted from all pixels in the corresponding target region.

So the sole purpose of this function is to correct for a drifting bias offset over time. If the bias offset of your camera is stable, you don't need Overscan calibration.

Generally, if you decide to use the Overscan correction, you need to
- recapture all the calibration frames with overscan enabled in the camera driver and
- and apply overscan-calibration to each of the individual calibration frames in use when preparing the master calibration files.

Definitely you must not use Overscan calibration for the calibration of the light frames, and at the same time continue using older master calibration files that are produced without overscan calibration -- this approach will yield a wrong calibration result.

Bernd

[smccully]

k, well if that is definetly the case then i will turn off overscan. My misunderstanding was that with overscan on, it just included this drifting bias and when turned off it wouldn't have an overall affect. So far the calibration i've done with the overscan on, have turned out alright.

Just to clarify the issue i am having, is that the source and target regions work as expected except that i get an error "inconsistent geometry" for the first image in each set. The rest of the images process fine.

This is what the QHYCCD Documentation says about overscan

The optic black area and overscan area is usually used for precise calibration and for the calibration without the bias frame and dark frame, or other scientific using. Due to it is an non-image area ,  QHYCCD does not guranttee the signal quality in the overscan area. If you do not use this area, you can select the option "ignore overscan area" in the ASCOM driver or select a ROI of effective area range in SharpCAP.

https://www.qhyccd.com/index.php?m=content&c=index&a=show&catid=30&id=260

[smccully]

Maybe you could help answer one last question, is i am still a little confused about how the overscan feature is utilized. From the QHYCCD Documentation i referenced previously,

The optic black area and overscan area is usually used for precise calibration and for the calibration without the bias frame and dark frame

So i just realized that actually the first overscan area, Source 1 is actually what QHYCCD calls the "Optic Black Area".

1. Would the Optic Black Area still be used in the overscan definition?
2. Would the overscan region be for bias and the optic black area dark frame?
3. If using Overscan/Optic Black, do you not need to utilize a master bias? master dark?
3. If you are using a master calibration frames and the overscan would the results be better?


Thanks

[bulrichl]

The only effect of enabling 'Ignore overscan area' in the camera driver is that the outputted image will be cropped. In case of the QHY600, the full image will be cropped by 24 pixels at the left side and by 34 pixels at the bottom.

Yes, I also read the part of the QHY600 manual that covers optically black and overscan region in the section "Image Area Layout". It seems very general and vague to me. They don't give any hints about how to use these areas of the sensor for the calibration.

Generally, the optically black area is a region on the sensor that is equipped with photosites that are covered with a nontransparent layer, and the overscan area does not contain any photosites. So the intensities of the optically black area contain the constant bias offset + dark current + read noise, whereas the intensities in the overscan area contain only the constant bias offset + read noise.

Since PixInsight subtracts the median of the source region (i.e.: a constant value) from the target region, Overscan calibration cannot be taken as a replacement for applying a MasterDark.

As I wrote before, if you decide to use Overscan correction, this must be applied as well to all calibration frames in use, otherwise the calibration result will be wrong. The only purpose of Overscan calibration is to correct for a drifting bias offset over time.

Regarding your questions:
1. I guess that either could be used, but definitely not both at the same time. The value computed from the optically black area would additionally contain the average dark current, and the value computed from the overscan area would not.
2. One could think of it this way.
3. Using PixInsight's Overscan calibration cannot be taken as a replacement for applying a MasterDark. Since Overscan calibration only subtracts a constant value from all pixels in the corresponding target region, fixed pattern noise is not at all corrected by this procedure.
4. In case of a drifting bias offset, the calibration result will be more precise. If the bias offset of your camera is stable, using Overscan calibration in PixInisght will not improve the calibration result.

Bernd

[smccully]

Thank you for this, i guess i should try to determine if the bias is drifting at all.