Flats issue with my OSC

Jkulin

Well-known member
Apr 14, 2017
99
9
Hi All, before I commit myself into a mental hospital can anyone advise.

I don't have any flats issues now that I sorted some rotation problems with my 8300 in my observatory, but I cannot get around the issue with my other rig with is a Esprit 100, OAG and a Altair Astro 26c Camera (26mp): - https://www.altairastro.com/altair-hypercam-26c-aps-c-colour-camera-16bit-6451-p.asp

The problem is that I take the flats at the end of each session and when I look at them at 30K ADU in SGP they look like this, I took a total of 25 flats at 13.6secs using my flat panel, I have even used two different ones and experienced a similar issue: -

1618510666109.png
When I stack them using the latest WBPP, with a similar stretch they look like this: -
1618510720164.png

As you can see there does not appear to be any correctional dust bunnies in the Master Flat, so I presume that if there is none visible then the WBPP doesn't have any artefacts to remove from the Master Light?

I have has some nice images, just ruined because the flats are not correcting in the master light.

I am not sure if the issue is because I am using a OSC camera and I am expecting too much from it or if I need to change a parameter, I kept a process icon for it and this is the script details: -

var P = new Script;

P.filePath = "$PXI_SRCDIR/scripts/WeightedBatchPreprocessing/WeightedBatchPreprocessing.js";

P.md5sum = "0bb9c89079eae8fb76daa249db97a815";

P.parameters = [ // id, value

["version", "2.0.2"],

["saveFrameGroups", "false"],

["outputDirectory", "F:/Captures/M86_2nd/Processed_150421"],

["upBottomFITS", "true"],

["saveProcessLog", "true"],

["generateRejectionMaps", "false"],

["darkOptimizationLow", "3"],

["darkOptimizationWindow", "1024"],

["darkExposureTolerance", "10"],

["evaluateNoise", "true"], // row 10

["overscanEnabled", "false"],

["overscanRegionEnabled_1", "false"],

["overscanSourceX0_1", "0"],

["overscanSourceY0_1", "0"],

["overscanSourceX1_1", "0"],

["overscanSourceY1_1", "0"],

["overscanTargetX0_1", "0"],

["overscanTargetY0_1", "0"],

["overscanTargetX1_1", "0"],

["overscanTargetY1_1", "0"], // row 20

["overscanRegionEnabled_2", "false"],

["overscanSourceX0_2", "0"],

["overscanSourceY0_2", "0"],

["overscanSourceX1_2", "0"],

["overscanSourceY1_2", "0"],

["overscanTargetX0_2", "0"],

["overscanTargetY0_2", "0"],

["overscanTargetX1_2", "0"],

["overscanTargetY1_2", "0"],

["overscanRegionEnabled_3", "false"], // row 30

["overscanSourceX0_3", "0"],

["overscanSourceY0_3", "0"],

["overscanSourceX1_3", "0"],

["overscanSourceY1_3", "0"],

["overscanTargetX0_3", "0"],

["overscanTargetY0_3", "0"],

["overscanTargetX1_3", "0"],

["overscanTargetY1_3", "0"],

["overscanRegionEnabled_4", "false"],

["overscanSourceX0_4", "0"], // row 40

["overscanSourceY0_4", "0"],

["overscanSourceX1_4", "0"],

["overscanSourceY1_4", "0"],

["overscanTargetX0_4", "0"],

["overscanTargetY0_4", "0"],

["overscanTargetX1_4", "0"],

["overscanTargetY1_4", "0"],

["overscanImageX0", "0"],

["overscanImageY0", "0"],

["overscanImageX1", "0"], // row 50

["overscanImageY1", "0"],

["combination_1", "0"],

["rejection_1", "999"],

["minMaxLow_1", "1"],

["minMaxHigh_1", "1"],

["percentileLow_1", "0.20000000298023224"],

["percentileHigh_1", "0.10000000149011612"],

["sigmaLow_1", "4"],

["sigmaHigh_1", "3"],

["linearFitLow_1", "5"], // row 60

["linearFitHigh_1", "3.5"],

["ESD_Outliers_1", "0.30000001192092896"],

["ESD_Significance_1", "0.05000000074505806"],

["combination_2", "0"],

["rejection_2", "999"],

["minMaxLow_2", "1"],

["minMaxHigh_2", "1"],

["percentileLow_2", "0.20000000298023224"],

["percentileHigh_2", "0.10000000149011612"],

["sigmaLow_2", "4"], // row 70

["sigmaHigh_2", "3"],

["linearFitLow_2", "5"],

["linearFitHigh_2", "3.5"],

["ESD_Outliers_2", "0.30000001192092896"],

["ESD_Significance_2", "0.05000000074505806"],

["combination_3", "0"],

["rejection_3", "999"],

["minMaxLow_3", "1"],

["minMaxHigh_3", "1"],

["percentileLow_3", "0.20000000298023224"], // row 80

["percentileHigh_3", "0.10000000149011612"],

["sigmaLow_3", "4"],

["sigmaHigh_3", "3"],

["linearFitLow_3", "5"],

["linearFitHigh_3", "3.5"],

["ESD_Outliers_3", "0.30000001192092896"],

["ESD_Significance_3", "0.05000000074505806"],

["combination_4", "0"],

["rejection_4", "4"],

["minMaxLow_4", "1"], // row 90

["minMaxHigh_4", "1"],

["percentileLow_4", "0.20000000298023224"],

["percentileHigh_4", "0.10000000149011612"],

["sigmaLow_4", "4"],

["sigmaHigh_4", "3"],

["linearFitLow_4", "5"],

["linearFitHigh_4", "3.5"],

["ESD_Outliers_4", "0.30000001192092896"],

["ESD_Significance_4", "0.05000000074505806"],

["flatsLargeScaleRejection", "false"], // row 100

["flatsLargeScaleRejectionLayers", "2"],

["flatsLargeScaleRejectionGrowth", "2"],

["lightsLargeScaleRejectionHigh", "true"],

["lightsLargeScaleRejectionLayersHigh", "2"],

["lightsLargeScaleRejectionGrowthHigh", "2"],

["lightsLargeScaleRejectionLow", "true"],

["lightsLargeScaleRejectionLayersLow", "2"],

["lightsLargeScaleRejectionGrowthLow", "2"],

["imageRegistration", "true"],

["lightExposureTolerance", "2"], // row 110

["generateDrizzleData", "true"],

["cosmeticCorrection", "false"],

["cosmeticCorrectionTemplateId", ""],

["pixelInterpolation", "10"],

["clampingThreshold", "0.30000001192092896"],

["maxStars", "0"],

["distortionCorrection", "false"],

["structureLayers", "5"],

["minStructureSize", "0"],

["noiseReductionFilterRadius", "0"], // row 120

["sensitivity", "0.10000000149011612"],

["peakResponse", "0.800000011920929"],

["maxStarDistortion", "0.5"],

["useTriangleSimilarity", "false"],

["referenceImage", ""],

["subframeWeightingPreset", "1"],

["generateSubframesWeights", "false"],

["useBestLightAsReference", "true"],

["forceMeasurement", "false"],

["FWHMWeight", "5"], // row 130

["eccentricityWeight", "10"],

["SNRWeight", "20"],

["pedestal", "65"],

["integrate", "true"],

["keywords", "W10="],

["frameGroups",

Any help would be appreciated.

Thansk

Regards,

John
 

fredvanner

Well-known member
Apr 17, 2019
1,197
156
70
Wells, Somerset, UK
I'm puzzled. Your Altair is an OSC camera, so your integrated flats should be RGB. It looks as though they have been integrated without debayering, and left as a greyscale bayer-matrix. If you processed them with WBPP, I think you forgot to set them as CFA.
 

Jkulin

Well-known member
Apr 14, 2017
99
9
Hi Fred,

Thanks for your reply, I did check the CFA box as can be seen and for the lights I set it as RGGB: -

1618512603430.png

I am advised by the resellers that there is no need for bias subs.
 

Attachments

fredvanner

Well-known member
Apr 17, 2019
1,197
156
70
Wells, Somerset, UK
The master flat is of course greyscale, buy you can't compare that with a debayered image (such as the first image). If you debayer your new master flat, and then apply an STF, I think you will find the dust bunnies.
 

Jkulin

Well-known member
Apr 14, 2017
99
9
Thanks Fred, yep your are correct and I can see them now as below: -
1618514502547.png

So my questions are: -
1. Does the WBPP script only debayers lights and not flats?
2. If so should I debayer the flats before adding them into the WBPP script
2. Is the green cast going to be an issue.

I really appreciate your help Fred, I owe you one!
 

fredvanner

Well-known member
Apr 17, 2019
1,197
156
70
Wells, Somerset, UK
I am advised by the resellers that there is no need for bias subs.
You can probably live (and maybe even do better) without bias subs, but only if you replace them with a valid workflow including darks and "dark flats". (this effectively includes the classic "bias" component in the darks, but with a value measured over the correct exposure time).
 

Jkulin

Well-known member
Apr 14, 2017
99
9
Thanks Fred, OSC is new to me as I have been using for years CCD which is so much less complicated.

I haven't got my head around dark flats, I assume that I have to take darks at the duration as the darks or am I getting confused?
 

fredvanner

Well-known member
Apr 17, 2019
1,197
156
70
Wells, Somerset, UK
1. Does the WBPP script only debayers lights and not flats?
The undebayered master flat is used to calibrate the lights. You only needed to debayer it because you wanted to examine it.
2. If so should I debayer the flats before adding them into the WBPP script
As above - the debayering was only for inspection, and is otherwise not needed (WBPP will correctly integrate the undebayered flats and use them to calibrate the undebayered lights before debayering).
3. Is the green cast going to be an issue.
No. it may be there for various reasons (filters; genuine camera channel imbalance), but will be the same in the lights that it is calibrating.
 

fredvanner

Well-known member
Apr 17, 2019
1,197
156
70
Wells, Somerset, UK
I assume that I have to take darks at the duration as the darks or am I getting confused?
This sentence is certainly confused!:sneaky:
"dark flats" are darks (i.e. with the aperture covered - and processed as dark frames) with the same duration as the flats. If your flats are shorter exposures than ~2s then it is worth trying to make them longer - some modern C-MOS cameras prefer it (wind your flat panel down a bit or put one or two sheets of paper in front of it).
 

Jkulin

Well-known member
Apr 14, 2017
99
9
This sentence is certainly confused!:sneaky:
"dark flats" are darks (i.e. with the aperture covered - and processed as dark frames) with the same duration as the flats. If your flats are shorter exposures than ~2s then it is worth trying to make them longer - some modern C-MOS cameras prefer it (wind your flat panel down a bit or put one or two sheets of paper in front of it).
Thanks Fred, I can certainly wind the flats down but I doubt that I could achieve a matching duration to the lights, or am I getting confused again?


As above - the debayering was only for inspection, and is otherwise not needed (WBPP will correctly integrate the undebayered flats and use them to calibrate the undebayered lights before debayering).
That's the issue Fred the flats are not being subtracted from the lights using the WBPP script as you can see in the partially processed image, long way to go but the dust bunnies are clearly visible, this has just been DBE'd and PCC'd nothing else as I stopped when I saw the dust still present, the camera has not moved at all: -

1618515937138.png
 

fredvanner

Well-known member
Apr 17, 2019
1,197
156
70
Wells, Somerset, UK
or am I getting confused again
Possibly. The idea is not to match (or even approach) the lights. It is just to avoid very short flat exposures. I'm not sure that the astro community has entirely settled on a verdict on what C-MOS camera want, but flat exposures >2s seems to be an emerging opinion.
I think you may need a more expert opinion than mine on your flat correction problem, but your dust artefacts look to me to have the slightly "embossed" appearance that I associate with a slight shift of the dust pattern (e.g. a slight mechanical movement of a filter between taking the lights and taking the flats).
 

Jkulin

Well-known member
Apr 14, 2017
99
9
Thanks Fred, I need to work on my knowledge on OSC and Flat Darks.

There can't be a filter shift as there is no filter as this is OSC and everything is locked up, I have since getting this OSC camera lots of Flat removal problems.

Is there a way to debayer the flats prior to processing as I tried and it doesn't like it.

Looks like another virtual beer for you.
 

ngc1535

PixInsight Ambassador
Feb 1, 2014
758
136
AdamBlockStudios.com
Uncorrected dust donuts that take the "embossed" form are generally not only due to "filter shift"- in the more general sense it means that the angle the light was illuminating the chip is different. This could be due to a number of mechanical issues- as well as scattering in the optical system causing unwanted off-axis light rays to illuminate the sensor.

-adam
 

fredvanner

Well-known member
Apr 17, 2019
1,197
156
70
Wells, Somerset, UK
Is there a way to debayer the flats prior to processing
A valid workflow for an OSC camera had some unavoidable constraints that you need to understand.
Because the OSC image is composed of bayer matrix cells, all calibration up to the point of debayering must keep rigid alignment of this matrix. Effectively all calibration processing is done independently on each (undebayered) pixel stack (i.e. each image (x,y) position). This includes dark and flat integration to form masters, and dark and flat calibration of the light frames with these masters (and cosmetic correction if you choose to do it). Only after all this calibration are the final calibrated light frames debayered. The only reason for debayering any of the calibration frames (biases, if used; darks - of all sorts; flats) is for inspection - and because they are never used in this debayered form, this inspection is generally of little value (inspecting flat frames is a bit of an exception, since it can reveal valuable information, even though the flat is never used in the debayered form). The debayered versions will never contribute to a calibrated image.
So - if you are thinking of trying to debayer your flats and then using them for calibration it simply doesn't make sense - all frames forming part of a bayer matrix (OSC) calibration workflow must be in pixel-aligned bayer matrix (CFA) greyscale format.
 
Last edited: