flat issue

[Cafo78]

Hi all,

here I am again with a Flat issue. I have a Dslr (canon550D) modified camera and I took lights (240"/800iso) darks (240"/800iso) and flats (0,8"/800 iso to get 25400adu equivalent to 6360 adu in 14 bit, with flat wizard in Sgpro). The problem is the stacking/calibration process in Pixinsight. The final image coming out is very strange with excessive luminance on the borders as if the flat frames weren't doing their job.

The issues came out since I started using Sgpro: I save all the files in Fits format. Using APT instead, saving the files in the native raw format, everything works fine.

What's wrong in here? Any suggestion? Should I change the saving format in Sgpro in raw instead of fits? Did i make something wrong in taking flats?
Reading in internet in various forum, seems like I'm not the only one having this kind of issue with sgpro and flats...

[Cafo78]

I forgot to add one thing: a strange thing I noticed is when I open the flat in Pixinsight (linear image) is brighter then a STF one. The screen stretched one is darker.. should not be the opposite? So maybe the flats are overexposed for some reason?

[Cafo78]

here the images. The left is linear, the right one is stretched

[pfile]

did you mix CR2 and FITS anywhere in this process?

SGP scales CR2/NEF files to 16-bits when saving as FITS so they will become incompatible with CR2/NEF which PI opens unscaled.

rob

[Cafo78]

Hi rob,

nope, I did not. Each files was taken with Sgpro and saved as Fits.
As I mentioned, flats have been taken aiming for 25.000adu. Measuring the statistics in pixinsight it shows about 6000 ADU in 14 bit, which I think should be ok.
But the final image evidence is flats were overexposed.
Do I have to try lowering the target ADU to 20.000 or less?
Reading different forum seems like a lot of people had the same problem taking flats with sgpro..

[pfile]

generally this kind of over-flattening occurs because of bad flat calibration. the flats are probably not overexposed.

try calibrating the flats with a master bias only and see what you get. if you are going to use darks to calibrate the flats then they should be 0.8" in length. 240" darks with no scaling subtracted from the flats will definitely mess things up. since you didn't mention a master bias, then i assume you didn't use dark scaling (optimization.)

rob

[Cafo78]

I used a superbias... and I put everything in batch preprocessing leaving PI to decide what’s the best to do. I did not get any error.

The strange thing is that before using sgpro, I used apt and I’ve never had such issues. I never took dark-flats of the same duration and I never had problem. I was used to take darks of the same duration of light, and flats of about half second (just to have the histogram halfway of the scale) and everything worked fine.

So I suppose the problem is the way SGPRO saves/scales/I don’t not what it does...

[pfile]

no, the problem is that WBPP or BPP is probably trying to apply the 240s dark to the flat by scaling it. you can read the console log to see what happened. i think there is now a button that says "calibrate with flat darks" which should cause WBPP/BPP to try to find darks that match the flats, but since i never use BPP or WBPP i am not exactly sure how it finds the matching dark. since you are using FITS files there is a pretty good chance that it would be able to make the match.

so you might try taking some 0.8" darks and loading those too and then ticking that box and see what happens.

as long as you are using all files of the same type (meaning every single frame, light, dark, flat dark, bias, flat) came out of SGP in the FITS format it should work out fine.

rob

[Cafo78]

Ok, I tried to do all the calibration process manually, checking the optimization box in the dark section...but I got the same result in the final image: lighted edges as if the flats were overexposed.

So, summing up, it’s not a problem with Batch pre-processing. It’s not a problem with different file format (everything is in fits); If I check the “optimize” box in dark section, from what I know, the darks should calibrate the flats even if they have different exposure time.
I will try taking flat darks of 0,8” but my question and my doubt still remain: why when I used apt I did not have all of these issues (I never had the necessity to take flat darks) and now I can’t have a decent calibration with files in fits format from sgpro? I really can’t understand what’s wrong or different...

[pfile]

if you upload the files to dropbox or google drive or similar someone can take a look...

rob

[Cafo78]

Here's the link where I put some lights, darks, flats and bias.
If someone can take I look I really appreciate.

Thanx



https://www.dropbox.com/sh/36s1lyb7n1st22h/AACsQQ9pYVla11yV7ITS7_vFa?dl=0

[wsmith1]

Hi Cafo and all
just my twopence worth
My last experience using the batch preprocessing script where you add all your lights darks bias and flats all in one place and run the thing, i also got a final stacked image where it looked like the stack "added" my flats rather than "subtracted" dust specs and vignetting...

Another coincidence is that im also a new user to SG Pro and all my subs darks etc were taken in sg pros format...

I also didnt add bias as this created problems for me to state the overscan area of my QHY10 camera so i knocked that whole bias integration on the head... 

wayne

[bulrichl]

H Mirco,

Used camera: Canon EOS 550D (14-bit ADC, FITS -> RGGB).

Statistics of the superbias compared with a dark frame:

Code: [Select]

superbias
count (%)   100.00000
count (px)  18024930
mean        2048.359
median      2048.562
variance    1.738
stdDev      1.318
avgDev      1.406
MAD         1.641
minimum     2044.464
maximum     2052.339

Code: [Select]

Dark frame: ic434_2019_12_31_240sec_ISO800_281_amb_0_4C_dark_frame9
count (%)   100.00000
count (px)  18024930
mean        8198.4
median      8195.0
variance    20122.6
stdDev      141.9
avgDev      56.7
MAD         53.4
minimum     7239.0
maximum     61223.0
The median of the superbias is 2048.6, the median of the single dark frame is 8195. My conclusion: The superbias was generated from NOT scaled bias frames (presumably from bias frames in CR2 format). So the superbias does not match all the other frames.

I made the following steps with your frames (just quick and dirty -- not to be recommended):
Multiplied (in PixelMath) the superbias by factor 4.0, calibrated the flat frames with this modified superbias and integrated the calibrated flat frames to a MasterFlat. Integrated the dark frames to a MasterDark. Calibrated the light frames with the MasterDark (both 'Calibrate' and 'Optimize' disabled) and the MasterFlat ('Calibrate' disabled).

The image that resulted after debayering, star alignment and integration (see appended integration.JPG) has a bright stripe (of about 230 pixels in height) at the bottom. This artifact is in all calibrated lights as well, so it does not result from dithering or drift. It could be caused by the fact that the red channel in the flat frames is very dim. Apart from that, the integration result doesn't look wrong.

Suggestions:
1) Delete the superbias.

2) If your bias frames are in CR2 format (this is the only reasonable explanation for the above finding), delete them.

3a) If you don't want to try dark frame optimization, you need not capture bias frames at all. In this case, take flat darks (same ISO and exposure as the flat frames), integrate them to a MasterFlatDark and use this MasterFlatDark for flat frame calibration. Integrate the calibrated flat frames to the new MasterFlat.

Calibrate the light frames with the MasterDark (both 'Calibrate' and 'Optimize' disabled) and the MasterFlat ('Calibrate' disabled).

3b) If you want to try dark frame optimization, In this case, take new bias frames with SGP in FITS format, integrate them to a MasterBias and use this MasterBias for flat frame calibration. Integrate the calibrated flat frames to the new MasterFlat.

Calibrate the light frames with the MasterDark (both 'Calibrate' and 'Optimize' enabled) and the MasterFlat ('Calibrate' disabled).


Compare the resulting integrations, and for the future use the process that provides superior results (either 3a) or 3b)).

Bernd

[pfile]

thank you Bernd, you are truly the PI calibration expert!

rob

[Cafo78]

Hi Bernd,

thank you very much for the explanation. I was getting crazy 
You really solved the mess...
Thanx to everybody for the contribute 

Mirco