Issues with removing vignetting, errors with batch preprocessing zero signal

[markm75]

I have a sort of two part issue going on.  I'm still somewhat new to PI though, so maybe its something obvious that needs adjusted

My equipment is the edge 11 with hyperstar v3 + asi294mc pro.  I take flats with a spike a flat at near the lowest level of brightness with 2 "thin sweatshirts" of material over it (otherwise too bright).  In this example at least i think my flats were close to centered and around 24,000 adu.  This is with a zwo electronic filter wheel and 1.25" l-enhance filter in place.  I take the flats then kill the spike a flat in the dark and do the dark flats with the exact same settings (using APT flat wizard).

The issue is that i still end up with vignetting once the batch process finishes.  This example isnt the same target as the one i'm referencing in the second part but gives the idea:


And using background extraction no matter what i do, i cant get rid of the vignetting, it only makes things worse, i end up with rings.
Here is the histogram for the flats.



I opted to rerun the batch process and try again with a different target (flaming star nebula) and now i'm getting this error at the end with: failure to integrate light frames and zero or insignificant signal detected.

I've uploaded this example to this location if anyone wants to take a stab at it and see what could be going on.  I simplified it on this last attempt, keeping it at around 10 light frames, 20 dark flats/flats and 50 darks. 

Thanks for any suggestions

[ngc1535]

Mark,

Let me preface this by saying, few people listen to me in terms of advice! (Because I am wrong very often.)
So with that out of the way- one thing that strikes me as problematic from my years of experience with flats- is you never want to take darks following flats.
Many chips have some degree of RBI (Residual Bulk Image/Charge) that will leave a raised pedestal in your darks- and this will mess everything up when you calibrate your flats with these darks.
If your exposures are short anyway, it might be better to just use bias frames (but again, don't create biases or darks after flats).

So my idea is a simple one to test. Take your darks (or biases) when you have your camera at temperature but *before* you take the flats.
See if you get different results when you calibrate your images.

-adam

EDIT: You are using a CMOS chip I believe..I am certain someone will scream RBI isn't possible. OK
However, please see this CloudyNight post which seems on-topic and justifies my comment I think:
https://www.cloudynights.com/topic/662887-not-rbi-on-cmos-but-this-looks-similar/

[markm75]

Thanks for the suggestion.  I didnt know this about the dark flats.

The only issue is, that i've read the asi294 should be done with dark flats not bias's.  So i guess if this is the case i have to do the darks first (or how long to wait after a light is done or to reset things before a dark).

My other issue is that as i add layers to dim this panel, as even at a setting of around 3 or 5 its too bright for certain filters, i end up with a merged histogram where the first two peaks are embedded in each other.

Then there are Ha filters, where i only get two good peaks and 3rd flat one on the right.

[bulrichl]

I use a ASI294MC Pro by myself, so I know the camera and how the calibration files should look like.

1. In the data that you uploaded I don't find 'flat darks' (this is the right term for darkframes that are captured with the exposure time of the flat frames, in your case 4.25 s). The 20 files in the folder "Dark Flats" in fact are flat frames! Please remove the files in that folder and provide the flat darks.

2. I have never seen dark frames that show such uneven intensities. I integrated the dark frames to a MasterDark (screen section appended). The intensity in the center of the MasterDark is 2860 ADUs, and at the corners 2400 - 2650 ADUs -- this is not plausible at all. Either what Adam described in his EDIT note applies or you got light incidence during dark frame acquisition.

3. The minimum intensity in the first light frame (see appended screen section) is 2084 ADUs, the minimum in the MasterDark is 2360 ADUs. When the MasterDark is subtracted from this light frame, negative values result for 25 % of the pixels. Predominantly the red channel is affected by the clipping. This is the reason for the error message "zero or insignificant signal detected" during ImageIntegration. Certainly the present dark frames are completely unusable.

It is always a good idea to check some calibration files, light frames and calibrated light frames with ImagesStatistics and HistogramTransformation. Incorrect image calibration cannot be ironed out in post-processing.

Bernd

[markm75]

I use a ASI294MC Pro by myself, so I know the camera and how the calibration files should look like.

Thanks for the reply and for looking at that data set.

That was a brainfart on my part, i should have checked those flat darks when i did them.  Not sure how that happened.  Same deal with the 180sec darks.

I dont know why the 180 sec ones ended up like that.  When i do darks at least, the camera goes in my drawer in the nexdome, inside part way the carrying case, with vents left sticking out. 
I have a new data set at the same location that can be redownloaded.  A different set of flats and flat darks as well.  I included a set of darks that are 180sec, but this set was from an older 121gain matching set i had done, not sure they are useful here being off by 1, maybe? 

Curious, how do you determine the adu near the center vs the edge, is there a tool in PI that you can hover ?

Here is the histo for them:


The main issue is the flats and flat darks i believe.
I think its been said that my flats are "inaccurate" before, unsure what that means, maybe in terms of the histogram?

Here is the flats histogram for this set:



What i've found is it seems like the more material i have on the flats panel the more distorted the histogram can become (IE: 2 channels seemed to merge like for L-enhance), so for some sets i peeled back a layer to compensate.  For Ha, they always seem to have 2 peaks (more left shifted and not centered too) along with the flat 3rd.

I was considering trying 20% window tint instead of the sweatshirt material, as some said it worked.  Maybe only need one layer:
https://www.homedepot.com/p/Gila-Ultra-Shield-Max-5-Super-Limo-Black-Automotive-Window-Tint-10382377/203185049

I now have the spike a flat usb device so i can dumb down the light to a "5" or so in intensity.

[markm75]

I had a chance to go through the batch preprocessing with the last set of flats/darks and much to my amazement, it worked, perfectly i think.

The initial image was "blue" but no signs of vignetting at all.

So i did the automatic background extraction. and this was the result so far (you can see the need for darks with the upper right, i didnt apply the darks in this example, just flats and flat darks, the flats with the histogram above).  Not sure why other attempts at different targets have had terrible success, but maybe it was flat dark related.  This flat wasnt perfectly centered either.

[markm75]

So here is another example of a failure of sorts..

I tried using flats that were done pretty much after this image was taken (original date of this target) and its quite ugly:


However, if i switch and use the flats and flat darks from 11-27 that worked in the previous post for that target, i get a good result initally (25x30sec) after manual background extraction this time

 
If anyone would like to look at the data i've put it here along with the masters for the failed attempt (and log)  here (contained here is the flats/flat darks from 11-27 that worked in both cases along with the ones that are not working that really should).

I tried opening at least the flats one by one to see if i could figure out what might be causing the issue, but nothing jumped out at me right away (not sure if there is a faster way to go one by one, i tried blink but i wasnt sure if the histo was actually updating)

[bulrichl]

Lights were created with N.I.N.A: 1.10.0.10 (gain 120, offset 30), flat darks and flats are created with APT v.3.7.1 (gain 120, offset not written to the FITS header). We had it in another thread just recently:

It is not recommended to use different capturing software for light frames and calibration frames!

---

FlatDarks captured at 25th November 2019 (folder "/flat darks/"):

The flat darks in the folder "/flat darks/" have a median of about 77 ADUs which is not plausible at all. Presumably these frames were captured with a different (higher) offset setting than most of the other frames. Unfortunately the offset is not visible for the frames captured with APT.

Accordingly the MasterFlatDark "dark-BINNING_1-EXPTIME_3.5.xisf" has a median of 77.257.

All of these files (flat darks of 25th November 2019, MasterFlatDark and the integration of the calibrated light frames) are unusable and should be deleted.

---

FlatDarks captured at 27th November 2019 (folder "/flat darks 2019-11-27 L-enhance fw 120 sweatshirt GOOD peaks good 2.5sec (WORKS)/"):

The first file in the folder "/flat darks 2019-11-27 L-enhance fw 120 sweatshirt GOOD peaks good 2.5sec (WORKS)/", "F_5-Lenhance_2019-11-27_18-42-16_Bin1x1_2.5s__-10C.fit", is a flat frame, NOT a flat dark. It belongs to the folder "/flats-2019-11-27 L-enhance fw 120 sweatshirt GOOD peaks good 2.5sec done (WORKS)/".

The remaining 35 flat darks have a median of 1916 ADUs which is in the expected range.

---

Correct dark frames and consequently a correct MasterDark are missing.

Additional to using the incorrectly captured flat darks and the missing dark frames there are mistakes using the script Batch Preprocessing:

==================================================================================

[2019-12-04 20:08:04] ************************************************************
[2019-12-04 20:08:04] * Begin integration of dark frames
[2019-12-04 20:08:04] ************************************************************
[2019-12-04 20:08:04]
[2019-12-04 20:08:04] ImageIntegration: Global context
[2019-12-04 20:08:04]
[2019-12-04 20:08:04] Opening files:
[2019-12-04 20:08:04]
[2019-12-04 20:08:04] B:/Data/astronomy/Calibration Captures/Dark Flats/2019-11-25 L-enhance fw hyperstar 1.25 120 sweatshirt 3.5sec GOOD/F_5_2019-11-25_20-46-31_Bin1x1_3.5s__-10C.fit
[2019-12-04 20:08:04] 36 FITS keywords extracted.
[2019-12-04 20:08:04] Reading FITS image: 16-bit integers, 1 channel(s), 4144x2822 pixels: done
[2019-12-04 20:08:04] Computing image statistics: done
[2019-12-04 20:08:05] Weight          :     1.00000

...

[2019-12-04 20:08:32] ************************************************************
[2019-12-04 20:08:32] * End integration of dark frames
[2019-12-04 20:08:32] ************************************************************
[2019-12-04 20:08:32]
[2019-12-04 20:08:32] * Writing master dark frame:
[2019-12-04 20:08:32] B:/Data/astronomy/Captures/M42 Orion Nebula M42 hyperstar fw L-ehance/2019-11-25 30s 120 gain/PPoutput no darks/master/dark-BINNING_1-EXPTIME_3.5.xisf
[2019-12-04 20:08:32] Writing image 'integration': w=4144 h=2822 n=1 Gray Float32
[2019-12-04 20:08:32] 186 FITS keyword(s) embedded.
[2019-12-04 20:08:32] 16 image properties embedded.
[2019-12-04 20:08:32] Writing image 'rejection_low': w=4144 h=2822 n=1 Gray Float32
[2019-12-04 20:08:32] Writing image 'rejection_high': w=4144 h=2822 n=1 Gray Float32
[2019-12-04 20:08:33]
[2019-12-04 20:08:33] * Searching for a master flat dark frame with exposure time = 30s -- not found.
[2019-12-04 20:08:33]
[2019-12-04 20:08:33] * Calibration of flat frames skipped -- no flat darks found.
[2019-12-04 20:08:33]
[2019-12-04 20:08:33] ************************************************************
[2019-12-04 20:08:33] * Begin integration of flat frames
[2019-12-04 20:08:33] ************************************************************

...

[2019-12-04 20:09:18] ************************************************************
[2019-12-04 20:09:18] * End integration of flat frames
[2019-12-04 20:09:18] ************************************************************
[2019-12-04 20:09:18]
[2019-12-04 20:09:18] * Writing master flat frame:
[2019-12-04 20:09:18] B:/Data/astronomy/Captures/M42 Orion Nebula M42 hyperstar fw L-ehance/2019-11-25 30s 120 gain/PPoutput no darks/master/flat-FILTER_L-enhance-BINNING_1.xisf
[2019-12-04 20:09:18] Writing image 'integration': w=4144 h=2822 n=1 Gray Float32
[2019-12-04 20:09:18] 244 FITS keyword(s) embedded.
[2019-12-04 20:09:18] 16 image properties embedded.
[2019-12-04 20:09:18] Writing image 'rejection_low': w=4144 h=2822 n=1 Gray Float32
[2019-12-04 20:09:18] Writing image 'rejection_high': w=4144 h=2822 n=1 Gray Float32
[2019-12-04 20:09:18] Writing image 'slope_map': w=4144 h=2822 n=1 Gray Float32
[2019-12-04 20:09:20]
[2019-12-04 20:09:20] * Searching for a master dark frame with exposure time = 30s -- best match is 3.5s
[2019-12-04 20:09:20]
[2019-12-04 20:09:20] ************************************************************
[2019-12-04 20:09:20] * Begin calibration of light frames
[2019-12-04 20:09:20] ************************************************************

==================================================================================

According to this section of the logfile generated by BPP there are several flaws:

1) The flat darks are taken as dark frames. The "MasterDark" in fact is the MasterFlatDark.
2) Surprisingly the script searches for a MasterFlatDark with an exposure time of 30 s (this is the exposure time of the LIGHT frames, the flat frames had an exposure time of 3.5 s!). It does not find one and skips the ccalibration of the flat frames. The uncalibrated flat frames are then integrated and the result is used as the MasterFlat.
3) Then the script searches for a MasterDark with an exposure time of 30 s, finds the one with 3.5 s and takes this.

So there is big confusion. Obviously this is caused by a wrong selection of the sort of frames (light, dark, flat, flat-dark). Normally this has to be set in the capturing software and the rest is done automatically. Alternatively one can denominate the files accord to their sort of frame  (e.g. L for light, D for dark, F for Flat) and select them manually in BPP with 'Add Custom'.

The bottom line is: the flat frames were not calibrated at all. For the calibration of the light frames the MasterFlatDark made from the incorrect flat darks (instead of a MasterDark) and the "MasterFlat" (without subtraction of the bias offset) were used. The result is - not surprisingly - rubbish.

I strongly recommend you to perform the preparation of the master calibration files and the calibration of the light frames at least once separately.step by step (not with BPP but with the processes ImageIntegration, ImageCalibration, Debayer, StarAlignment). See my guide https://pixinsight.com/forum/index.php?topic=11968 for a detailed description and explanation of the preprocessing.

Bernd