trouble with calibration.

dpdallolmo@aol.com

Well-known member
Good day,
I have been using Pixinsight for close to three years and have been delighted with the continuing evolution of WBPP. I have calibrated my OSC camera (both QHY 600 and ZWO 2600) lights (subs) with or without filters like the L enhance or L pro, together with Bias frames (I have a library), Dark frames (more libraries) and Flat frame (which I take after every imaging session). I have used SGP to calculate the time required for my flats (flats calibration wizard). I have tried sky flats (white T shirt method) too. I have had no issues when I populate WBPP with all required frames and check all appropriate boxes to complete WBPP all through to and including image integration.
EXCEPT... The last three imaging sessions I have performed (with a significant moon 60-90% and some smoke) my master light looks as if it was never flat calibrated and the object of interest is washed out or not visible. Most of the stars are also removed and actually my individual subs look better than the master light. I have worked backwards with WBPP, eliminating one step at a time to see where the problem occurs. Where the problem arises is in calibration. It seems as it should be something simple, yet I can't seem to get a good calibrated image. I have run WBPP without flats and the final calibrated image looks just fine, except for vignetting. My files are too large to attach and I'm not sure how to make them smaller. Any suggestions would be appreciated. My computer skills are limited so I suspect it is something simple, especially since I've processed for some time without this issue. Thanks in advance for any help
Dan
 
Sorry for this cheesy way to display the image but I think you can appreciate the diff between a single sub (6119) and the integrated image of over three hours (6120). Excuse my lack of sophistication,
Dan
 

Attachments

  • IMG_6120.jpg
    IMG_6120.jpg
    66.1 KB · Views: 60
  • IMG_6119.jpg
    IMG_6119.jpg
    139.4 KB · Views: 78
There is obviously something seriously wrong with the calibration (though I guess a few of the artefacts come from photographing the screen).
Could you post a screenshot of the WBPP "control panel" tab configured the same way that it was before the run that produced this image.
 
Thank you for looking into this. Yes there are a few artifacts from photographing the screen (reflections). Here are the requested screen shots and my lights and flats "page".
Dan
 

Attachments

  • Screenshot (2).png
    Screenshot (2).png
    305.2 KB · Views: 59
  • Screenshot (3).png
    Screenshot (3).png
    302.1 KB · Views: 61
  • Screenshot (4).png
    Screenshot (4).png
    352.5 KB · Views: 48
  • Screenshot (5).png
    Screenshot (5).png
    327.4 KB · Views: 55
Here are the Master Bias, Master Dark and a portion for size reasons) of a sub prior to any calibration. I've also included a similar sized portion of a registered file so you can better appreciate the changes after calibration. Thank you.
Dan
 

Attachments

  • Screenshot (11).png
    Screenshot (11).png
    246.1 KB · Views: 72
  • Screenshot (12).png
    Screenshot (12).png
    989.2 KB · Views: 53
  • Screenshot (14).png
    Screenshot (14).png
    811.1 KB · Views: 49
  • Screenshot (15).png
    Screenshot (15).png
    684 KB · Views: 53
Hi @dpdallolmo@aol.com,
I have run WBPP without flats and the final calibrated image looks just fine, except for vignetting.
you already investigated and confirmed that the issue occurs when using your flats. I think you should investigate what's different with respect to other sessions where they worked. Very often, when flat calibration leaves that big flat-like footprint, the issue is hidden behind the flat calibration, typically due to a pedestal mismatch issue between the master bias or dark and the flat frames.
I would double-check that your camera parameters and drivers were really the same with respect to the bias frames that you used to create the masterBias. If you confirm that they match then I would do the same check with respect to the settings you've used to capture the light frames, ensure that the camera settings were the same for both light and flat frames, the only parameter that should change is the exposure time.

Robyx
 
Thanks. I am not sure I understand what you are asking me to do. When I use the QHY camera, I always use the same settings; 0 gain 10 offset.
Should I re install the driver for this camera? Where does pedestal play into this? I did try changing the pedestal in WBPP, first to 50, then to 500. No change in result. I wouldn't think a full moon (or close to it) should not cause this issue. Smoke has been plentiful but also, I don't think this factors in.?. How would I go about telling if there was a mismatch in pedestal between bias and flat or dark and flat frame. I didn't see it in the FITS header.
Again thank you for helping me figure this out.
Dan
 
Thanks. I am not sure I understand what you are asking me to do. When I use the QHY camera, I always use the same settings; 0 gain 10 offset.
Should I re install the driver for this camera? Where does pedestal play into this? I did try changing the pedestal in WBPP, first to 50, then to 500. No change in result. I wouldn't think a full moon (or close to it) should not cause this issue. Smoke has been plentiful but also, I don't think this factors in.?. How would I go about telling if there was a mismatch in pedestal between bias and flat or dark and flat frame. I didn't see it in the FITS header.
Again thank you for helping me figure this out.
Dan
There were several cases where users used two software to acquire darks/flat/light frames, one was configured to use native camera drivers, the other was using ASCOM drivers, the two drivers (native/ascom) internally uses two different offsets causing the calibration issue. If you always used one single software with the same camera configuration then I think this is not an issue.

Flats ADU values could also play a role depending on the camera, if you stay within the linearity range of the camera then it's ok, if ADUs are too high and your camera shows a non-linear response in that range then you could have an issue. You can check if your flat's ADU min/max value are more or less the same as the flats that worked in the past. Honestly, this is a theoretical consideration that I never saw reported in practice so do this check but I don't think the non-linearity is the main responsible of your issue.

WBPP pedestal plays no role on flats, it's simply not used since a flat after the bias or dark subtraction never contain zero or negative values.

my suggestion:
There is an empyrical way to change your masterFlat in order to work, this makes sense under the assumption that the calibration fails because of an erroneous additive term that is subtracted during the calibration of flat frames (which is what I think it's happening by looking at the images you've attached). You can proceed this way:
  1. open a single uncalibrated light frame (name it "L") and the master dark (name it "D")
  2. open your master flat (name it "F")
  3. use the calibration formula in pixel math (L-D)/(F+xxxx)*avg(F) where xxxx is a numeric value that could be negative or positive between 0 and 1 (it should be very low anyway).
  4. find a value for xxx that generates a nicely calibrated light frame without any visible vignetting pattern
  5. use the pixel math formula "F+xxxx" (where xxxx is the found value) to adjust the offset of your masterFlat and save it as a new corrected masterFlat
  6. use the corrected master Flat in WBPP, this should generate a better master light free form the vignetting issue
  7. if the vignette is still visible then you can fix it with ABE/DBE or go back to point 4 and find a better additive xxxx value
As said, this works if the masterFlat's calibration issue is due to a wrong additive calibration term.

Hope his helps,
Robyx
 
Thank you again!
I had tried this method after scouring the internet looking for solutions and came across Adam Block's tutorial about adjusting flats. I tried numbers like he was using for xxxx (such as .2 or .3). It didn't work with these values or anywhere around these numbers. Today, I went crazy and started to use numbers like 6 or7......IT Worked! So that tells me my master Flat is way off (too dark)when it comes to adjusting the light frames. The original light frames were taken with an Optolong L pro LP filter in place and a 60%moon. I had used the flats calibration wizard in SGP to get me to around 22000ADU for my flat exposure (full well on this camera is about 51000e). I use a tracing panel that fits over my dew shield. It illuminates evenly.It has always worked before?? Do you have any suggestion regarding further flat
 
Sorry prior message was almost complete.
Do you have any recommendations for taking future flats? It's concerning that this was so far off base when for over two years I have had no issues. I will make sure all my parameters are equal for Bias(Flat Darks), Darks, Flats and Lights. The flats wizard seems to have done a great job in the past.?.
 
Thank you again!
I had tried this method after scouring the internet looking for solutions and came across Adam Block's tutorial about adjusting flats. I tried numbers like he was using for xxxx (such as .2 or .3). It didn't work with these values or anywhere around these numbers. Today, I went crazy and started to use numbers like 6 or7......IT Worked! So that tells me my master Flat is way off (too dark)when it comes to adjusting the light frames.
That's really weird, 6,7 is a non-sense out of range value! it's like adding 6/7 times the maximum ADU value of your camera :) you would never get to these values with proper calibration. but it's hard to tell why it works without having the images to inspect.


Do you have any recommendations for taking future flats? It's concerning that this was so far off base when for over two years I have had no issues.
My recommendation is simply to follow the procedure that worked in the past, note down everything regarding its configuration and repeat it consistently. Telescope position, panel orientation and intensity, flat durations per filter, sensor temperature, camera configuration... everything. Moreover, use dark flats and not bias frames if you want a more robust and repeateble procedure to calibrate your flats, unless you really know how your sensor responds or you've measured your sensor response directly on your own and you know that it's linear on exposure time.

If something does not work then investigate why it happened, find what was different and correlate it to the result. That's my simple empirical suggestion.

Regarding my suggestion on using flat darks: take into account that if you changed the panel intensity (for any reason) the flat exposure changed and for a very short exposure time the response of some CMOS sensors could hold some surprises. If this is the case (exposures changed significantly with respect to the working flats) I would try also to generate a master dark flat with the exact exposure of your flats and use it. If you find that this solves the issue then try to tune the panel such that the flats exposure remains more or less the same and for enough seconds to stay far from that non-linear region, finally always take the corresponding dark flats instead of using bias frames if needed (do the linearity test on the exposure).

Robyx
 
Thanks Robyx!
Just so you know, My Bias frames are actually Master Flat Darks, taken at the same exposure length and temperature as the Flats. I have a library of Master Flat Darks (or master bias in my book) that match flats time for various filters (L-enhance, L-pro, none). I choose the appropriate master flat dark (appropriately timed master bias) and place it in the bias tab of WBPP. I place my master dark in the master dark tab of WBPP. When I check diagnostics, prior to running WBPP, I get a message saying "master bias will be used to calibrate flats since there is no appropriate master dark of the same exposure length". So I think , great! My Master Bias, (which is a master flat dark of the same time duration as my flat) will be used to calibrate my flats. I could always change the name of the master bias to master flat dark but the way I see it, the flats will be calibrated with the same master flat dark, regardless of what I call it.
 
Thanks Robyx!
Just so you know, My Bias frames are actually Master Flat Darks, taken at the same exposure length and temperature as the Flats. I have a library of Master Flat Darks (or master bias in my book) that match flats time for various filters (L-enhance, L-pro, none). I choose the appropriate master flat dark (appropriately timed master bias) and place it in the bias tab of WBPP. I place my master dark in the master dark tab of WBPP. When I check diagnostics, prior to running WBPP, I get a message saying "master bias will be used to calibrate flats since there is no appropriate master dark of the same exposure length". So I think , great! My Master Bias, (which is a master flat dark of the same time duration as my flat) will be used to calibrate my flats. I could always change the name of the master bias to master flat dark but the way I see it, the flats will be calibrated with the same master flat dark, regardless of what I call it.
I see. Pay attention only to one fact: master bias is matched on binning only, so duration plays no role in matching master bias. Don't expect to add two master bias of 1 and 2 seconds and have a proper exposure match for them, the first found will be selected disregarding the exposure. If you want the best match based on the duration then add your masterBias as masterDarks in the dark panel.

Robyx
 
Excellent. Note made. I will change the name of my master bias files to master flat darks and avoid any confusion (all my binning is 1x1). Thank you.
Now, I look forward to my next night out to see if I can get back to flats which calibrate my lights without issue. Wish me luck.

Dan
 
Well I was able to image last night. I obtained 300sec light frames of the pelican nebula. Immediately after the session I took flat frames of 0.11 sec. I used 300 sec master dark to calibrate the light frame and a 0.1sec master flat dark to calibrate the flat frame. All my camera setting for flats and lights are identical. I get a warning in WBPP that says:
Warning [156]: C:/Program Files/Pixinsight/src/scripts/Weighted Batch Preprocessing/Weighted Batch Preprocessing-engine.js, line 4996: assignment to undeclared variable activeFrames
The end result is the same as before....a very washed out calibrated/debayed light frame. The original light frame looks way better than the calibrated light frame.
Any more ideas?
Thank you,
Dan
 

Attachments

  • Screenshot (19).png
    Screenshot (19).png
    852.1 KB · Views: 45
  • Screenshot (20).png
    Screenshot (20).png
    1,013.3 KB · Views: 38
When I check under Programs and select Pixinsight...
Does the error message mean anything to you?
Thanks, Dan
 

Attachments

  • Screenshot (23).png
    Screenshot (23).png
    318 KB · Views: 49
  • Screenshot (24).png
    Screenshot (24).png
    297 KB · Views: 38
  • Screenshot (25).png
    Screenshot (25).png
    291.6 KB · Views: 52
  • Screenshot (26).png
    Screenshot (26).png
    332.5 KB · Views: 36
  • Screenshot (27).png
    Screenshot (27).png
    308 KB · Views: 45
  • Screenshot (28).png
    Screenshot (28).png
    334 KB · Views: 39
Hi Dan,

The end result is the same as before....a very washed out calibrated/debayed light frame. The original light frame looks way better than the calibrated light frame.
Any more ideas?
Please post mean and median values of the MasterFlatDark and the MasterDark that were used in the last WBPP run. The screenshots of the masters are not helpful for the judgement of the bias level.

Bernd
 
Back
Top