WBPP-Problem with image calibration

[dchamberlain]

I've run into a strange situation that only has happened on processing subframes for M42. But I have isolated the occurrence to the Weighted Batch PreProcessing script. I am using version 1.4.4 of the script.

I started out with 200 subframes of M42 with my OSC ZWO ASI1600MC-Pro at 5 seconds each. I ran them through WBPP to completion and the master stacked image was blotchy, dark (when autostretched) and just illegible. At first I thought the number of subframes might have something to do with it, so I pared it down to 50 and got the same result. The dark and flat frames looked fine to me. I eventually ran it through without calibrating with darks or flats and the result looked acceptable.

Then in an attempt to isolate it further, I took the same master dark, master flat and 50 light frames and calibrated with the ImageCalibration process and the resulting calibrated frames looked wonderful. So it looks as if the WBPP ImageCalibration is doing something different than the ImageCalibration process.

Here is a link to the raw subframe, master flat, master dark, WBPP calibrated subframe and the IC calibrated subframe: https://www.dropbox.com/sh/b42894qrm6se4wy/AAA0K_F15ybdDZR0v9g2ODBea?dl=0

Or at least if someone can tell me what I doing wrong?

Thanks

Dale

[bulrichl]

Hi Dale,

the MasterDark has mean = 842.3, median = 842.8. The uncalibrated light subframe has mean = 345.1, median = 336. This cannot not work at all. The darks don't match the lights. Did you use the same acquisition software for capturing  light and dark frames?

According to the FITS headers, both the dark frames and the light subframe were exposed for 5 s, and EGAIN was 1.00111 (which corresponds to gain setting of 139 for the ASI1600). I also can imagine that different offset settings were used for the acquisition of light and dark frames. You'll need to repeat the dark frames with the same acquisition software and the same offset setting as used for the light frames.

However, there are further issues. The manually (with ImageCalibration) calibrated subframes may look wonderful to you, but they are not. In this calibration, the (faulty) MasterDark and MasterFlat were used, in the 'Master Dark' section the option 'Calibrate' was disabled and the option 'Optimize' was enabled. Thus dark frame optimization was applied, the Dark scaling factor was 0.063. These settings are wrong and yield an incorrect calibration result.

When you got matching dark frames, please stick to manual calibration with the ImageCalibration process for now and take a look at https://pixinsight.com/forum/index.php?topic=14663 , reply #9 for the recommended approach and settings for pre-processing.

Bernd

[dchamberlain]

Hi Bernd,

My image acquisition software is Sequence Generator Pro which I used for all the lights, flats and darks. I set my gain to 139 in the software and offset to 21 using the Ascom driver "unity" setting for the ASI1600MC. These settings are the same for other objects I've imaged before and after with much different results.

I will take a look at the link you suggested.

The only other difference on this imaging session is I used an Optolong LeHance filter. Could that affect the numbers you stated?

Dale

[bulrichl]

It is very strange that the mean of the MasterDark is considerably higher than the mean of the light subframe. The filter is irrelevant in this regard. It doesn't look like light leakage to me either. The most likely explanation for your problem is different offset setting.

In this thread https://www.cloudynights.com/topic/672344-gain-and-sub-exposure-calculation-spreadsheet-for-the-zwo-asi183-and-294/page-2 #42, offset calculation is explained. For the Panasonic MN34230, a value of (offset setting * 16 ADU) is added during A/D conversion. The factor of 16 is the scaling factor for the scaling from 12 bit (intensity range from 0 to 4096) to 16 bit (intensity range from 0 to 65535) that the camera driver applies.

It is very likely that for the acquisition of the light frames, an offset of 21 was used, leading to a median of the light subframe of 336. However, I assume that for the dark frames an offset of about 50 was set. So my suggestion is: take new dark frames making sure that the offset is 21 this time. I guess these new dark frames will match your light frames then. Check the new dark frames with histogram and Statistics in PixInsight. If the mean / median suit to the values of the light frames, make a new MasterDark and please follow the recommendation in the last paragraph of my preceding post.

Bernd

[dchamberlain]

Hi Bernd,

I took a look at the individual dark frames that made up the master dark. Keep in mind the master dark was produced through WBPP.

The mean of the individual darks range from 332 to 340.6 which is closer to the lights. No where near the master dark.

I ran the very same individual dark frames through ImageIntegration and produced a new master dark with a mean of 334.29. Looks like something went wrong during the creation of the master dark in WBPP.

Dale

[bulrichl]

Could you please upload the new MasterDark and the new integration (following my suggestion) as well? I would love to take a look at it.

Bernd

[dchamberlain]

Could you please upload the new MasterDark and the new integration (following my suggestion) as well? I would love to take a look at it.

Bernd

Hi Bernd,

Here is where you can find the new MasterDark, new MasterFlat and the integration. I used all the original raw files that were used in the WBPP. The only difference is these were done manually without WBPP. I think this proves there was something quite different between the WBPP and the manual runs.

Here is the link: https://www.dropbox.com/sh/tct5pucogji59m5/AAAsRRl9mU_P_zHxe9AHW-_fa?dl=0

Thanks,

Dale

[bulrichl]

Thank you for uploading the new files. Currently I am sitting at my notebook for the acquisition of images and have no access to your old data. I will take a close look tomorrow and report.

Definitely the new MasterDark has a mean = 334.3, median = 334.4, and the new integration is barely clipped (count % R: 99.84, G: 99.85, B: 99.86), so the new MasterDark matches the light frames pretty well, and the calibration procedure was successfull. The peaks in the histogram of the integration using the new calibration result look very clean. In the image I see a small brightening at the upper right corner. Otherwise the  integration using the new calibration result looks very well to me, I guess it is a great result for 50 x 5 s = 4 min 10 s.

More about it tomorrow. I am particularly curious about the camparison with the integration you obtained after manual calibration with the old Master calibration files, but wrong settings for the ImageCalibration, and of course about the reason why WBPP produced an unusable MasterDark.

Bernd

[dchamberlain]

Thank you for uploading the new files. Currently I am sitting at my notebook for the acquisition of images and have no access to your old data. I will take a close look tomorrow and report.

Definitely the new MasterDark has a mean = 334.3, median = 334.4, and the new integration is barely clipped (count % R: 99.84, G: 99.85, B: 99.86), so the new MasterDark matches the light frames pretty well, and the calibration procedure was successfull. The peaks in the histogram of the integration using the new calibration result look very clean. In the image I see a small brightening at the upper right corner. Otherwise the  integration using the new calibration result looks very well to me, I guess it is a great result for 50 x 5 s = 4 min 10 s.

More about it tomorrow. I am particularly curious about the camparison with the integration you obtained after manual calibration with the old Master calibration files, but wrong settings for the ImageCalibration, and of course about the reason why WBPP produced an unusable MasterDark.

Bernd

Hi Bernd,

Let me post a fresh link to the old data: https://www.dropbox.com/sh/b42894qrm6se4wy/AAA0K_F15ybdDZR0v9g2ODBea?dl=0
It has more data than before at the request of Roberto who is looking into WBPP.

Thanks for taking a look at this. I don't know why WBPP gave me different results but I am running WBPP against the full set of lights using the master darks/flats I created using the manual method. I would expect the results to be entirely different than before. It will take the rest of today to finish.

Dale

[bulrichl]

Up to now I always used manual preprocessing with PixInsight's individual processes. So this was a premiere for me: I tried WBPP with a project of mine (camera: ZWO ASI071MC Pro) and used the individual calibration frames (not using finished master calibration frames) for this test.

I seleceted no frames in the section 'Bias', 110 dark frames and 100 flat-darks in the section 'Darks', 40 flats in the section 'Flats' and (for this test) only 5 light frames in the section 'Lights'. 'Calibrate with flat darks' was enabled in the tab 'Flats'. 'Calibrate only' was enabled in the tab 'Lights'. In 'Global Options', 'CFA images' was enabled and 'Optimize dark frames' was disabled (see appended screen section "WBPP_settings.JPG"). Clicking in 'Diagnostics' resulted in the warning: "** Warning: No bias frames have been selected.". This warning was ignored: the process was started and 'Continue' confirmed when the warning appeared again.

WBPP integrated the dark frames to the MasterDark and the flat-darks to the MasterFlat-Dark, calibrated the flat frames with the MasterFlat-Dark and integrated the calibrated flat frames. Then the 5 light frames were calibrated with the MasterDark and the MasterFlat, not using dark frame optimization. Finally the calibrated light frames were debayered. The whole procedure took 9 min 10.2 s. Smart report:

************************************************************
Group of 110 Dark frames
BINNING  : 1
Exposure : 300.00s

Rejection method auto-selected: Generalized Extreme Studentized Deviate
Integration completed: master file C:/Users/Bulrich/Downloads/test1/master/masterDark-BINNING_1-EXPTIME_300.xisf
************************************************************

************************************************************
Group of 100 Dark frames
BINNING  : 1
Exposure : 0.01s

Rejection method auto-selected: Generalized Extreme Studentized Deviate
Integration completed: master file C:/Users/Bulrich/Downloads/test1/master/masterDark-BINNING_1-EXPTIME_0.0055.xisf
************************************************************

************************************************************
Group of 40 Flat frames
BINNING  : 1
NoFilter
Exposure : 0.01s

[Calibration data]
  Master bias: none
  Master dark: C:/Users/Bulrich/Downloads/test1/master/masterDark-BINNING_1-EXPTIME_0.0055.xisf
  Master flat: none
Calibration completed
Rejection method auto-selected: Generalized Extreme Studentized Deviate
Integration completed: master file C:/Users/Bulrich/Downloads/test1/master/masterFlat-BINNING_1-FILTER_NoFilter.xisf
************************************************************

************************************************************
Group of 5 Light frames
BINNING  : 1
NoFilter
Exposure : 300.00s

[Calibration data]
  Master bias: none
  Master dark: C:/Users/Bulrich/Downloads/test1/master/masterDark-BINNING_1-EXPTIME_300.xisf
  Master flat: C:/Users/Bulrich/Downloads/test1/master/masterFlat-BINNING_1-FILTER_NoFilter.xisf
Calibration completed
Debayer completed
************************************************************

The master calibration files and the calibrated and debayered light frames were virtually identical with the files previously obtained by manually performing the processes. The minuscule differences resulted from different pixel rejection settings for the integration of the dark frames and flat-darks.

In summary: WBPP performed the preparation of the master calibration files and the light calibration flawlessly, the calibration result is correct. The only criticism I have is that the warning "** Warning: No bias frames have been selected." is out of place with my calibration approach (not pre-calibrated MasterDark, no use of bias frames or a MasterBias and no application of dark frame optimization).

So I am unsure why you obtained an unusable MasterDark from WBPP initially. I can only imagine that your settings in WBPP were not appropriate. Please closely compare your settings with mine.

Also very important: if you used the "masterDark-BINNING_1-EXPTIME_0.35.xisf" (mean = 781.7, median = 781.6; so here the same issue as with the MasterDark applies!) for the calibration of the flat frames, this would produce an incorrect MasterFlat which would result in an incorrect flat field correction. So the whole procedure for the preparation of the MasterFlat has to be repeated as well.

Bernd

[dchamberlain]

Up to now I always used manual preprocessing with PixInsight's individual processes. So this was a premiere for me: I tried WBPP with a project of mine (camera: ZWO ASI071MC Pro) and used the individual calibration frames (not using finished master calibration frames) for this test.

I seleceted no frames in the section 'Bias', 110 dark frames and 100 flat-darks in the section 'Darks', 40 flats in the section 'Flats' and (for this test) only 5 light frames in the section 'Lights'. 'Calibrate with flat darks' was enabled in the tab 'Flats'. 'Calibrate only' was enabled in the tab 'Lights'. In 'Global Options', 'CFA images' was enabled and 'Optimize dark frames' was disabled (see appended screen section "WBPP_settings.JPG"). Clicking in 'Diagnostics' resulted in the warning: "** Warning: No bias frames have been selected.". This warning was ignored: the process was started and 'Continue' confirmed when the warning appeared again.

WBPP integrated the dark frames to the MasterDark and the flat-darks to the MasterFlat-Dark, calibrated the flat frames with the MasterFlat-Dark and integrated the calibrated flat frames. Then the 5 light frames were calibrated with the MasterDark and the MasterFlat, not using dark frame optimization. Finally the calibrated light frames were debayered. The whole procedure took 9 min 10.2 s. Smart report:

************************************************************
Group of 110 Dark frames
BINNING  : 1
Exposure : 300.00s

Rejection method auto-selected: Generalized Extreme Studentized Deviate
Integration completed: master file C:/Users/Bulrich/Downloads/test1/master/masterDark-BINNING_1-EXPTIME_300.xisf
************************************************************

************************************************************
Group of 100 Dark frames
BINNING  : 1
Exposure : 0.01s

Rejection method auto-selected: Generalized Extreme Studentized Deviate
Integration completed: master file C:/Users/Bulrich/Downloads/test1/master/masterDark-BINNING_1-EXPTIME_0.0055.xisf
************************************************************

************************************************************
Group of 40 Flat frames
BINNING  : 1
NoFilter
Exposure : 0.01s

[Calibration data]
  Master bias: none
  Master dark: C:/Users/Bulrich/Downloads/test1/master/masterDark-BINNING_1-EXPTIME_0.0055.xisf
  Master flat: none
Calibration completed
Rejection method auto-selected: Generalized Extreme Studentized Deviate
Integration completed: master file C:/Users/Bulrich/Downloads/test1/master/masterFlat-BINNING_1-FILTER_NoFilter.xisf
************************************************************

************************************************************
Group of 5 Light frames
BINNING  : 1
NoFilter
Exposure : 300.00s

[Calibration data]
  Master bias: none
  Master dark: C:/Users/Bulrich/Downloads/test1/master/masterDark-BINNING_1-EXPTIME_300.xisf
  Master flat: C:/Users/Bulrich/Downloads/test1/master/masterFlat-BINNING_1-FILTER_NoFilter.xisf
Calibration completed
Debayer completed
************************************************************

The master calibration files and the calibrated and debayered light frames were virtually identical with the files previously obtained by manually performing the processes. The minuscule differences resulted from different pixel rejection settings for the integration of the dark frames and flat-darks.

In summary: WBPP performed the preparation of the master calibration files and the light calibration flawlessly, the calibration result is correct. The only criticism I have is that the warning "** Warning: No bias frames have been selected." is out of place with my calibration approach (not pre-calibrated MasterDark, no use of bias frames or a MasterBias and no application of dark frame optimization).

So I am unsure why you obtained an unusable MasterDark from WBPP initially. I can only imagine that your settings in WBPP were not appropriate. Please closely compare your settings with mine.

Also very important: if you used the "masterDark-BINNING_1-EXPTIME_0.35.xisf" (mean = 781.7, median = 781.6; so here the same issue as with the MasterDark applies!) for the calibration of the flat frames, this would produce an incorrect MasterFlat which would result in an incorrect flat field correction. So the whole procedure for the preparation of the MasterFlat has to be repeated as well.

Bernd

Thanks Bernd,

I've had a growing suspicion that my installation of PixInsight might have been damaged at one point so I have downloaded the latest 1.8.8-4 version, uninstalling the one I have and reinstalling.

The other thing, the darks I had been working with seemed to have quite a bit of variation in brightness for the same exposure time. I'm not sure what caused this, I've never had this happen before. Maybe the sensor temperature wasn't where I thought it was. In any event, I am starting over with a new, more consistent set of darks to rebuilt the masters from. BTW, I'm not sure how meaningful or accurate it is, but I used Blink to look at the darks and that is what shows me the flashing of brightness when I run through the frames. My understanding is they should be fairly consistent.

I will post my results when they are ready.

Thanks again,

Dale

[bulrichl]

You are right, the brightness of the dark frames should not vary by much. In order to check a large set of dark frames, it will be best to use Blink and its function 'Series analysis report': you load the dark frames in Blink and click on the column chart symbol. Check 'Write text file', choose an output folder and then click 'OK'. Blink will write the file "Statistics.txt" to the specified folder which contains the statistics of the files loaded in Blink.

Bernd

[dchamberlain]

You are right, the brightness of the dark frames should not vary by much. In order to check a large set of dark frames, it will be best to use Blink and its function 'Series analysis report': you load the dark frames in Blink and click on the column chart symbol. Check 'Write text file', choose an output folder and then click 'OK'. Blink will write the file "Statistics.txt" to the specified folder which contains the statistics of the files loaded in Blink.

Bernd

Hi Bernd,

You have helped me quite a lot in determining the cause of my image processing problems. I took new darks for each exposure group (5 sec, 10 sec, 15 sec and 30 sec, including the dark flats). This time I used TheSkyX Pro for capturing the flats. After examining all of them, the frames were uniform and the mean was in the 320s for each, with the exception of the flat darks which were 313. I took them all through WBPP and it completed successfully with integrated images for each exposure. This was a first on this project.

I have attached the resulting image. I am still uncertain as to what caused the problems with the previous darks. In the past I used TheSkyX Pro for image capture, but recently have been using SGP. A recent update to SGP announced they weren't supporting the ZWO camera driver any longer and that I had to switch to the ASCOM driver. I have suspicions something may be wrong with the ASCOM driver. For one thing, you can only set gain and offset in the ASCOM interface, not in SGP. But it doesn't seem to affect the lights frames.

I've ruled out temperature changes of the sensor as the darks all have the same temperature reading.  Not sure what else to look at. Any insights would be helpful.

Thanks again,

Dale

[bulrichl]

I am using SGP for capturing with the ASCOM driver for my ZWO ASI cameras. There is nothing wrong with the ASCOM driver. However, it is generally not a good idea to use either different acquisition software or different camera drivers for light frames and calibration frames of one project.

Bernd

[dchamberlain]

I am using SGP for capturing with the ASCOM driver for my ZWO ASI cameras. There is nothing wrong with the ASCOM driver. However, it is generally not a good idea to use either different acquisition software or different camera drivers for light frames and calibration frames of one project.

Bernd

Well then I'm out of answers as to what happened.  All I know is I took 2 sets of darks using SGP and the ASCOM driver for my ZWO ASI1600MC, and problems developed.  But the ones taken with TheSkyX Pro allowed the WBPP script to complete without error.

Dale