ImageCalibration error calibrating master dark with master bias during flat IC

[Ara Jerahian]

I'm running the latest PI (PixInsight Core 01.08.06.1475 Ripley (x64)).  I'm doing a manual calibration of my flats using ImageCalibration.  I have loaded 50 flat frames, selected an output directory, checked Evaluate Noise, selected a Master Bias (Calibrate unchecked) and a Master Dark (Calibrate and Optimize both checked), Optimization threshold of 3.0000.  I've done this exact thing many times before with previous versions of PI.

However, now when I click Apply Global, I get the following error message in the console:

Code: [Select]

Applying bias correction: master dark frame ...

Dark frame optimization thresholds:
Td0 = 0.00000000 (2 px = 0.000%)
** Warning: The dark frame optimization threshold is probably too high (channel 0).
** Warning: The dark frame optimization pixel set is too small - disabling dark frame optimization (channel 0).
*** PCL Win32 System Exception: At address 00007FFCD304A388 with exception code C0000005 :
Access violation: invalid memory read operation at address 0000000000000008
<* failed *>
With the options set as described above, if I uncheck the Master Bias section, it runs fine.
With the options set as described above, if I only uncheck Calibrate for my Master Dark, it runs fine.
With the options set as described above, if I only uncheck Optimize for my Master Dark, it runs fine.

So, this issue has something to do with the calibration or optimization of the master dark with the master bias.

I've checked both my master bias and master dark files and they seem fine (load in PI without any issues).  Both the master bias and master dark are XISF format.

Is this a known issue or is anyone else dealing with this?

-Ara

[Juan Conejero]

Hi Ara,

Each time we release a new version, there is invariably a strong tendency to consider the new version as the culprit of all problems on earth... 

Nothing has changed in the ImageCalibration tool in the latest versions of PixInsight. The first warning message gives you all the information you need to diagnose the problem:

** Warning: The dark frame optimization threshold is probably too high (channel 0).

Try lowering the optimization threshold parameter, or disabling dark frame optimization.

On the other hand, the last access violation should obviously not happen, so it may denote a small bug that I have to investigate. Thanks for reporting.

[Ara Jerahian]

Thanks Juan.  I've been in software development for 3 decades now, so I know exactly what you mean.  My apologies, I wasn't trying to blame the latest version, but only trying to isolate the issue for you and team to look in to.

That said, I should have included in my bug report, based on the warning message, I indeed tried all values for the Optimization Threshold, from 0 to 10, but all still resulted in the same error.

Regards,
-Ara

[Ara Jerahian]

One more thing...I can reproduce this error with a minimum of 3 files:  1) flat fit, 2) master bias xisf, 3) master dark xisf.

Let me know if you would like me to provide those files to you for easy reproduction and debugging, in case there is any chance it is file related or if you cannot reproduce.

Regards,
-Ara

P.S.  I love PI.  Thanks for the hard work you guys are doing!!

[Juan Conejero]

Hi Ara,

Thank you so much for your appreciation of PixInsight! Yes, of course if you can upload those files I'll take a look to see what happens. Thank you in advance for that.

[Ara Jerahian]

Great, thanks Juan.  I have placed the 3 files in my dropbox.  Here are the links (they are obviously fairly large files):

Flat_0.03sec_1x1_gain_139_R_frame1.fit
https://www.dropbox.com/s/r66qoep3840ss6b/Flat_0.03sec_1x1_gain_139_R_frame1.fit?dl=0

Bias_1x1_g139_MASTERv2.xisf
https://www.dropbox.com/s/b03eorurx2uvq1z/Bias_1x1_g139_MASTERv2.xisf?dl=0

_Dark_03s_g139_MASTER.xisf
https://www.dropbox.com/s/uonu9810gqq39nt/_Dark_03s_g139_MASTER.xisf?dl=0

I have also attached the screenshot of the ImageCalibration settings as they are when producing the error.

Regards,
Ara

[bulrichl]

Hi,

Same observation here regarding warning and System Exception. But...

These are the statistics of your MasterBias and Master Dark:

_Dark_03s_g139_MASTER       Bias_1x1_g139_MASTERv2
count (%)   100.00000       count (%)   100.00000
count (px)  16389120        count (px)  16389120
mean        117.748         mean        285.114
median      118.400         median      285.120
variance    49.716          variance    73.842
stdDev      7.051           stdDev      8.593
avgDev      6.719           avgDev      8.472
MAD         6.168           MAD         8.540
minimum     43.102          minimum     193.488
maximum     297.920         maximum     496.640

Mean of MD is 117.7 and mean of MB is 285.1. This doesn't make sense at all - except if you already pre-calibrated the MD with the MB. If this is the case, you have to alter the setting in the calibration window and disable 'Calibrate' in the 'Master Dark' section. Performing the calibration with this setting yields no warning, no System Exception and a correlation coefficient k0 of 0.919.

However, I suggest you NOT to pre-calibrate the MD. In this case, your settings for the calibration would be just fine.

This finding also means that indeed an error in the program flow occurs that is not trapped properly.

Bernd

[jpaana]

Some cameras with Sony sensors, like ASI178MC I have, perform some kind of automatic dark "correction" which is over correcting for dark current and causes longer dark exposures to actually have lower ADU values than bias or shorter darks. I has a discussion with ZWO support about it at https://bbs.astronomy-imaging-camera.com/viewtopic.php?f=29&t=8426 and the end result is that there is no fix for it currently and the calibration frames need to be exactly same length and not use bias or dark optimization. So now I take both "light" darks and "flat" darks and no bias at all.

[bulrichl]

The ZWO ASI1600MM does not contain a Sony sensor but a Panasonic MN34230ALJ sensor.

I cannot comment on your ASI178MC, but the ASI1600 does not behave like your ASI178MC, and my ASI294MC and ASI071MC do not either.

The finding above (mean of MasterDark: 117.7, mean of MasterBias: 285.1) for the ASI1600MM is definitely not attributable to any on-sensor or on-camera "correction". So this is a different story. If these values resulted for a NOT pre-calibrated MasterDark, the camera is defective or different gain/offset values were used for the acquisition of dark and bias frames.

Bernd

[Ara Jerahian]

The finding above (mean of MasterDark: 117.7, mean of MasterBias: 285.1) for the ASI1600MM is definitely not attributable to any on-sensor or on-camera "correction". So this is a different story. If these values resulted for a NOT pre-calibrated MasterDark, the camera is defective or different gain/offset values were used for the acquisition of dark and bias frames.

Bernd, thank you for your analysis.  The finding is strange indeed.  I now question what my offset may have been for the original 50 bias frames which integrated into that master.  I looked, but since I think I used the ASCOM driver for those bias frames, I couldn't set the offset value and it does not seem to have been persisted in the FITS header.

That said, based on your finding, I took some fresh bias frames, integrated them, and the new master dark mean and median values are much more reasonable.  Also, I have now calibrated my flats successfully.

Quite strange.  Once again, thank you for analyzing the files and finding the cause.

-Ara