Image calibration issue

[DaveB]

I've seen a smattering of other folks reporting similar issues, so I'll add my data in case it helps resolve the issue (or educate me as to what I'm doing wrong). This is basically a copy of a reply that I made to someone who was seeing similar issues in the General forum.

This issue occurred on about 1/3 of my Ha frames for a particular target, but I will narrow it down to a sequence of four 10-minute Ha frames that I took consecutively. When I look at the raw frames, they look pretty much identical except for an occasional satellite or cosmic ray hit. However, after I run them through the image integration process, three out of the four have their signal nearly eliminated. After running the calibration script, images 1, 2, and 4 in the sequence look nearly black (even after STF), while image 3 looks similar to the raw frame. I tried eliminating the flat frame from the integration, but that didn't change anything. I also tried changing the noise evaluation algorithm, but that didn't matter either.

To see it visually, I've captured a screenshot showing two raw frames on the bottom row (image 3 & 4 in the sequence) along with the corresponding calibrated frames above them. STF was applied to all four images.

The four raw frames, four calibrated frames, and master dark/bias/flat can be found here: https://www.dropbox.com/sh/jeg7g5r14yuvbds/AAB8C5jOvCZn8fBsoZSTCcNAa?dl=0

I've attached a csv file from a run with a BatchStatistics script against the four raw frames. The data looks nearly identical between all four images, at least from what I can see.

Below is the console output from the integration process for the four frames. Note the significant differences in k0, s0, and n0 values between image 3 (the good image after calibration) vs. the other three images:

Writing output file: C:/Users/Kathy/Desktop/PI_cache/IC5146/Cal2/IC5146_Ha_600sec_1x1_0C_20170823_0005_c.xisf
Dark scaling factors:
k0 = 0.545
Gaussian noise estimates:
s0 = 5.145e-10, n0 = 0.857 (K-Sigma)
Writing image: w=3326 h=2504 n=1 Gray Float32
70 FITS keyword(s) embedded.

Writing output file: C:/.../IC5146/Cal2/IC5146_Ha_600sec_1x1_0C_20170911_0001_c.xisf
Dark scaling factors:
k0 = 0.550
Gaussian noise estimates:
s0 = 8.517e-09, n0 = 0.821 (K-Sigma)
Writing image: w=3326 h=2504 n=1 Gray Float32
70 FITS keyword(s) embedded.

Writing output file: C:/.../IC5146/Cal2/IC5146_Ha_600sec_1x1_0C_20170911_0003_c.xisf
Dark scaling factors:
k0 = 0.158
Gaussian noise estimates:
s0 = 3.018e-04, n0 = 0.258 (MRS)
Writing image: w=3326 h=2504 n=1 Gray Float32
70 FITS keyword(s) embedded.

Writing output file: C:/.../IC5146/Cal2/IC5146_Ha_600sec_1x1_0C_20170911_0002_c.xisf
Dark scaling factors:
k0 = 0.552
Gaussian noise estimates:
s0 = 2.247e-08, n0 = 0.807 (K-Sigma)
Writing image: w=3326 h=2504 n=1 Gray Float32
70 FITS keyword(s) embedded.

Writing output file: C:/Users/.../IC5146/Cal2/IC5146_Ha_600sec_1x1_0C_20170911_0004_c.xisf
Dark scaling factors:
k0 = 0.559
Gaussian noise estimates:
s0 = 9.518e-09, n0 = 0.820 (K-Sigma)
Writing image: w=3326 h=2504 n=1 Gray Float32
70 FITS keyword(s) embedded.

If I can provide any additional data that might help, let me know.

Thanks!

[sharkmelley]

Thanks for the files you posted.  This made it very easy to reproduce and diagnose the problem.

In the ImageCalibration process you need to tick the Calibrate option in the Master Dark section.  The effect of this is to subtract the master bias from the master dark before dark optimization takes place.  The dark noise is the difference between the master dark and the master bias and it is only this dark noise that needs to be scaled and subtracted from the lights.  Otherwise the light has the master bias subtracted and a proportion of the bias in the master dark.  That's why the calibrated lights ended up dark and clipped to zero - too much was being subtracted.

As an added bonus, the dark optimization will perform much better and more effectively remove the dark noise.

Here's the Process Console output from the run with the ticked Calibrate option - notice how the Dark scaling factor is now pretty close to 1.0 where previously it was 0.55 or 0.15

Writing output file: D:/MyDownloads/PI_Calibration_Issue/calibrated/IC5146_Ha_600sec_1x1_0C_20170911_0001_c.xisf
Dark scaling factors:
k0 = 1.017
Gaussian noise estimates:
s0 = 3.316e-04, n0 = 0.846 (MRS)
Writing image: w=3326 h=2504 n=1 Gray Float32
70 FITS keyword(s) embedded.

Writing output file: D:/MyDownloads/PI_Calibration_Issue/calibrated/IC5146_Ha_600sec_1x1_0C_20170911_0002_c.xisf
Dark scaling factors:
k0 = 1.011
Gaussian noise estimates:
s0 = 3.483e-04, n0 = 0.872 (MRS)
Writing image: w=3326 h=2504 n=1 Gray Float32
70 FITS keyword(s) embedded.

Writing output file: D:/MyDownloads/PI_Calibration_Issue/calibrated/IC5146_Ha_600sec_1x1_0C_20170911_0003_c.xisf
Dark scaling factors:
k0 = 1.016
Gaussian noise estimates:
s0 = 3.479e-04, n0 = 0.869 (MRS)
Writing image: w=3326 h=2504 n=1 Gray Float32
70 FITS keyword(s) embedded.

Writing output file: D:/MyDownloads/PI_Calibration_Issue/calibrated/IC5146_Ha_600sec_1x1_0C_20170911_0004_c.xisf
Dark scaling factors:
k0 = 1.020
Gaussian noise estimates:
s0 = 3.392e-04, n0 = 0.861 (MRS)
Writing image: w=3326 h=2504 n=1 Gray Float32
70 FITS keyword(s) embedded.


Mark

[DaveB]

Awesome! Thanks for the quick turnaround! I post an issue before dinner, and have the solution after dinner 

EDIT: I reread the appropriate section in Keller's book, and he says to do the same thing that you do, which is to check Calibrate (and Optimize) for the master dark. I missed that point when setting up my workflow, which was intended to follow his workflow for the most part. Sorry for the pilot error.

[jamesRC]

people,

Thank you very much for the help on this problem. I just discovered these replies and I'm about to try the solution (tick "calibrate" under masterdark) now.

Cheers!

James