Question about the "BASIC CCD Parameters" Script

[ruediger]

Attached is a new version of the Basic CCD Parameters script with the following major changes:

- CFA checkbox for working with CFA grayscale(!) images. Script does not work with color CFA images.
- Numeric input for "Readout bits" to scale normalized [0,1] range to ADU units
- Dark exposure time read from FITS Header EXPTIME, if available
- ISO and exposure time shown next to selected image, if FITS Header ISOSPEED and EXPTIME are set
- Region of interest can be defined by a preview. When the input is CFA gray, the user has to make sure on his own that the upper left corner does not lie within a bayer superpixel
- internal calculation of mean, stddev completely in javascript to avoid pitfalls with library calls, especially no problems with temporary negative values anymore
- Export images button removed, because in the current version no temporary images are created which could be exported.

Known problems:

- fullwell capacity is only computed from gain and A/D bit depth. It does not take into account the real maximum value the camera can deliver. For a 14 bit Canon, the real max. value could be 3000 ADU less than the theoretical maximum of 2^14 - 1

Odd things:

- negative dark currents, when mean( Dark - Bias1 ) is negative. The computation however is correct, but I don't know yet how to interpret the result. I also noticed a lot of "0"-values inside my master darks, all at the positions where "dark < bias" holds.

Rüdiger

[NKV]

- negative dark currents, when mean( Dark - Bias1 ) is negative. The computation however is correct, but I don't know yet how to interpret the result. I also noticed a lot of "0"-values inside my master darks, all at the positions where "dark < bias" holds.

Are you speak about Canon DSLR ?

AstroPhoto Insight 2009 April/May.
"One possible explanation for this behavior is that Canon is using the “optical black” (an area around the main image portion of the sensor) to read the dark current and scale the image intensity to bring this back in line."

[ruediger]

"One possible explanation for this behavior is that Canon is using the “optical black” (an area around the main image portion of the sensor) to read the dark current and scale the image intensity to bring this back in line."

This perfectly makes sense. Contrary to the article, I observe this behaviour also in my 5D II darks, i.e. the mean value of a dark file is in the same range as the bias mean value, but the stddev of course is much larger in the dark.

This has implications to the way we generate Canon master darks. Following the DSLR RAW workflow tutorial, we subtract the master bias from each single dark file, which leads to negative values for the pixels from the left side of the dark's histogram, which are then silently truncated to zero.

Rüdiger

[pfile]

so is there a way to add a negative pedestal value to the bias? or do we just have to do some pixelmath tricks on the bias before subtracting it from the dark?

does the 'black point correction' in the DSLR_RAW module have any effect on this?

[georg.viehoever]

...This has implications to the way we generate Canon master darks. Following the DSLR RAW workflow tutorial, we subtract the master bias from each single dark file, which leads to negative values for the pixels from the left side of the dark's histogram, which are then silently truncated to zero.
...

I did not know about this issue until now. According to the article, some Canon DSLRs do some processing on RAW files to adjust for the changes caused by dark current, and they show a significant drift in average dark values over time. Questions for me:
- how can this be handled in the script discussed here? Even if we would allow to enter some adjustment value: How would we arrive at this value?
- For the DSLR calibration workflow: How can it be handled there? Or is it automatically handled by Dark Optimization?
Georg

[ruediger]

I did not know about this issue until now. According to the article, some Canon DSLRs do some processing on RAW files to adjust for the changes caused by dark current, and they show a significant drift in average dark values over time. Questions for me:
- how can this be handled in the script discussed here? Even if we would allow to enter some adjustment value: How would we arrive at this value?

The authors in the article are speaking of an "intensity scale", but I think, the Camera internally measured dark noise is subtracted from the recorded light+dark values. This leaves only the increase in stdDev for measurement inside the script. Since the stdDev is invariant under addition/subtraction operations, I currently see no way to get the real dark noise. Maybe it's possible under an additional assumption, that dark noises doubles at fixed temperature intervals and distribution is gaussian type.

How does other DSLR (Nikon etc.) handle dark current? Some volunteer to run the script? Maybe other problems in the script arise because Nikon adds no bias, but I would like to see some results.

- For the DSLR calibration workflow: How can it be handled there? Or is it automatically handled by Dark Optimization?

This is the really interesting question. For the screenshot I used a Canon 5D II at 200 ISO with a short dark exposure (32s), just for personal interest. But for astrophotography purposes, I prefer to use a spectrum modified 500Da. For a typical 6min dark taken at ambient temperature of about 15°c (EXIF temperature: 29°C), the Image Statistics readout values are:
Mean   1025.8
StdDev    88.9
The dark Mean is nearly the same as the typical Bias mean (1025.1). The dark histogram, with the Mean value brought to 0.5, is slightly unbalanced towards the left, see screenshot. Most of the values from the left side get lost during master dark generation. This is not desirable, but I doubt this has impact on image quality, especially when working with dithering.

What worries me much more: If inside a lightframe some faint signal is hidden inside the dark noise, does it mean, some considerable amount of real signal get's lost once we subtract the master bias?

I'm setting up a test project right now to find out.... on one hand I hope to find a way to improve image quality  8), on the other hand I'd be happy to not start from the beginning on some projects  -- especially because I spent two weeks after purchasing the license to set up a working calibration, registration and stacking pipeline. 

Rüdiger

[georg.viehoever]

I have repeated the experiment shown in Fig 2 of the Craig Stark's article (measure how mean and noise in darks develop over time):
- I used 18 darks I shot with my CanonEOS40D a while ago, each 120 secs with 10 seconds breaks between them
- I used an R script to evaluate the FITS files (converted with PI)
- the first screenshot shows the data and a plot of sd (noise) vs time. The trend is clear and may indeed be caused by the heating of the chip.
- Screenshot 2 shows the mean value. The most significant effect here is an occasional jump by 1 ADU. But there may also be a trend in the mean values (not conclusive though).

The increase in noise is much lower than Craig Stark had in his measurements (1.2 vs 50 ADU), the change in mean was also much lower (1.2 ADU vs. 20 ADU). At least the temperature drift did not have big effect -- in this example.

Georg

[georg.viehoever]

Hi,

I have tried to repeat some of the experiments from Craig Stark's paper for my Canon EOS40D. For this purpose, I took 10 darks each at different exposure times from 1/8000s (effectively a bias) to 960 seconds, all at room temperature 19 degree Celsius, with 30-60 seconds between them to give the sensor some time for cooling. The raw statistical data is attached to the end of this post.

The data suggests that the bias is 1024.6. The dark current appears reasonably linear with exposure time, see screenshot 1 (note log scale on both axis). The wiggles at exposure times <1sec may be due to the jumps of +-1 that I reported in my previous post, but their absolute values are irrelevant to dark calibration. The remaining wiggles may well be due to changes in the environment, such as changes in room temperature (18.9-19.7 degrees C), sensor heating (see below) and the change of batteries at 120 seconds. I do not see the apparent change of bias that Craig observed in Fig. 1 of his paper for other Canon cameras.

Looking at the variance (noise) in screenshot 2, it grows stronger than expected for exposures<15 seconds. The variance (=stdDev*stDev)should be roughly linear with exposure or dark current. Clearly something is reducing noise for <15 seconds images.  If we plot variance against elapsed time in screenshot 3, we can also see a certain heating effect for the long exposure images - one minute apparently is not enough for cooling the sensor. This can also be seen in a plot of mean vs. elapsed time (not shown).

Looking at the histograms of masterBias (10 shots at 1/8000s, screenshot 4, note log scale on y) , we see the clear peak at 1024 that actually shows a nice normal distribution (=read noise?), a long tail going to 1200 (=warm pixels?), and occasional pixels with values >1200 (defects).

For masterDark (10 shots at 30s, screenshot 5, note log scale on y), we again can see the 3 sections again. The warm pixel section of course goes up to 7000 now. Screenshot 6 shows the histogram of the masterDark for all pixels<1100. We clearly see the two peaks.

MasterDark-MasterBias has a histogram as show in screenshot 6, limited to x=+-100, and the statistics shown below. To avoid negative values, I apparently need a pedestal.

Code: [Select]

> summary(as.numeric(masterDark-masterBias))
     Min.   1st Qu.    Median      Mean   3rd Qu.      Max.
 -105.100    -3.473    -0.593     7.565     2.407 15370.000


Here is what I seemingly can lean:
- I need not worry about a changing bias for my DSLR.
- Darks are roughly linear for exposure times>15 seconds
- Noise is also linear, but for exposures<15 seconds some kind of noise reduction kicks in.
- I need more than 60 seconds between shots to avoid heating effects, or I need more than 10 shots to reach an equlibrium for heat.
- I need a small pedestal of maybe 120 to avoid negative values during calibration.

Georg

Code: [Select]

      exposure     mean median        sd
1     0.000122 1024.170   1024  12.90816
2     0.000122 1024.859   1025  12.89027
3     0.000122 1024.872   1025  12.91453
4     0.000122 1024.564   1025  12.88926
5     0.000122 1024.591   1025  12.89589
6     0.000122 1024.611   1025  12.90209
7     0.000122 1024.900   1025  12.89584
8     0.000122 1024.840   1025  12.89873
9     0.000122 1024.889   1025  12.91116
10    0.000122 1024.893   1025  12.89672
11    0.010132 1024.644   1025  13.95698
12    0.010132 1024.212   1024  13.96718
13    0.010132 1024.898   1025  13.95350
14    0.010132 1024.896   1025  13.96014
15    0.010132 1024.183   1024  13.94923
16    0.010132 1024.197   1024  13.92837
17    0.010132 1024.895   1025  13.95674
18    0.010132 1024.725   1025  13.94495
19    0.010132 1024.595   1025  13.95277
20    0.010132 1024.651   1025  13.96647
21    0.040526 1024.788   1025  14.01994
22    0.040526 1024.963   1025  14.01980
23    0.040526 1024.947   1025  14.03727
24    0.040526 1025.006   1025  14.02093
25    0.040526 1024.961   1025  14.03190
26    0.040526 1024.141   1024  14.01569
27    0.096388 1025.099   1025  14.16422
28    0.096388 1025.041   1025  14.20052
29    0.096388 1025.052   1025  14.19465
30    0.096388 1024.768   1025  14.18104
31    0.096388 1024.269   1024  14.20591
32    0.096388 1024.769   1025  14.20028
33    0.096388 1025.035   1025  14.17686
34    0.096388 1025.027   1025  14.19935
35    0.096388 1024.318   1024  14.18619
36    0.096388 1024.581   1025  14.18765
37    0.324210 1024.721   1025  15.18252
38    0.324210 1025.038   1025  15.08274
39    0.324210 1024.741   1025  15.14556
40    0.324210 1024.720   1025  15.18813
41    0.324210 1025.097   1025  15.13278
42    0.324210 1024.452   1025  15.18279
43    1.000000 1025.280   1025  20.68702
44    1.000000 1025.204   1025  20.67807
45    1.000000 1025.273   1025  20.75284
46    1.000000 1025.308   1025  20.71311
47    1.000000 1025.229   1025  20.87962
48    1.000000 1025.212   1025  20.71697
49    1.000000 1025.320   1025  20.73751
50    1.000000 1025.268   1025  20.76022
51    1.000000 1025.289   1025  20.77404
52    1.000000 1025.238   1025  20.77030
53    5.187358 1025.980   1025  62.42621
54    5.187358 1026.380   1025  62.40257
55    5.187358 1026.350   1025  62.42279
56    5.187358 1026.366   1025  62.66998
57    5.187358 1026.256   1025  62.73121
58    5.187358 1026.357   1025  62.84192
59    5.187358 1026.418   1025  62.95821
60    5.187358 1026.383   1025  63.12903
61    5.187358 1026.309   1025  62.98107
62    5.187358 1026.307   1025  63.09342
63   16.000000 1028.759   1025 176.12136
64   16.000000 1028.771   1025 176.36056
65   16.000000 1028.735   1025 176.69793
66   16.000000 1028.827   1025 177.13879
67   16.000000 1028.881   1025 177.49895
68   16.000000 1028.941   1025 177.90368
69   16.000000 1028.868   1025 178.35236
70   16.000000 1028.831   1025 178.69796
71   16.000000 1028.893   1025 178.97281
72   16.000000 1028.935   1025 179.34006
73   30.000000 1030.907   1025 262.09529
74   31.000000 1030.659   1025 263.51427
75   31.000000 1030.635   1025 263.65746
76   31.000000 1030.904   1025 263.94066
77   31.000000 1031.047   1025 264.08750
78   31.000000 1030.643   1025 264.22004
79   31.000000 1030.682   1025 264.41856
80   31.000000 1030.886   1025 264.53648
81   31.000000 1030.891   1025 264.62104
82   31.000000 1030.754   1025 264.78100
83   59.000000 1031.774   1023 304.21146
84   60.000000 1031.547   1023 304.40356
85   60.000000 1031.782   1023 304.58206
86   60.000000 1031.862   1023 304.76579
87   60.000000 1032.400   1024 304.87072
88   60.000000 1031.939   1023 305.01412
89   60.000000 1031.767   1023 305.13291
90   60.000000 1031.742   1023 305.20812
91   60.000000 1032.561   1023 305.32153
92   60.000000 1032.590   1023 305.39698
93  120.000000 1034.068   1021 318.78778
94  120.000000 1034.336   1021 319.28860
95  120.000000 1034.982   1021 319.57189
96  120.000000 1034.874   1021 319.82557
97  120.000000 1034.989   1020 320.06215
98  120.000000 1035.478   1021 320.31031
99  120.000000 1034.975   1020 320.44354
100 120.000000 1035.632   1021 320.62560
101 120.000000 1035.541   1020 320.76830
102 120.000000 1035.705   1020 320.92059
103 240.000000 1045.834   1021 364.27368
104 240.000000 1047.338   1021 366.80720
105 240.000000 1048.437   1021 368.85005
106 240.000000 1049.630   1022 370.28304
107 240.000000 1050.043   1022 371.67265
108 240.000000 1050.536   1021 373.03865
109 240.000000 1050.837   1021 373.88219
110 240.000000 1051.077   1021 374.42117
111 240.000000 1051.733   1022 375.14148
112 240.000000 1052.089   1022 375.59733
113 480.000000 1073.041   1021 474.20141
114 480.000000 1073.743   1019 480.56519
115 480.000000 1076.079   1020 484.70915
116 480.000000 1076.910   1020 488.05670
117 480.000000 1077.300   1019 490.87133
118 480.000000 1077.994   1019 493.06115
119 480.000000 1078.469   1019 495.12496
120 480.000000 1079.176   1019 496.71547
121 480.000000 1079.226   1019 499.48021
122 480.000000 1080.426   1019 503.63972
123 960.000000 1119.885   1007 711.80435
124 960.000000 1119.289   1006 716.64172
125 338.000000 1065.605   1020 437.09502

[georg.viehoever]

Screenshots 5+6

[ruediger]

Attached is a new version of the Basic CCD Parameters script.

Changes:
- some GUI improvements
- additional numeric control field for entering the maximum ADU the Camera can deliver. This is easily measured with an overexposed shot
- dynamic range gets computed (fullwell capacity / readout noise)
- optional second dark file to measure dark current for those cameras (especially Canon DSLR), where an on-chip circuit is measuring and subtracting dark noise from the signal.

The second dark file allows calculation of the dark current without using the mean values of bias and dark frames. If the mean value of the first dark file is not at least sigma(dark) larger than the mean of bias1 file, then the dark current is instead computed from the change of sigma from dark1 to dark2. This requires, that the exposure time of dark2 is much larger than that of dark1 (e.g. factor 10) and that the camera temperature is exactly the same. I recommend taking 5x10min, then 5x1min and throw away the first two shots, because they are only needed to "warm up" the camera. Extract the exif temperature with exiftool and select a matching pair of each a 1min and a 10min frame.

Inside the dark files D1 and D2 two noise sources are added: readout noise and heat noise, therefore:
sigma(D1)^2 = sigma(D1_noise)^2 + sigma(bias)^2, same for D2.
The longer exposed D2 has n times more noise than D1, therefore sigma(D2_noise) /sigma(D1_noise) = sqrt(n), allowing to compute the dark current.

The results I now get, are very comparable to Christian Buils results (http://www.astrosurf.com/buil/50d/test.htm). What is still strange too me, are the differences across the subpixels of the bayer matrix.

Rüdiger

[georg.viehoever]

I computed the histograms of a number of master darks created from the data in http://pixinsight.com/forum/index.php?topic=4086.msg29086#msg29086, and made the attached movie showing them as exposure time increases. Note that y-scale in log, the exposure times does not increase in a regular fashion, and although most master darks were created from 10 shots, some have fewer (due to technical issues during data capture).

It is interesting to see how different "families" of hot pixels depart from the central peak at 1024 with increasing exposure times. I did not expect these families.

Georg

[georg.viehoever]

Attached is a new version of the Basic CCD Parameters script.

Hi Ruediger,

I tried your script on some of the data of my Canon EOS 40D, and compared it to the data reported by Clark in http://clarkvision.com/articles/digital.sensor.performance.summary/index.html.

-on the first launch, I got an error message (that I ignored). You may have to restart PI to see them as well. I just ignored them.

Code: [Select]

run --execute-mode=auto "C:/Users/georg/Downloads/CCDParameters.js"

Processing script file: C:/Users/georg/Downloads/CCDParameters.js
CCD Parameters Script started
Version Info:PixInsight Core 01.07.06.0793 (x86_64),undefined,false,undefined,undefined
** Warning [156]: C:/Users/georg/Downloads/CCDParameters.js, line 613: assignment to undeclared variable onvalueUpdated
- You should explain in your help how to set the format parameters for reading RAW to get useful results. I used those in screenshot 1. As an alternative, you may want to offer loading the data as part of the script (with format hints "raw cfa").
- I got the results in the attached screenshot 2:
-- readout noise seems reasonable
-- the script reports no dark current, despite the 120 seconds darks. This may be due to the fact that this was a freezing cold night <-10deg celsius.
-- the full well capacity and dynamic range dont make sense and dont match at all with Clarks data.
-- The gain (rescaled in Clark's report to 12 bits) seems too low for ISO800
-- The dynamic range probably is in linear units (and apparently much too low). Usually, people rather want to see the number of stops (log2).
Glad you are taking care of the script!
Georg

[ruediger]

Hi Georg,

thanks for the report! This encourages me to further work on the script.

-- the script reports no dark current, despite the 120 seconds darks. This may be due to the fact that this was a freezing cold night <-10deg celsius.

There was no dark current measured because mean(D1-B1) is close to zero and exptime(D2)=exptime(D1), so the fallback code is silently skipped. D2 must have much longer exposure time than D1, so that stddev(D2) >> stddev(D1) holds.

-- the full well capacity and dynamic range dont make sense and dont match at all with Clarks data.
-- The gain (rescaled in Clark's report to 12 bits) seems too low for ISO800
-- The dynamic range probably is in linear units (and apparently much too low). Usually, people rather want to see the number of stops (log2).

I trust Christian Buil's data (and my own code  ) much more than Clark's data. All the values you measured for ISO 800 make perfectly sense compared to Buil's result of the 40D at ISO 400. (http://www.astrosurf.com/buil/50d/test.htm).

Your measured  full well capacity at ISO 800 (6.100e) is nearly 50% of 12.900 (ISO 400), so probably correct. Also your gain at ISO 800 (0.37) is not so far away from 50% of Buil's 0.84 (ISO 400). Same for the dynamic range (ISO 800: 1180 ~ ISO 400: 1900). This is indeed the number of tonal values that can be distinguished, log2 will give the f-stops.

For a more exact computation of fullwell capacity and dynamic range you must measure and enter the maximum ADU from your camera. There is also an error inside the code: I forgot to subtract the bias pedestal to get the effective ADU range of the camera.

Corrected version follows soon

Rüdiger

[tsaban]

Hi,

...
How does other DSLR (Nikon etc.) handle dark current? Some volunteer to run the script?
...

I don't own a Nikon camera but had the chance to talk to a Nikon engineer 2 years ago. From what I kept in my memory he said that every manufacturer would do such adjustments because of production imperfections of the sensor, the coloring variations of the Bayer matrix and inperfections of microlenses on the ensor. He said that every unit is tested and correcting multiplicators for every pixel are uniquely saved for each sensor. The blackened (unused) pixels at the border serve as reference values.

[mschuster]

Hi Rüdiger,

Georg pointed me here, thanks for your updates to this script. I made two minor tweaks and it works fine now on my frames. Using PixelMath I did the calculations manually and results match the script's results. Here are the tweaks:

- My darks are 2400s so I increased the dark exposure time limit.

- In my frames the FITS ISOSPEED keyword value is 'Low Gain' which parses as NaN in the script. So I added a "... && !isNaN(isoSpeed)" to avoid displaying "ISO:NaN".

Finally, my sensor saturates at about 0.8 (normalized units) so the script's full well capacity calculation is about 20% too large. I know this 0.8 value by measuring the ADU of the cores of the saturated stars in my subs. But of course there is no way for the script to know this without additional information. So for now I just note this as a known limitation of the script.

Thanks,
Mike