Question about the "BASIC CCD Parameters" Script

[Christoph Puetz]

Dear PI Community,

I'm getting deeper into CCD technology and I wanted to try the above script.
I own a canon DSLR. My question ist:
Do I have to debayer my FITS Files after conversion to use this script or not ?

Thanks in advance for your reply,
Christoph

[Philip de Louraille]

My understanding of bayered data is that is it an encoding of data as captured by the CCD.
You can work with bayered data as long as you use the entire dataset (no subsampling, cropping, rotation) and you can subject each frame with corrections with flat, bias and dark master frames as long as those are also bayered and were entirely calculated with frames that were left in a bayered state and, basically, were never ever debayered.

By the way, I do not see the script you mentioned (not that I'd be particularly able to understand it) but someone else might.

PS: I have been wrong before. ;-)

[Christoph Puetz]

I tried the script with debayered and non-debayered images from my Canon EOS500d, taken under the same circumstances
and within about 5 minutes.

If I do not debayer them, the results seem to be right to me (as compared to some other Canon SLRs),
esp the gain and dark-current values seem to be fine.

But if I debayer them, the results are getting "strange" (also slightly depending on the debayer algorithm).
This makes some sense to me, so I think I should not debayer them (or there might be an error anywhere  )

Thanks for looking and helping ....

[georg.viehoever]

I just noticed that I am the original author of this script...done several years ago when I was young 

- this script never got a careful review by someone who really understands the physics, and I remember http://pixinsight.com/forum/index.php?topic=2097.msg13589#msg13589 that said the estimates where off. If you feel like it: Please review the script, and compare it to the basics described in HAIP http://www.willbell.com/aip4win/Errata%20to%202nd%20%20Printing%202nd%20Edition%20of%20HAIP.pdf, compare it to the results generated by other tools, ... . The core is just 300 lines of code.
- at the time of writing the script, I did not really understand debayering.  The values you see for the debayered version clearly make no sense. Did you try Superpixel-debayering? That should limit the effects generated by debayering. Maybe you also try saving the debayered images as 16bit integers instead of floats. Anyway, for reliable estimates, the script should work directly on the CFA pattern, which it currently cannot do.
- If you currently apply it to an un-debayered image,  you will get values that are a kind of average for the 3 channels. I am not sure how much sense these make- for example it is very likely that the e/ADU values are different for R,G+B.

Georg

[Christoph Puetz]

Hi Georg, nice to meet you again 
You wrote it when you were young ? Well, what's a human life compared to the periods of space ? 

OK, then I should take a deeper look into your script. I found a small error, but this could not explain the differences.
I also own AIP4WIN (but it prefers to crash with an overflow error, if I load the fits files for diagnostics  , nothing really
compares to PI!).

It will take me some time to test and check for correct results.
I will contact you (maybe by PM), if I have some questions.

Thank you for your advice !
Christoph

[georg.viehoever]

Nice to meet you again too. Considering the size and age of the universe, it is highly improbable to meet anyone twice...

Yes, feel free to PM me. Make any improvements to the script that are necessary. Just make sure you publish them.
Regarding the handling of CFA: maybe you should take the same route as Cosmetic correction: Just handle the 4 pixels of a bayer 2x2 pixel group as 4 channels. That makes sense because the two green pixels of such a group may have different characteristics.

Enjoy!
Georg

[ruediger]

For me, the following text http://theory.uchicago.edu/~ejm/pix/20d/tests/noise/noise-p2.html#readandshot was more understandable. For gain measurement, the authors describe a setup very similar compared to the one from HAIP. It should be sufficient to take only one pixel from the bayer matrix. The gain can't depend on the color, only on the relationship between measured ADU and noise via the Poisson-noise relationship.

I totally forgot about the BasicCCDParameter and SkyLimitExposure scripts. Will try them out as soon as possible.

Rüdiger

[ruediger]

I tested the Basic CCD Parameters script with EOS 5D II data, taken at ISO 200 and ISO 1600 and did not get any meaningful results, even negative gain values. Here my quick analysis of what's going wrong with this script:

- the cropped and analyzed part of the image might lie on half pixel positions. Minor and irrelevant, but leads to an unitialized rightmost column in the cropped image parts that are used for calculation. My input image size is 5634 x 3753. If you use the "Images" button from the script, you get 375x375 images, which after autostretch show the defect column.

- Images are used for storing temporary results of sums and diffs of flat and bias frames. One would expect to get the same stddev from "flat1 - flat2" and "flat2 - flat1", but this is not true in this case. The reason lies in the signedness of the image container. Once you run into a situation where stddev( flat1 - flat2 ) is much smaller than stddev( bias1 - bias2 ), the scripts runs amok.

My flats have following properties, taken from Image->Statistics:

            Flat1      Flat2
Mean    4982.5  5156.7
Median  5736.0  5943.0
AvgDev  874.2   912.2
StdDev  1401.8  1462.6

For the differences (generating temporary images with Pixelmath, as in the CCD script), the following holds:
stddev( flat1 - flat2 ) = 2.2 (wrong and much too low, same value as CCD scripts reports)
stddev( flat2 - flat1 ) = 86.4 (reasonable)

Only if adding an pedestal during the diff, you get an usable result, i.e. stddev( flat1 - flat2 + 0.015 ) = 86.8.

Rüdiger

[georg.viehoever]

Hi Rüdiger,

...It should be sufficient to take only one pixel from the bayer matrix. The gain can't depend on the color, only on the relationship between measured ADU and noise via the Poisson-noise relationship....

- I am using multiple pixels because the statistics provided by a single (or very few) pixels are not reliable enough. It is also not possible to compute stdDev for single pixels 
- I had the impression when playing with the script (some years ago) that gain for R,G,B is indeed different on my CanonEOS40D, even when working with RAW. However, this may be the result of my use of debayered images.

...
- the cropped and analyzed part of the image might lie on half pixel positions. Minor and irrelevant, but leads to an unitialized rightmost column in the cropped image parts that are used for calculation. My input image size is 5634 x 3753. If you use the "Images" button from the script, you get 375x375 images, which after autostretch show the defect column.

- Images are used for storing temporary results of sums and diffs of flat and bias frames. One would expect to get the same stddev . from "flat1 - flat2" and "flat2 - flat1", but this is not true in this case. The reason lies in the signedness of the image container. Once you run into a situation where stddev( flat1 - flat2 ) is much smaller than stddev( bias1 - bias2 ), the scripts runs amok.
...

I did not run tests, but read the code again:
-You are right about the cropped part. As computed by computeTargetRect() (line 270), it may indeed lay on half a row/column. To fix this, you would need to fix this method, in particular the moveTo() call (line 276).
- I am not so sure about your analysis regarding the image differences. By default, PixelMath rescales results to the range [0,1], which the apply() calls (e.g. line 358) probably does not (after reading the doc for the corresponding C++ operation in http://pixinsight.com/developer/pcl/doc/html/classpcl_1_1Generic2DImage.html#a96646100c18c95a1de5de737e56daf29).  Are you sure?

Christoph, Ruediger: Feel free to improve the script where necessary.

Georg
 

[Christoph Puetz]

Hi Rüdiger and Georg,

some interesting information: Try to convert all FITS files from RAW to 16bit integer and compare the results.
You can also debayer them (but keep in the 16bit integer range).
I checked Georgs computation and it seems to be right (as written in the mentioned errata of the AIP book).
I assume that the main source of error is the computation between FITS pixel depth and the given pixel depth of the camera, as given in the input application form. The results do vary with the FITS "pixel depth" (and this should only depend
on the pixel depth of the camera, right ?).
Can you confirm this ?

It's great to have such driven people here !

[ruediger]

- I am not so sure about your analysis regarding the image differences. By default, PixelMath rescales results to the range [0,1], which the apply() calls (e.g. line 358) probably does not (after reading the doc for the corresponding C++ operation in http://pixinsight.com/developer/pcl/doc/html/classpcl_1_1Generic2DImage.html#a96646100c18c95a1de5de737e56daf29).  Are you sure?

In PixelMath, there is a "rescale result" checkbox, which gives same stddev for "flat1-flat2" and "flat2-flat1", when applied. In your script Flat1/Flat2 and Bias1/Bias2 are not commutative, leading to totally different results. So obviously no "rescaling" is applied. But "rescaling" in this context is a deadly operation, because it renders the stddev value unusable. Since we want to measure the photon shot noise from the difference of two consecutive flats, we are interested in the distribution of the ADU values of the unscaled result! So instead of a "rescale" we need a "offset" operation. Right now I'm not too familiar with the PixInsight class library, so I'm not able to provide a clean solution.

Good news: There is an ugly, yet working workaround to quickly measure RAW files:

- patch line 250 of CCDParameter.js as follows:
      targetRect.moveTo( Math.ceil( (width-newWidth)/2 ), Math.ceil( (height-newHeight)/2 ) );

- load the CR2 files as 16 Bit Integer. This is just the default when you drag them from explorer to PixInsight. The DSLR Raw settings must be set to "RAW CFA bayer monochrom" and "No black point correction".

- apply PixelMath "$T + ( 1024.0/65536.0 )" to bias1, flat1, dark1, but not to flat2, bias2 before using them in the script. Make sure to not check "rescale". This makes the three difference operation within the script safe without having an side effect to the computations.

The results then are looking very promising and comparable to what I found in the internet, but there is still the mix of the RGGB channels from the bayer matrix. I have no quick solution to this.

Rüdiger

EDIT: The attached screenshot is labeled with ISO 1600, but it shows the result from ISO 200!

[ruediger]

I assume that the main source of error is the computation between FITS pixel depth and the given pixel depth of the camera, as given in the input application form. The results do vary with the FITS "pixel depth" (and this should only depend on the pixel depth of the camera, right ?).
Can you confirm this ?

Confirmed. The scripts makes calls to img.stdDev() and img.mean() with results always in [0,1] independent of the image's pixel depth. So this value has to be scaled for the ADU range. Within the script, the scaling factor is derived from pixel depth as maxSample = "2^(pixel depth) - 1" (function computeMaxSample).

The natural bit depth of a RAW image is 16 bit (not necessary the camera bit depth). When you change the bit depth to 32, stdDev and mean in [0,1] stays the same. So the correcting factor must always be 2^16 - 1 to map e.g. the bias mean to the expected "1024".

I have an idea of how to change the script to work correctly both with monochrome and CFA data. In case of CFA data, I'm going to replace "channel 0 ... 2" to "subpixel 0...3". For extracting the subpixels and the computation on images (diff, sum etc.) I will only use javascript computation with "img.pixel()" and "img.setPixel()" to avoid any pitfalls with library calls etc.

Rüdiger

[georg.viehoever]

...The natural bit depth of a RAW image is 16 bit (not necessary the camera bit depth). When you change the bit depth to 32, stdDev and mean in [0,1] stays the same. So the correcting factor must always be 2^16 - 1 to map e.g. the bias mean to the expected "1024".

There are cameras that are working with 8 bits (e.g. WebCams). The way that PI reads CanonEOS40D RAWs is by reading it into 16 bit integers, using only 14 bits. If I remember correctly, PI's JavaScript always sees float values in the range [0..0.25[. The bit depth is used to rescale the values read to the ADU (integer) values. So probably the correct name for this factor would be scale bit depth or something like this.

I have an idea of how to change the script to work correctly both with monochrome and CFA data. In case of CFA data, I'm going to replace "channel 0 ... 2" to "subpixel 0...3".

Thats exactly what I would recommend. Nice thing about this is that it is also independent of the concrete bayer pattern which you only would need to label the subpixels with "R","G" or "B".

Great analysis!
Georg

[Christoph Puetz]

Hi Ruediger,

in order to avoid duplicate work for us both - will you go on to correct the script, as you are
currently already deper "inside" the algorithms ?
Of course I will be helpful for testing and corrections ....

[ruediger]

in order to avoid duplicate work for us both - will you go on to correct the script

I wanted to start this weekend, so if you are not in a hurry and can wait a week, there is really no need that we both work on it. Keeping the functionality as is and only doing the needed corrections should be relatively easy. But I'm thinking also about useful additions like:
- optional master bias to enhance measurement of photon/thermal noise
- skipping hotpixel within the observed rectangle
- reading the camera temperature from raw files (e.g. with exiftool), dependence of thermal noise from temperature
- user supplied rectangle, useful e.g. when flats are taken from grayscale/color patch test charts
- measurement of pixel response non uniformity

Rüdiger