PixInsight Forum

PixInsight => General => Topic started by: Stonius on 2018 November 26 18:41:43

Title: Non-linear Dark current
Post by: Stonius on 2018 November 26 18:41:43
Hey all, I've been following Craig Stark's tutorial on defining your camera's noise characteristics.

According to him, Dark Current should show a linear regression.

I notice my dark current over time increases in a non-linear fashion, more inline with a power series (y=ax^b rather than y=ab^x).

I'm assuming that the dark current scaling algorithm in Pixinsight expects a linear regression (twice as much time = twice as much dark current), so for me the scaling won't work.

I guess I could just take dark flats, but I'd ultimately like to figure out why this is so.

Someone suggested amp-glow may account for this non-linearity.

It could also be an issue with my bias frames which are required for this calculation, but I can't think what. As far as I can tell, the bias and resultant integrated master are fine. Just to be sure, I took the bias series at the same settings as the darks (except for exposure length, of course!)

ASI1600MM Pro, Filterwheel plugged through cameras onboard hub.
Slowest download speed (40). Cable from 1600 going direct to computer
Set point -15 C
It seems a consistent curve shape at exposures ranging from 30 to 600 secs at all gain levels.

Many thanks for any help.

Markus
Title: Re: Non-linear Dark current
Post by: pfile on 2018 November 26 19:47:21
PI doesn't rely on any characteristics of the input image (gain, exposure time, temperature, etc.) when scaling darks. what it does is iteratively scale the dark while measuring the noise in the calibrated result (on a square region at the center of the frame). when the noise is minimized, PI uses that scaling factor to scale the master dark and calibrate the whole light.

so in theory it shouldn't matter how the dark current behaves as aa function of time or temperature.

but in regards to your actual problem... i dunno. maybe someone with experience with CMOS cameras can chime in.

rob

Title: Re: Non-linear Dark current
Post by: Stonius on 2018 November 26 22:05:33
Thanks Rob, that's good to know

Best,

Markus
Title: Re: Non-linear Dark current
Post by: mcgillca on 2018 November 29 08:56:11
Dear Markus,

Hi - I measured my dark current for this camera recently as well.

What is a little odd is that there appears to be amp glow when reading an image, but not in the bias frames (not sure why).

So my dark current (as measured by difference between bias frame and dark frame), scaled as a constant plus a linear term. The linear term was very close to that reported by ASI on their web site.

Not sure why your's is exponential. Could you post your results as a graph?

Colin
Title: Re: Non-linear Dark current
Post by: RickS on 2018 November 29 22:11:36
What is a little odd is that there appears to be amp glow when reading an image, but not in the bias frames (not sure why).

Hi Colin,

Amp glow is usually of thermal origin, so it builds up over time.  I notice a lot more amp glow in my longer darks.

Cheers,
Rick.
Title: Re: Non-linear Dark current
Post by: mcgillca on 2018 November 30 10:44:09
Thanks, Rick - as I understand it, with the Pro model, the image is downloaded into a DDR so the amp glow is significantly reduced (it seems to be only turned on during reading the image, not during the imaging itself). That was one reason ASI introduced the memory - people were reporting significant amp glow with USB 2, but much less with USB 3 since it was a much faster download.

I should plot up my data and confirm it actually behaves that way!

Colin
Title: Re: Non-linear Dark current
Post by: RickS on 2018 December 02 19:04:30
Here are the results of testing dark current build up in my ASI1600mm Pro.  For frames longer than a few seconds it is quite linear.  For short frames (which I believe are timed by the camera and not the capture software) the results show some sort of CMOS weirdness.

Cheers,
Rick.
Title: Re: Non-linear Dark current
Post by: Stonius on 2018 December 02 19:12:49
Dear Markus,

Hi - I measured my dark current for this camera recently as well.

What is a little odd is that there appears to be amp glow when reading an image, but not in the bias frames (not sure why).

So my dark current (as measured by difference between bias frame and dark frame), scaled as a constant plus a linear term. The linear term was very close to that reported by ASI on their web site.

Not sure why your's is exponential. Could you post your results as a graph?

Colin

Sure. Maybe I'm reading things wrong.

Here's the only linear graph I get - when I look at total mean dark current for the entire exposure.

But when I look at the noise in terms of e-/sec you see the non-linear aspect of things.

The first graph shows you that dark current increases over time.

The second tells you that the rate of increase is not linear.

That's what I'm confused about.

Oh, I should point out, all graphs in 12 bit numbers because it relates to the camera better.

Also my 'strange' gain settings are because I wanted the driver gain settings to reflect e-/ADU conversions.

My Unity Gain sits at 135 (not 139 as specified) which is exactly twice the gain of 195 (the first being 1e-/ADU, the second being 0.5 e-/ADU). That way I'm working with actual stops of light, which makes a lot more sense to me.

Cheers, Markus



Title: Re: Non-linear Dark current
Post by: RickS on 2018 December 02 19:29:04
Hi Markus,

If you're getting linear growth in total ADU (bias + dark current) then that's exactly what I'd expect.  What exactly are you measuring in your second graph?  Dark current noise should be the square root of the dark current (because it's a form of shot noise).  Dark current noise and read noise will be the two major sources of noise in your dark frames.

Cheers,
Rick.
Title: Re: Non-linear Dark current
Post by: RickS on 2018 December 03 14:41:21
Gain 72 added to the graphs and I have also added plots of dark e- as well as total ADU.
Title: Re: Non-linear Dark current
Post by: Stonius on 2018 December 03 15:55:35
Hey Rick

The second graph is what happens for me when I take a gain setting and run bias and darks of varying lengths, then subtract the bias from the result and divide by the number of seconds I exposed for. IOW, it's the number of electrons per second.

Maybe I'm confused, but I would have assumed that given the dark current is supposed to be linear in nature, the number of electrons *per second* should be the same no matter the length of the exposure (as long as the gain stays the same).

So, at say, 135, the number of electrons *per second* varies between 0.031 and 0.008.

The rate at which the dark current accumulates is almost 4 times greater for a 30 second shot, as opposed to a 10 minute shot.

Also, I think I've been using the terms 'dark current' and 'dark noise' interchangeably. Sounds to me like you're saying the dark current *causes the dark noise at a level which is the square root of the dark current. Just trying to get my terms correct :-)

Cheers,

Markus
Title: Re: Non-linear Dark current
Post by: RickS on 2018 December 03 16:56:59
Hi Markus,

Here's my graph of dark current in e-/sec vs duration.  Interestingly, the differences between gain 72 and gain 200 I see are very small which is not the case in your graph.  There does seem to be some other factor at play at the short end of the graph but still the overall result still looks pretty close to a straight line if you look at the graph of mean e- vs duration.  I have a couple of ideas to try to see if I can tease out some additional info.  Will post if I find out anything interesting.

Dark current and dark current noise are definitely not the same thing!  The signal to noise articles by Craig Stark are a good introduction to some of the things you should understand if you want to dig into this: http://www.stark-labs.com/craig/articles/articles.html

Cheers,
Rick.
Title: Re: Non-linear Dark current
Post by: Stonius on 2018 December 03 19:23:59
Ah, that's interesting, thanks for posting. At least your graph has the same shape as mine :D

Not sure where the offsets are coming from. I might look at the bias frame data for mine. I used an integration to get an average of all biases, but maybe a single bias may remove the offset better.

Cheers
M
Title: Re: Non-linear Dark current
Post by: RickS on 2018 December 03 19:37:04
Not sure where the offsets are coming from. I might look at the bias frame data for mine. I used an integration to get an average of all biases, but maybe a single bias may remove the offset better.

Hi Markus,

I load all the frames into Blink and use the "Series analysis report" button to dump the data to a text file.  Then I import it into a spreadsheet and crunch the numbers there.

Cheers,
Rick.
Title: Re: Non-linear Dark current
Post by: Stonius on 2018 December 03 21:20:39
Ah, I didn't realise you could do that! Muchas Gracias!
Title: Re: Non-linear Dark current (Solved)
Post by: Stonius on 2018 December 09 23:54:45
For the sake of others following this thread, I got some answers from Chad at ZWO which echoed some of the findings in John Upton's Cloudy Nights thread quoted above.

Apologies if everyone else knew this already. Everywhere I had read seemes to say bias frames should be as short as possible, and that they should then be used as a basis for assessing dark current in longer exposures. As you'll see below, neither is true (at least of this camera - maybe other CMOS cameras behave the same way?)

The trick was not to compare the darks to a bias frame, but to a set of 2 second frames. Attached is the result of that simple change. You can see the graph looks much more like you'd expect.

The thread from ZWO can be found here.

The long and the short of it - When trying to measure dark current per second, the shortest exposure you should use is no less than 2 seconds. A bias frame is no good and will give you the same results I did. In addition, bias frames should be shot at 0.1ms, not 0.000032s as I did (I mistakenly shot at 0.000032 because it's the shortest possible exposure on the 1600. Don't do what I did, it was wrong!).

And if you really want to get technical, you can follow John Upton's advice in post #3 on the above thread and calibrate your Bias frames using a Y intercept of a plot of multiple Dark Frame exposure times.

Relevant bits from Chad and John quoted below.

HI Markus?
We tested the 1600 MM Pro with your method. It has the similar result with yours.
So thanks for your support first.
For the result, we think it may be the following reasons.
When we have an operation, for the circuit, it's like dropping a pebble on a calm water. There will be some disturbance in the circuit. When the gain is large, the disturbance is greatly amplified. For long exposure, there are some different operations with short exposure.
Beside, consider that the short exposure is different with long exposure in the camera, so I think it is better to measure at long exposure. It means you should use 2 seconds exposure time to replace the image of 32uS exposure.
Thanks
Chad

Hi Markus?
For this test, because it is a test about the dark current. If the exposure time is short, we usually think the dark current is not the main noise. the main noise should be read noise and other noise. That is why I suggest that use the 2 second to make the test.
But for bias field, usually, we want to use it to calibrate the read noise. So it should not a long exposured image. Also consider the fluctuations of the circuit, I suggest that you can use 0.1ms instead of 32?S to make the bias field. It should be better.
Thanks
Chad

And from John Upton;

"The convoluted process for scaling Dark Frames begins with having a Bias and Dark Frame library taken at the same Temperature, Gain, and Offset. The first step is to determine the Dark Current slope and Y intercept of a plot of multiple Dark Frame exposure times. My Dark Library uses 50 each 0, 60, 120, and 240 second exposures. The data for Mean ADU values of each average integrated frame exposure is plotted against Exposure time in a spreadsheet. The slope and Y intercept of such a graph is easily obtained using the LINEST() function. The results will give us the parameters we are looking for. The Slope gives us the Dark Current rate for the sensor while the Y intercept gives us the equivalent Mean ADU value for a CCD-like Bias Frame taken at 0 seconds exposure.

This Dark Frame-derived Bias Mean just tells us what the mean of our camera Bias should have been. It is just a number and contains no information whatsoever about the pattern noise from our camera. Our actual Bias Frame from the camera will have the pattern noise we need but the Mean value is off what it should have been. The second step is to subtract the difference between the Mean of the Bias image and the Y intercept on a pixel by pixel basis. This can be done using PI PixelMath with an equation of “$T – (mean($T) – Y_Intercept_Of_Dark_Plot)”. After this adjustment, we now have an Adjusted Bias that can be used to calibrate a Dark Frame so that it can be scaled.

Scaling the Dark Frame for use in calibrating our lights can best be done using PixelMath again. Here, we simply scale the Dark by the ratio of exposure times between what we have and what we need. For example if we have Dark Frames in our library for 60 and 120 seconds but took our target lights at 90 second exposures, we would use PixelMath on our 120 second Library Dark Frame and write “($T - Adjusted_Bias) * (90 / 120)”. This gives us the scaled Dark Frame for use in the ImageCalibration for our lights. We would also plug in the Adjusted Bias as the Bias file in ImageCalibration. A similar pre-calibration process should be used on the Flat Frame we will use for ImageCalibration."