Dark subtraction question

[johnw3d]

Hi all.  I'm trying to understand what I think are issues with dark-current calibration on my new QHY9M, maybe it's finger-trouble, any help would be appreciated.     The following stems from me trying to debug lots of hot-pixels left after an optimized-master-dark batch-preprocess in PixInsight, turning optimization off and using darks with exact same temperature and exposure time.  I realize now I should probably use dark-optimization coupled with cosmetic-correction to deal with the hot-pixels, but I'd like to understand what's causing the odd simple dark-subtraction behavior I'm seeing.

Below are 200% crops from an example exposure of the center of the Heart Nebula, Baader Ha filter, 10 minute, binned 2x2, -20ºC, and below that the result of subtracting an integrated 25 subframe master-dark, with same temp, binning and exposure time (confirmed in all FITS headers).  Both are auto-stretched screen-grabs from PixInsight.   It seems the master-dark is subtracting way too much from the light, see all the black holes, some of which are hard 0.     Is this expected behavior?

I've tried recapturing the darks several times, on the bench with a metal cap & a box over the camera, and thinking there might have been light leakage, at night at the original observing site & ambient conditions.   

The master-dark was integrated using Vicet's recommended master-calibration-frame procedure and I also tried Nebulosity's built-in pre-processing function, all variations get similar results. 

All captured with Nebulosity 3 on a Mac, QHY9M calibrated at gain 7 & offset 112 to give pedestal of about 760 & max white of about 64500.

Any suggestions?





Here's the master-dark, fwiw:



And here are the calibrated sub and master-dark at the same stretch as the original light:

[Nocturnal]

Hi John,

see? I found your post here :-)

Can you measure those black pixels and confirm they really have a value of 0?

[mschuster]

Maybe try a sanity check on your subframes and masters. This might flag a problem in master generation.

In PI open one bias frame, one dark frame, the master bias and the master dark. Use the Statistics process and measure the median of the four images. Use the NoiseEvaluation script and measure their noise. Make a table and see if things seem reasonable. Here is my data:

frame, median, noise:
bias frame,   4.3e-3, 1.8e-4
bias master, 4.3e-3, 1.9e-5
dark frame,   4.9e-3, 2.2e-4
dark master, 4.9e-3, 8.5e-5

Some checks:

Medians of bias frame and bias master should be roughly the same. Similarly for darks.
Dark frame medians should be larger than bias frames median (darks are not bias subtracted).
Master noises should be less than noise of component frames.
Dark noises should be larger than bias noises.

Mike

[johnw3d]

Sander> Can you measure those black pixels and confirm they really have a value of 0?

Hi Sander.   If you look at the the stats block in the dark-subtracted light image I posted, you can see the minimum value is 0, whereas the minimum value in the master bias is around 640.   I also just looked at the (unrescaled) dark-subtracted light again in the PixInsight HistoTransform tool and it shows 58K pixels or 2.8% are clipped at a shadow threshold of 0.0000153 (1 / 65535).


Mike> Maybe try a sanity check on your subframes and masters. This might flag a problem in master generation....

Mike, thanks for the check suggestions.  Here are the results:

  median, noise s.d.
bias frame: 648.0, 3.863e-04
bias master: 646.500, 6.149e-05  (50 subs)
dark frame 1: 905.0, 2.438e-03;  2: 811.0, 7.819e-04;  3: 883.0, 2.608e-03
dark master:  888.357, 2.212e-03 (46 subs); 886.200, 2.017e-03 (26 subs);  896.200, 2.017e-03 (20 subs)

Things look largely OK except for the noise level in the master-darks, it's not significantly better than the dark frame samples I measured, worse than one of them.   Even more oddly, the noise doesn't seem to get better with subexposure count, I built new masters as I was shooting the darks, going from 20 to 46 frames, noise a tad worse in the 46 sub master!?   Could this be from excessive non-linear hot-pixels?   Also tried masters generated by the pre-processing script and they are identical to those made by hand with Vicet's recommended integration settings.

Thanks all

[Nocturnal]

Hi,

fortunately dark subtraction is a simple operation. That means you can look at pixel values and do the math yourself and see if it makes sense. It is unlikely that the subtraction operation in PI is broken so the problem appears to be that your dark frames are too bright. Find a few example pixels that are zero in your result image and track down why this is.

Your one dark frame is substantially brighter than your master. I would plot the stats for all your dark, bias and light frames and see if there are trends.

An understanding of the underlying math and diligent following of the numbers will quickly get you your answer. Assume nothing, check everything. The stats window shows at least a single 0 value pixel. I did not want to -assume- all black pixels were really zero, it could have been the only one. Use the histogram tool to see how many black pixels you have, if you like.

[johnw3d]

Thanks for the tips Sander.  I've simulated the subtraction in PixelMath as well and get the same result, it certainly looks like the master is too bright.   I looked at several dark-frame samples and their medians are distributed about the master median, so I think the dark-frame integration is right.

However, you are right, I should be able to find a 0 pixel in the dark-subtracted light (there are 58000 of them according to the PixInsight histogram tool) and trace back the contributing pixels in the dark subs and see where things are going wonky, its a simple unnormalized average.   Will report what I find if it's worthy.

Cheers,
John.

[mschuster]

John, the darks do seem to bright, also there is IMO a large amount of variance between them. You mentioned light leaks, maybe that or something related is an issue.

For comparison, the median ADU of a dark frame on my camera grows by about 1 ADU per minute of dark exposure. So the darks are not much brighter than the biases. The variance between equally exposed darks is also small.

Mike

[johnw3d]

I did a simple test with a master dark from 6 random subs from the 10m darks I've been working with and confirmed that it produced a lot of zero pixels after subtraction, and tracked down a few of those pixels in the master subs and confirmed their values in the master dark were indeed the simple average.   For some reason the darks are too light.   

So, trying again with a series of darks 1 min through 5m, seeing what the variance is between subs and between masters.  There's also a chance that the QHY9 support in Nebulosity on the Mac is wonky, it was only recently added, perhaps timing or TEC control or download, so I'm flipping to windoze and will try EZCap (from QHY) and a trial copy of MaximDL.

Mike, quick question re TEC control - what variations do you see in temp across your dark subs?  I'm seeing upwards of a degree in mine (at -20c).  Also, how many darks do you normally take?

Another random Q - is there a source for documentation on the Cosmetic Correction process in PixInsight?

[mschuster]

John,

I don't see any temp variations with the QSI, it stays nailed at my -20.0C set point, except for one situation. When taking biases temp does drift due to the apparently uncontrolled TEC during image download. My workaround is to add a 10 sec delay between bias exposures, by trial and error I found that this gives enough time for the TEC to recover. There is a QSI firmware update that fixes this problem but I have not installed it.

I take 64 darks and rebuild masters maybe once every two or three months or so.

Mike

[NKV]

is there a source for documentation on the Cosmetic Correction process in PixInsight?

I am documentation. But first, be sure that you use latest version of module, because module distributed with bugs.
Also you can look instruction in topic PCL: CosmeticCorrection module

[Nocturnal]

John,

when doing this type of detective work I find it useful to collect statistics for my images and plot that in a spreadsheet. That way I can see trends in average pixel values and stddev over time.

[johnw3d]

Thanks Sander.   I've been building some stats, and so far, the darks are simply too light, the zero pixels in the dark-subtracted light show correct dark subframe averages in the corresponding pixels in the dark master.   

But before I push much deeper in that direction, wanted to ensure there's no capture software problem, so I've switched from Nebulosity on the Mac to EZCap and MaximDL on the PC and I'm in the throws of pulling a set of test dark frames at different exposure times now, and will see how they behave.

It does appear that MaximDL does a better job at holding TEC steady, it's +/- 0.2º, whereas Neb is +/- 0.8º, not sure if this is the issue.   Anyway, will report back once I've completed this series.

On a related note, I can't seem to get a good gain calibration for 2x2 and higher binning on MaximDL - even at 0 gain, I get full-well in every pixel.   About 12% gives me ~64000 max on 1x1.   What's the trick here?

And thanks for the offer of being human-documentation NKV, will no doubt bug you when I get past these current issues., ;-}  (I do have the latest PixInsight updates).


Thanks,
John.

[Nocturnal]

John,

a few degrees difference will not substantially change your dark current. If it did then all hope would be lost because these thermostats can easily oscillate by that much.

I don't know what the well size is on the 8300 but I can see how 4 of those wells won't fit in 64K. That's best asked on the QHY forums.

It seems unlikely that switching to a different platform would make a difference though. Your camera doesn't know it's taking a dark vs. a light and neither does your capture app. In other words there is no way either the driver nor the software can make a dark brighter than a light unless there's operator error. Either settings are wrong or you're letting light into the camera. Take the camera off the telescope and stick it in a dark room. Take darks and measure them.

OK so this is not a PixInsight problem. We can continue to discuss it here but you're more likely to get camera specific help on the QHY forums. Most likely though you now have all the information you need and you simply need to methodically check your work. As I said, I doubt it's a Mac problem.

[johnw3d]

Thanks again Sander.  I will hop back over to the QHY forum, in fact just re-activated an old topic that addressed the same gain-for-2x2-binning issue.   I do understand how the summed 2x2 well-counts could exceed 64K, but wondering how one should set the gain in that case.   The recommendation in the old topic was to 'reduce the light-source' during calibration exposures, but that seems to me pretty arbitrary, how much do you reduce it and how can that be any kind of calibration?   We'll see what is said.

Re software, it seemed that the power % sent by Nebulosity to hold -20º was much lower than that sent by MaximDL for the same ambient conditions, wondering if sensor-temp readout is buggy, so I will continue to A-B the software for a bit.

Cheers,
John.

[Nocturnal]

Hi John,

a camera has 1 optimal gain setting. At this gain setting the noise is minimized. Really the name of the game in astro photography is signal to noise ratio. Everything we do aims to increase it.

When you bin for one reason only: to increase SNR by digitizing 4 (or more) pixels in one shot. More signal with the same noise means higher SNR. Now if your signal is so high that your ADC can't handle 4 pixels without overflowing there are two things you can do: stop binning or reduce your exposure length. Do not change your gain. The aim is not to squeeze all those electrons into that ADC and throw away precious signal by dividing it (lower gain).

So reconsider why you're binning and if maybe that's not the right thing to do or if you should reduce your exposure length.

It could be that different software makes the cooler behave differently but if you always use the same software this is not a problem. At least it should not cause your darks and lights to mismatch. Of course you're welcome to pursue any angle you like.