Master dark creation- is it acceptable to combine darks with different exposures

[astropixel]

I am seeking an answer as it relates to PI processes, capabilities and limitations.

When creating a master dark per Vicent's tutorial, should all dark frames be of the same exposure length, or is it valid to integrate dark exposures of various durations?

Normally, darks would correspond to light frames using other programs. My understanding is that PI is much more flexible than that.

[Nocturnal]

Hi,

the best darks match your exposure time. Anything else is trickery based on assumptions of linearity. I recommend you do not compromise and take the appropriate darks. Someone else with more experience using the PI calibration tools will tell you how to use PI with non-matching darks.

[astropixel]

Thanks Sander. I guess I'll have to learn how to do that properly, because I've been combining darks with different exposures all along. I suspect that misunderstanding may be the cause behind several issues, including the flat calibration thing that Juan is presently working on.

[Nocturnal]

Hi,

I'm not sure what's there to learn When you take 4 minute exposures, use 4 minute darks All this depends on your camera of course. With many cooled Sony based astro cams you don't need darks as there's no dark current.

[astropixel]

Yes I know. Possibly, my interpretation of this post is incorrect - http://pixinsight.com/forum/index.php?topic=2118.msg13737#msg13737

[astropixel]

Reading more closely - yes, I would like to learn how to combine darks of different exposures. That would be very useful.

[vicent_peris]

Reading more closely - yes, I would like to learn how to combine darks of different exposures. That would be very useful.

Hi,

at first, I don't see any problem with combining darks of different exposures, if all of them have sufficiently long exposure (to say, not completely dominated by read noise).

To be able to integrate darks of different exposure lenght, you must subtract the bias to all of them. Then, to combine them, these are the parameters fo ImageIntegration:

* Image Integration:
- Combination: Average.
- Normalization: Multiplicative, to fit the dark signal of differing exposure lenghts.
- Weights: Don't care.

* Pixel Rejection:
- This depend on the number of frames. Usually, put high clipping values (I use to put 4 sigmas for low and 3 for high), because you want to remove only cosmic rays.
- Normalization: equalize fluxes,  to fit the dark signal of differing exposure lenghts.


That's all... let me know how it works for you, please.
Best regards,
Vicent.

[Juan Conejero]

When creating a master dark per Vicent's tutorial, should all dark frames be of the same exposure length, or is it valid to integrate dark exposures of various durations?

A dark frame is just data acquired with your CCD or DSLR camera. A dark frame gathers dark current signal and read noise, so we are talking about signal and noise after all. So your question reduces to: is it valid to integrate images of different exposures? The answer is yes.

As happens with images in general, however, don't integrate a 15-second dark with a 10-minute one, as this simply doesn't make any sense due to the difference in SNR (the statistical weight of the 15s frame would be close to zero). The 15s to 10m comparison is a bit radical but you get the idea.

What you shouldn't do is integrating darks acquired at different temperatures. That wouldn't be a good idea due to differences in linearity, especially for high values, including hot pixels, which would lead to integration of dissimilar data.

Normally, darks would correspond to light frames using other programs. My understanding is that PI is much more flexible than that.

Our dark optimization algorithm is a purely numerical solution. It does not depend on temperature or exposure time —the ImageCalibration tool ignores those items, in fact. It even does not make any assumption regarding linearity of the data. Our algorithm will always converge to the minimum noise solution in the calibrated image, even at the cost of an inferior correction of hot pixels. We think that achieving better SNR is more important than fixing hot pixels, which can be removed with standard rejection techniques during integration, and in some cases using defect mapping techniques.

The next version of ImageCalibration, due when we finish another important project we're working on now (surprise! ¹), includes a new version of dark optimization that will provide a minimum noise solution without degradation of hot pixel removal. We have been working also on a new algorithm that allows generation of high-SNR bias frames, and a new structure rejection algorithm for ImageIntegration, both using multiscale techniques.

¹ It isn't something related to PI development, such as new tools or features. It is something nice...

[astropixel]

Thanks Vicent and Juan. In general, that is the understanding that I have been coming to, but I had no idea how to properly integrate the images.

The master dark that I use is not properly integrated as a result, and this affects the calibration of the master flat. The Penny has dropped - I see the light!

The new tools sound great.

[yock1960]

What you shouldn't do is integrating darks acquired at different temperatures. That wouldn't be a good idea due to differences in linearity, especially for high values, including hot pixels, which would lead to integration of dissimilar data.

Juan,

I can't/won't argue this point, but given a temperature unregulated camera, how much tolerance is acceptable? My camera reports temperatures to the nearest .5C and my images, be they darks, lights, etc. can vary by several degrees C over the course of a session. There are times that I have gone through the trouble (ie. some calibrated images have obvious & unacceptable calibration flaws) of applying different dark masters to subs from the same imaging run, but there are times that I just don't have a suitable master. I thought that averaging the dark subs would even things out in the end and I have seen an improvement in my own calibrated subs since I went from 5 frame dark masters to 10. I would say, off the top of my head, that maybe 1.5C variation is typically what I limit combining and often it's 1.0C or less.

 I've recently thought about re-doing my dark masters by using darks of the same temperature from different time periods, I'm not sure if say 3 months or whatever time frame it might have been, would make a difference. Darks are a pain! Because I image with my cameras amplifier on, I need good ones to get rid of the amp glow! I'm looking forward to the day when I upgrade to a different camera!

Steve

[Nocturnal]

There is only one way to know for sure: experiment. Take darks at different temperatures and compare them. Most likely you'll find that temperature differences of a few degrees are inconsequential. Of course you could have guessed this as the regulation accuracy isn't very high and if this was really a problem no one would be creating any decent images.

[William McLaughlin]

Well, I have been esthetic imaging since 1993 (when chips were tiny) and I have always found it is better to take darks of the same temp and same length (as the lights) and on the same night. Not to say that the algorithms to correct for time and/or temp do not work, just that they work better when the darks are closer to the lights to start. Despite chips being regulated in temp, there is still some of what is called "coupling to ambient" where the actual chip temperature is not what the readout says (due to the location of the temp sensor). This varies with the make and model of the camera. The result is that if the OAT (outside air temp) is different for darks than lights, the actual chip temperature (and hence the dark current) will also be different. This argues both for using the proper dark creation and application method to reduce this effect but also to take darks and lights as close to the same ambient as possible.

Interestingly, the colder the ambient, the warmer the chip, since the temp sensors are often closer to the "outside" than the actual chip so if the sensor sees a lower temp (due to lower ambient) it causes the cooler to be driven less hard, resulting in a higher chip temp. So the ambient being cooler during darks will give you more "dark" pixels in the dark subtracted result as opposed to more "hot" pixels. This argues for taking darks after lights when the OAT is normally colder (early AM).

Modern cameras are better in this regard but many still exhibit this effect to some degree.