What is best practice using dark frames with an uncooled DSLR?

[the Elf]

Dear all,

using and understanding PI is one thing and as far as I understand this forum is about the software mainly. Best practice questions may be a bit off topic, but I did not get any usefull replies on that question on another forum which often was very helpfull so far. Thats why I put the question here and hope for some information.

What I do, thinking it is close to the optimum without cooling i.e. with no regulation of temperature:
I have a standard Canon 600D, unmodded, not cooled. I decided to take 5 min subs at ISO 800 whenever possible.
For darks I put the camera in places with different temperature (fridge, freezer, basement) and took hundrets of darks 5 min long, ISO 800. I used the little tool "exiftool(-k -CameraTemperature).exe" to read the cameras temperature tag. I do not know if this is right on the sensor or somewhere else, not if it is measuared at the begining or end of the exposure and not how precise it is, but it is the best guess I have. I sorted all darks by temperature and keep this library on my hard disk.
When I come home from an imaging session I check all lights temparatures with the same tool and start picking as many darks as possible with more or less the same temperature.

Example: when I have 20 lights they may have 10 times 18°, 3 times 20°, 5 times 22° and 2 times 15°. I try to find at least 2 times as many darks, 40 in this case in the range of say 18 to 22° and create a master dark from that.
Is this the best way to deal with the issue of different temperatures of the light frames? Is it better to stack the same temperature only (say all 20°)? Does the optimize option get the same result with darks that do not match the lights in temperature? Could I use a set of 100 darks at 30° instead and PI will scale it somehow?
Any answer is very much appreciated.

the Elf

[pfile]

let me preface this by saying that it's been years since i've used an uncooled DSLR for imaging and indeed it is a pain in the butt. and i probably don't know what current best practices are anymore.

having said that,

one thing to keep in mind is that most DSLRs do some kind of dark signal suppression in the camera firmware, and despite the file format being called "raw" CR2s are anything but - the camera firmware plays all kinds of tricks with the data before writing it to the file. this could make accurate calibration impossible even when you've matched temperature and time.

so getting matching darks for your lights can be very difficult, even if the duration and temperature are the same. furthermore until the camera has reached a steady state temperature, the first few lights/darks are always going to have a non-constant temperature thruout the duration of the frame. IIRC canon cameras sample the temperature at the beginning of the exposure, so if the camera body started out at 10C and ended at 30C after a 30min dark, the CR2 file will say 10C when in fact the true equivalent temperature is somewhere between 10C and 30C. if not a lot of time elapses between frames then over time the EXIF temperature probably becomes more accurate.

a complicating factor here is that the temperature written to the EXIF is not the sensor temperature, but rather the temperature of one of the asics in the camera. as such it's probably a good proxy for the sensor temperature but its possible that if you managed to cool one part of the body more than another that the exif temperature is no longer well correlated with the sensor temperature.

so anyway, i think the best one can hope for is to do what you are doing - group darks by temperature and make masters for each temperature, then match those masters with lights (manually). the PI dark optimzation routine is completely agnostic with respect to dark duration and temperature. all it does is scale the dark until the noise in the calibrated result is minimized and then use that scaling factor. it's an open question whether or not your results will be OK if you use a poorly temperature-matched dark with your lights. uncooled DSLR work is kind of a crap shoot...

imo one way to approach this would be to time and temperature match your darks to lights, then just turn off dark optimization and see what happens. also if you turn on dark optimization, don't forget to calibrate the master dark by subtracting a bias, or nothing's going to work right.


rob

[the Elf]

Rob,

thank you very much for your answer. I am well aware of the DSLR tricks. It looks like they read the optical dark pixels at the long and the short side of the chip and do some dark substraction for rows and columns. It is clearly seen when a gamma ray hits the border and produces a completely black column. I've seen this once in my subs. I am also well aware of the thermal steady state. If will never really happen, because the temperature changes during the night and after 10 shots I stop to check focus and again it is a different temperature.
Well, perhaps I should have introduced myself first. I am an engineer and work in quality management in automotive industry. The last 10 years I've been writing software and developing ccd camera based test machines for either dimensional measuring or photometrical measuring. So my question is not so much about the physics and basics but rather about the tricks you mention. I was hoping out of expericnce someone can tell. Sad thing is, there is never any information available for consumer products. For my industry cams I have data sheets and the values are reliable. (Some use the same sensors as astro cams.)
I somehow expected the temperature sensor is not on the chip. The DSLR is simply not designed for astro. I started this year and rather want to spend the next bunch of money for a premium mount, then the camera might be the next step after that. Having a bad mount is much more pain in the ass from my point of view. But yes, I agree.

So one question remains that is indeed 100% PI related:
I've seen one guys workflow (Light Vortex Astronomy) who uses ImageCalibration to load the raw darks and subtract the master bias. After that he integrates them with ImageIntegration. When working with flats and lights he has the "Calibrate" box under Master Dark unchecked. W.A. Keller in "Inside Pixinsight" starts with integrating darks and later has the "Calibrate" box checked, when working with Lights and Flats. (Both have the optimize box checked.)
Does this achieve the same result? I hope so!

Due to the non linear behaviour of hot or more sensitive pixels I am sure that darks with a larger temperture difference from the lights will ruin the result. My question is rather what is happening with the pixel rejection algorithm when the darks have different temperatues (not more than 3 or 4° in my case). As dark current doubles for a temperature raise of only 6° I try not to stack darks with a high temperture difference (not knowing the real temperture anyway.) Now that you don't say "OMG! You ruin it!" I'll continue my dark matching until I have more information.

Thanks again!
regards

the Elf

[pfile]

i dimly recall some threads here where people were experimenting with normalization of DSLR dark subs during dark master integration. normally one would never do this but i think because of all the DSLR firmware tricks some were considering it.

precalibrating the dark subs and calibrating the uncalibrated dark master should be equivalent, though if for some reason the bias and dark signal are very similar, its possible that the calibrated dark subs end up with some pixels clamped at 0. in this situation you might need to add a pedestal to the calibrated darks.


rob

[bulrichl]

Hi Rob,

precalibrating the dark subs and calibrating the uncalibrated dark master should be equivalent, though if for some reason the bias and dark signal are very similar, its possible that the calibrated dark subs end up with some pixels clamped at 0. in this situation you might need to add a pedestal to the calibrated darks.

did you mean "... you might need to add a pedestal to the UNcalibrated darks."? After calibration, the addition of a pedestal has no effect, because the clipping has occurred already.

I don't precalibrate neither the dark subframes nor the master dark because I observed severe clipping in the calibrated dark subframes and in the calibrated master dark respectively (Canon EOS 600D). PixInsight has the option of calibrating the masterdark on the fly during calibration of the light frames. So I integrate my dark subframes, and in the calibration process I check 'calibrate' and 'optimize' for the master dark. No need to add a pedestal when you do it this way.

The calibration with dark frame optimization will leave some hot pixels in the calibrated light frames. These hot pixels must be corrected with the CosmeticCorrection process.

Bernd

[pfile]

well i should have said added a pedestal during calibration of the darks.

rob

[bulrichl]

Yes. In order to clarify I will say it in other words:

Canon DSLRs (and presumably other DSLRs as well) have a dark suppression mechanism (in the hardware) that subtracts dark current. Thus dark frames and bias frames have similar average intensities, almost independent of the exposure time of the dark frames. Unfortunately long exposure times of the dark frames will also result in a peak broadening in the histogram which cannot be corrected by this suppression mechanism. By the way, this is the reason that Canon must use an excessively large offset (with the EOS 600D the offset is 2048!) which limits the dynamic range.

If you subtract MasterBias from the dark frames (or from the MasterDark) there will be some negative values. The broader the peaks in the dark frames (or the MasterDark) the more negative values will result after subtraction. If you do the subtraction in a preliminary step, negative values are cut off (clipped), and these data are lost.

So the data loss already occurs when you calibrate the dark frames or the MasterDark in a preliminary step. You can recognise that in the histogram of a precalibrated dark frame (or precalibrated MasterDark). In contrast data loss will never occur in the calibration of (sufficiently long exposed) lights with a MasterDark.

If you calibrate the MasterDark during light frame calibration 'on the fly', data loss by clipping of negative values in the intermediate result (the calibrated MasterDark) is avoided.

For that reason I don't understand a bit why in a tutorial covering preprocessing advice is given to precalibrate the dark subframes, e. g.:
http://www.lightvortexastronomy.com/tutorial-pre-processing-calibrating-and-stacking-images-in-pixinsight.html#Section2

Firstly it is not necessary to calibrate single dark subframes (you can calibrate the MasterDark), secondly - and more important - you will get clipping if you precalibrate the dark frames (or the MasterDark).

Bernd

[dld]

Excellent post Bernd,

I have similar concerns as yours. I am a DSLR imager and I have noted (and worried about) the clipping that occurs by following some tutorials. Such tutorials are invaluable for novices like myself. The problem is with the DSLRs which I think they should be treated differently from astro-dedicated CCDs.

Clipping messes a lot with the distributional properties of the bias/dark frames. This may do harm to PI algorithms which, for optimum results, may utilize some reasonable assumptions about the data statistical properties.

I am still experimenting...

[Warhen]

Hey guys, Let me preface by saying Kayron's LVA tutorials are generally excellent, as he's one very bright guy and a customer and friend of mine. I agree with Bernd, I see no reason to calibrate a master dark before calibrating lights or flats, and I think it can induce more harm then good. Whether taking a manual tack, or using the BPP script, I feel it's better to use the pristine master dark, allowing it to be bias-calibrated on the fly, after which optimization can be performed. Remember that since we're bias-calibrating and optimizing the flats when creating a master flat, one would need to be careful not to calibrate the flat master a second time, when calibrating the lights.

This next statement does NOT apply to Kayron or Harry, but I feel that we need to be careful about 'tutorials.' Anyone can post to YouTube. In our arena, as well as my other pursuit- music, I've seen both excellent, and simply awful tuts, and everything in between. Be judicious in who you follow. There is some very erroneous information out there. Stick with people who have earned their credibility and can produce successful images. We've done very careful research for both www.ip4ap.com and the book Inside PixInsight. Thanks!
       

[the Elf]

Hey all,

double reply.
1) @Bernd: exactly! I am seeking information quite a while what exactly Canon is doing there. I have some clue that fits my findings in my images. There is virtually no source for DSLR hardware data except Roger N. Clark. But he is in a different philosphy than I am. I spent quite some time appying a screen stretch to darks and master darks and the same to the bias to make sure the bias has lower values. (I know, use pixel math.) Until now it looks like my Superbias is never higher than any of my master darks but the difference is incredible small. I wish one could turn this off and get real raw. (Now I hear a sentence roaring like thunder from somewhere behind the intergalactical background noise: buy an astro cam you fools or you will be doomed.) Almost nothing in my life is perfect or optimal (one exception, but I won't tell you her name). Nevertheless when I do something I try to be well informed and do it the right way in order not to make it worse than it could be.

2) @Warren and about tutorials: there is a programmers forum called stack overflow (insiders get the joke). Any post or answer is rated. You need to earn a good rating to stay in the forum. Even to beginers it is easy to see if the answer is rated high or low. This should be done with tutorials as well. Alas Amazon is a negative example of rating. Let me preface by saying this does not apply to books people in this conversation wrote. One might think if someone publishes a book it contains information on a certain quality level. When I read books about things I really understand and have expericen with it turns out authors fill pages with black ink intending to generate money. There are plenty of fools who make it work. If it is about how to process photos thats fine, but if someone has suicidal tendencies as he or she is in serious trouble the wrong information might result in a terrible outcome. "Be judicious in who you follow." Agree. Applies to health, religion, partnership tipps, anything.
People know it and youtube is the best example: "Do you know you can charge your ..... phone in 20 seconds? Here is the video: put it in the microwave, select chicken, ...." Oh what a great fun! There are people stupid enough to try anything. (Did you try licking a lantern pole at -40?) Ok, you know what I mean. Had a conversation with Warren today about dumb questions and there is plenty of dump answers. I try not to get into it and still I feel save in the PI forum!

best, the Elf

[bulrichl]

Hi Warren,

I did not intend to put someone down personally. I did not want to imply that the tutorials on this website are generally poor either. But I gave reasons to my point of view and described in detail why I prefer not to precalibrate the dark frames (or the MasterDark), and we don't disagree on this.

You described the crux with tutorials well. "Be judicious in who you follow." I wished that those who write tutorials would also state WHY one approach is recommended, and for what set-up and operating conditions it is applicable. Then our judgement would be much easier.

Bernd

[Warhen]

Fully in agreement Bernd, and I did not misinterpret your comments as negative to anyone. Kayron, if your ears are ringing- excellent tutorials, well-received by many! Elf, funny stuff. :>)

[aeroman]

Hi 'the Elf'

I figured I would respond because I have been going through a similar process of using a DSLR for astrophotography and have begun using PI starting this last September.  I have been doing super wide field astrophotography with a Canon 5d MII camera and a Rokinon 24mm f1.4 lens.  I too am an engineer, but I work on building design and evaluation of construction.  I will be starting a photography business as well next year as I have been doing landscape photography for awhile now and began to take up astrophotography about a year ago.  Due to the wide field astrophotography, I elected to not spend too much money on the mount yet and bought a Celestron Advanced VX when it was on sale.  I can get 5 min subs at 24mm easily with no star trailing using this mount, so it has been perfectly good for my current purposes and have never needed to reject a light frame using it.

I started to try and build a library of darks at certain ambient temperatures, but I have since elected to take a dark frame at the beginning of my imaging session and then a dark frame at the end of my imaging session.  I then average the two frames to create my 'master dark' for that evening's imaging.  I live in a light polluted area, so I travel about 45 minutes to get to a good blue zone area for imaging.  As such, my imaging sessions for capture are only 1 or 2 hours long.  I find that this method has given me darks that are reasonably good for producing good results.

Now I'll venture off topic slightly, but it may assist you in figuring out your methods and processes to this particular question, as well as the rest of your processing workflow.  If you are in a similar situation as me, you started astrophotography with a decent DSLR.  I know this forum is particular to using PI only, but I had another well known program because it is excellent to use for landscape photography (you'll probably guess what that is by now).  When I first started astrophotography, I utilized that program and looked at all the tutorials that I could find online.  I obtained some pretty good results and pushed my images as hard as I could with that program and a couple of plug-ins.  However, seeing the astrophotos produced by Rogelio Bernal Andreo, as well as a processed moon image using PI I found online, I decided to try PI on a trial basis to see if I could get better results.  I did get significantly better results and I purchased PI last month.  You are correct there are various sources for tutorials using PI, and I have had to spend over 100 hours on various processing techniques and methods and have created a workflow that appears to work best for me.  Of course, new additions such as ArcsihnStretch make me need to revise my workflow and go back and reprocess images.  In all of this effort, I think I have learned one important lesson, and it relates to the equipment you buy, the amount of time you spend imaging, and the amount of processing time you are willing to do.  You need to know what exactly you want to do and accomplish with your astrophotos in the end result.

I'll go through my own personal journey as an example of what I have decided is needed at this time.  My engineering business is my own, and I have a family.  In addition to taking landscape photographs, I also have been taking wide field astrophotos and images of the moon.  Due to weather, lunar cycles, family commitments, and business commitments, I am lucky to get one or two imaging sessions in a month, which means I really only get on average 2.0 hours of imaging done per month.  That's only 24 hours of imaging in a year for the wide field astrophotography.  You don't have to be hard core into astrophotography to realize the benefits of using PI for processing; however, you are running with a crowd of users that are pushing the envelopes of serious amateur and professional astrophotography.  They will spend more than 24 hours of imaging one target to be able to pull out faint detail in IFN.  They also typically have cooled dedicated CCD cameras with very expensive mounts and telescopes.  I too have felt the inadequacy of using a DSLR reading the tutorials and looking at the posts, but purchasing that extra equipment now is not possible.  Not to ignore the many other outstanding astrophotographers on this site, I figured that Rogelio Bernal Andreo likely has at least $50k USD in field equipment that he uses, and he is likely dedicated to imaging most nights that he possibly can.  He, like many others on this site using PI, are trying to push every single aspect of imaging to the ultimate limit, so exact minutia is required from image capture, batch processing, and final processing in PI.  For me, that is not possible with other priorities in my life, and I find I can make sellable images with wide field astrophotography without being the best in the world.

From this process, I have found that a hybrid program approach works best for me at this time.  I didn't get any significant improvements trying to introduce a bias frames, so I only use dark frame subtraction.  As such, I create 'calibrated' light frames in 16 bit TIFF format from raw conversion and dark frame subtraction using other programs, then I use star alignment using PI and integrate images from there to get a 64 bit XISF file where I start my processing.  I use a hybrid approach because I could get rid of plane trails much more easily using another program, and my efforts strictly using a PI batch process wasn't getting the pixel rejection results that I wanted.  That is likely my own limitation in knowledge of using PI, but integrating the images as I do, which are upscaled and drizzled takes a moderate time in PI (or any other program), so experimenting multiple times with different variables just gets out of hand.  I know a hybrid program approach is akin to heresy here, but it works best for my needs at this time.  I personally don't think that using multiple programs should be disparaging to PI at all as I think it is a truly fantastic and ground breaking program for what it does and I am very pleased to have it.  So all this is being said to say I try and get excellent results for the time spent, but I have decided not to worry about whether the dark frame was taken at 18C versus the image capture at 20C.  While I have perfectionist tendencies, the extra effort at this time is not worth it to me personally.  This falls under the rule of diminishing returns.  I can get very good wide field images with an hour integration time and 3-4 hours of processing time.  I could spend 100 times more money and integration time/processing time for perhaps an improvement of 2 or 4 times the result.  That would be probably worth it if I wanted multiple APOD awards and to be considered one of the best in the world in astrophotography, but that isn't my goal at this time.  Perhaps I am painting too large of a brush stroke for those who don't have those particular stated goals and do spend those efforts, but I am just going through my own thought process.

I hope this helps, at least on some level.  Best of luck with your astrophotography!

[the Elf]

Hi aeroman,

thank you so much for your very long answer. This is by far the longest text someone has written in a forum to mee (only my best friends write more). Everything you say is in perfect harmony with me and I pretty much feel the same, althoug I am not running my own business and I do not have family in a clasic sense, but this is far off topic. You can indeed do anything and find lots of other who are doing longer, better, investing more time, investing more money and so on. I've been thinking long about the meaning of a hobby, how much time in life to spend for what and so on. The advantage of having work has the disadvantage of not having time. The advantage of living in a quite rich and save country like Germany has the disadvantage of bad central European weather and traffic jams all the time. Everything has got it's to sides. I am working in the automotive industry since 21 years now and quite a lot of it had to do with automation somehow. PLCs (little logic controllers) and PCs make a whole production work as long as each and every part behaves exactly as it should. While the uneducated workes only press some buttons and carry the product away I was responsible for making it go. Now when I read or see videos from the multi APOD winners they have remote PC in the observatory, a control PC in the house, automatic focus, programmed imaging, hundrets of cables, switches, converters and the automatic station is working all night long. Even cloud sensors stop imaging and close the automatic roof. This tastes like work to me. I would like to go without computers, which I hate. I would like to do things manually and have I feeling like I have "made" something, not programmed it. As explained before, I knew that PI would be to my taste and it is clear that it is designed for exactly this purpose. If you try to process a protrait with it, it does not work well. And there is no "one program" philosophy. If it leads to a result that make you happy, go that way! I used DSS before, but I am troubled with the "all settings done" and then wait. I like to see the outcome of a setting soon. I expected the results to be better with PI and they were much better. It is a nice feeling, if something is better than you expect.
I often have 8 or 12 weeks between two sessions. Often the sessions are only a few hours long. But I refuse to measure my hobby by efficiency or the like. It is my way to relax and to find peace with myself. After some frustration in the beginning I am now able to produce something that meets my expectations. This is enough. I don't mind the NASA being better with the HST.
As an add on I meet friendly people here and at Cloudy Nights. Not all are, but I find my way. Some are very friendly and offer any help, like Warhen for example. I found another nice guy at cloudy nights who is already retired and he gives me a clue, what my life might look like in 21 years when I can image any night with no need to get up early. This friendly excange with people also counts a lot. Again thank you very much!

yours
the Elf (I do not use my given name in real life very much. Even my girlfriend calls me Elf and never uses my name.)