Preprocessing Canon DSLR frames - a different approach

Perhaps the light frames are under sampled in places or the superbias is faulty. It is a 200 frame master bias. I do not have access to the superbias module, with the latest v1.8.
 
Hmmm... if I subtract a master bias from a dark frame I do get some zero value pixels. If I subtract a master bias from a light frame I don't get any zero value pixels - there are some pixels with very low values but no zeroes.
Unless you have a space telescope, it is likely that even the darker parts of your sky is somewhat brighter than the plastic cap of your telescope (in fact, even with HST, as witnessed by the Hubbles Deep Field)  ;). The bias value was probably selected by the manufacturer so that there is no negative value even in the noisiest darker part of an image (even without stacking, as most people do not stack their holiday pictures), unless we do something very extreme (which we may the way we use DLSR). So subtracting bias from an image should usually not cause negative values, but this may depend on the DSLR and other parameters.

Apparently some DSLR are 'partially calibrating' the images so that what it assumes is a dark area is near the bias value (and sometime below due to tolerances, but still above 0).  But the algorithm is probably more subtle than just rebiasing based on the darker pixel of the image, otherwise even the image of a sandy beach under the sun would have a significant dark area.

We try to second guess this algorithm (or amplifier regulation). Is the result really linear and predictable (to a fraction of an ADU) and similar with a wide range of image luminosity, DSLR, temperature, ISO settings, exposure time ... ?  I would like to see some numerical evidences before believing this (but I am ready to change opinion in face of evidence).

I am afraid that we arrive in an area where you may gain something by measuring and testing your DSLR under your conditions and adapting your process accordingly, but it may be difficult to state a rule applicable to everybody.

-- bitli
 
bitli. This exercise was investigatory and intended to generate comment and never intended to establish a rule.

Can I be sure of negative values in my images. Or should I assume that Canon assure values not less than zero in all frames.

I understand that Canon RAW data is modified, but I am not clear about what may be assumed of the data.
 
I am not sure the previous exercise was of much value. Inspection of deBayered frames show pixel values >0.

DSLR RAW data is not exactly linear and therefore I have decided not to apply the usual parameters to calibration frames, treating them as an unknown quantity and applying the same algorithms and rejection parameters across all frames, except flats, which require Multiplicative and Equalize Fluxes - ImageIntegration.

Furthermore, If scaling and weight factors during image integration close to 1 are considered a valid measure of successful calibration, the best method with my cold finger temperature regulated 1000D at -5C, was as follows.

1. Use a Master_dark_with_bias and master_flat to calibrate light frames - this combination produced scaling factors of 1.0 (often 1.00) and weight factors of 0.99; or

2. Use a superbias and master_flat to calibrate light frames - this combination produced scaling factors and weights marginally less than above. This is probably the most convenient method and avoids the need for darks - flats are calibrated with a superbias.

I see no reason to pursue this further. I am satisfied that there is a need to get away from the CCD style and parameters for DSLR calibration. If I have missed something and off on a tangent please say something.

I do however, think that these findings might relieve some of the frustration experienced by some DSLR users. The same calibration/preprocessing exercise could be conducted using a camera that is not cooled. It would be very interesting to compare the results.
 
Sorry! Replying to my own post... and I understand that this is all very preliminary and well, based on broad assumptions about the non-linearity of DSLR data.
The same calibration/preprocessing exercise could be conducted using a camera that is not cooled. It would be very interesting to compare the results.
Well! I did just that and frankly, the best results came from a master_dark_with_bias and flat - just the same as the -5C set. The integration scaling factors and weights were ~1 and better, 1.0 - 0.98 at times. No observable evidence of truncated data.

The effect of subtracting the bias from the darks was truncation. Subtracting the bias and then the dark from the lights produced truncation. As the image set was quite 'hot', dark subtraction is still required, but it seems better to leave the bias in the dark.

I understand that dark scaling is not available using the method described above.

Note: all frames except flats were subject to the same integration parameters - that is, the standard settings.
 
What about different situation when Subs are way underexposed? For example when imaging Globs? Just in time, I have imaged Omega Centauri with 120sec exposures @ F8 @ 2000mm. Cooled DSLR and frames were +5c. Calibrating those frames with Master Bias or Master Dark, equally causes truncated pixels. Going over the calibrated Sub there are many K=0 pixels.
Now, the question is, why we should care about those truncated pixels, since we going to integrate many Subs and they all will get averaged anyway?
I did some tests too, I had all of my Subs calibrated only with Master Bias, then Master Bias and Master Dark that got calibrated during light calibration but without optimization and third set the same but with Optimize option enabled. All of three sets were independently registered to the same frame and integrated with same reference (high SNR) image and no pixel rejecton.

Comparing all of the three final integrated images, the noise levels or median noise reduction factor were very close, almost the same. Stacking and averaging 50 subs produced smooth as silk background with no K=0 pixels. If I would decide to grade the stacks and be precise with numbers, the advantage goes to the stack that were calibrated with Master Bias and Master Dark (calibrated with Bias) and no Optimization enabled.
But again, the difference in numbers were such small, they were almost the same. And if comparing visually, all of three stacks were identical, besides the hot pixels of course, that I think this all not worth the hassle. The case maybe very different with un-cooled DSLR.

From my experience, I can easily get away with no Darks at all, just calibrating with Bias and Flats and get very good results.

 
Hi Sergio.

I'm just curious and would like to improve my image processing. The tests I did were not empirical but observational. However, I was careful not to introduce variables that might skew results when comparing preprocessing methods.

I'm sure that you can get away with bias and flat only - I have done the same with +5C images with good results and, I take your point about averaging.

I mentioned in the first post that scaling factors and weight near 1 was the metric used for testing, if that is desirable? I guess the difference shows up in post processing and I find that suboptimally preprocessed images lack the lustre of properly calibrated images. Since applying the bias_in_the_dark (BITD) method, images that were previously preprocessed using a separate bias and dark are much improved and scaling and weight is scarily close to 1.01 - in some cases 1.001 - is this valid?

A second criteria I used, but did not mention - equally important, is ease of post processing. Was the image easy to stretch and saturate, was noise reduction minimal, was detail easily enhanced and was dynamic range improved - this is true for the images that I was able to reprocess.

I guess the take home message is, that BITD will get you out of trouble with DSLR data, in pretty much most circumstances. A bias is required for flat calibration, so why not forget the darks if the images are cooled.

But here again DSLRs are tricky. One image showed line artifacts with bias only subtraction. These were not present with BITD subtraction - all other parameters and conditions were the same. I reasoned that the act of taking dark frames with CMOS sensors picks up artifacts generated during long exposure of lights (aside from thermal noise), which are not present in the bias.

 
The effects of data truncation could be negligible, I don't really know, possibly in some cases matters more than in others (low SNR areas). But in principle, the average of a truncated stack is biased (shifted from the real average, think of a truncated bell curve). If a simple change in workflow avoids the problem from the get go, I would just do it and forget about the problem altogether.

Now, if the lights are underexposed, that means that the skyglow shot noise statistics will be poorly captured and averaged out in a stack (Poisson-like vs Gaussian-like), aside of calibration, which could make things even worse. Well exposed lights should have no problem with data truncation during calibration.

Ignacio
 
Ignacio. I agree - if it works just do it. Data truncation can be significant with DSLR data and the bias_in_the_dark method avoids the problem from the get go, from what I can tell.

Rereading your post - the biased bell curve, is the idea in the back of my mind, related to truncation.
 
Bias_in_the_dark is a good method only if you have a good temperature match, since you need a bias-subtracted master dark for proper scaling. Besides, you need a good master bias anyway to calibrate the master flat. I rather build a very precise master bias with hundreds of frames (on-time job), an un-calibrated master dark, and a calibrated master flat, and use these to calibrate lights with dark optimization. This is my prefered approach... for the time being.

Ignacio
 
Thank you Ignacio. My DSLR has very accurate temperature regulation. The uncooled frames I used for testing were reasonably consistent, so I guess that eliminates scaling as a variable. I used a 200 frame master bias to calibrate the cooled darks and this also truncated data :eek:

It would seem, in my case, that I need dark and bias frames, but the bias is better left in the dark. I can't eliminate darks altogether because calibration with a master/superbias was not completely effective producing line artifacts in the low SNR regions of the light images, that were not evident until the image was stretched (dithering helped cover bad/hot pixels). Something going on that I do not understand with dark frames, which I think is related to camera firmware. However, BITD is very effective in every way, producing good results even in areas of low SNR, which could be stretched with ease, revealing detail that was previously obliterated.

Kinda think, I've done this to death. OCD!
 
Well, whatever Canon are doing to the Darks I guess they must be doing the same to the lights too, since the camera can't tell the difference...
 
yes this is true - however if things are going well the actual signal exceeds the dark signal by a lot and so whatever thermal noise suppression tricks they are doing are less evident, at least from a statistical view of the data.

when you are looking at *just* dark signal it does not seem to behave "right" with respect to a true 'raw' CCD image.

rob
 
That's a good summary Rob - the same thought crossed my mind. A more empirical approach, perhaps?

I made some very basic assumptions - does the camera know or care if the lens cap is on or off? Does it matter if a lens is attached? A 3 minute exposure is a 3 minute exposure? Sure flux makes a difference to the image, but what about the camera - does flux figure in firmware corrections to RAW "dark" noise.

The manufacturer designed the camera for daylight photography, principally. Leaving the lens cap on for extended periods, with the sensor operating, is not normal operation and something the maker probably does not consider important. Whereas, long exposure noise reduction is an option (as well a low light noise reduction) and the camera knows to take a dark, which includes the bias. Does the camera know it's been hoodwinked - lenscapped?

Some time ago I applied long exposure noise reduction to a dark. I should have documented the outcome. I recall it was not as expected.



 
I suspect that canon firmware has internal lookup tables (or fitted curves), to subtract a constant level when taking long exposures, as a function of exposure duration, ISO, and exif temperature.

Ignacio
 
It looks that way...

Without labouring the issue, the data set from a cooled DSLR (as opposed to supercooled -25C), generally speaking, as follows - given dark scaling is probably not an issue;

Bias - linear (near enough) - convert to fits and integrate

Flats - linear (near enough) - no calibration issues to speak of...

Darks - EXIF temperature possibly not representative of actual imaging temperature; therefore camera firmware correction not representative.

EDIT: More correctly - there is a disparity befween EXIF temperature and sensor temperature, which according to other sources is not a parameter used by camera firmware when modifying RAW data. http://pixinsight.com/forum/index.php?topic=7079.msg47768#msg47768

Bias subtraction causes truncation...

Lights - same as darks; however darks and lights subject to similar firmware corrections; and

Darks and Lights have bias and dark current, very similar. Therefore, calibrate with BITD to avoid accumulating/concatenating errors between bias calibration of darks and dark calibration of lights...

I think this is getting closer to a typical moderately cooled DSLR data set. The process and discussion has been a very helpful learning experience.

Thanks and Cheers

Rowland

I am still to work out the best integration settings for all frames. Up until now, default settings (with individual modification for flats and no noise evaluation for bias and darks) have been assumed, due to unknown state of RAW data.
 
Rowland: one final thought on this. I totally agree with your summery only if you have many darks (say, >30). Then, BITD is a good approach. But if you have only a few darks (say, 10-20), which is often the case, then BITD prevents you from a much better read noise calibration of your lights, which in a cooled camera (more so in super cooled), is the prevailing pattern noise source. So in those case I would definitely take advantage of a better master bias, calibrating the master dark so that data truncation is avoided.

I am done!
Ignacio
 
BITD prevents you from a much better read noise calibration of your lights
Sorry, but I fail to see how this can be possible. 
If you subtract the read pattern (an unwanted signal, not a noise) from the dark, then you decrease its effectiveness in correcting the read pattern from the light, you do not increase it. The read pattern is already part of dark, even if noisy. Maybe I missed something or some steps in your process?
It may be that it is better to use a non noisy bias instead of a dark and all you want is to correct a read pattern - but I have not way to test that method as my darks are different from my bias and I need them. This however depends on the DSLR.

-- bitli

 
bitli said:
BITD prevents you from a much better read noise calibration of your lights
Sorry, but I fail to see how this can be possible. 
If you subtract the read pattern (an unwanted signal, not a noise) from the dark, then you decrease its effectiveness in correcting the read pattern from the light, you do not increase it. The read pattern is already part of dark, even if noisy. Maybe I missed something or some steps in your process?
It may be that it is better to use a non noisy bias instead of a dark and all you want is to correct a read pattern - but I have not way to test that method as my darks are different from my bias and I need them. This however depends on the DSLR.

-- bitli

The point is that BITD approach has embedded bias information in the master dark, which can only be as good as the number of dark frames you have. So using a, say, 200-frame master bias instead of a, say, 20-frame master bias embedded in the (uncalibrated) master dark, will calibrate read pattern "noise" in a better way.

So, instead of (L - D20)/F,  you do ((L-B200) - optimized(D20-B200))/F, where L=light, F=master flat, D20= 20-frame BITD master dark, B200= 200-frame master bias.

Granted that if there is a perfect temp match between lights and darks, then this falls back to BITD, as the B200 terms cancel out.

Ignacio
 
Back
Top