What are the workflow steps to take if only using Lights(modified) and Flats?

[rtemen]

A quick lead in so you know where I am coming from.
Lately, I have been doing a lot of reading of posts on several big astronomy forums.
I have come across two major items.
1. One post explained that today's DSLRs usually subtract the bias information automatically before creating your raw result.  Therefore, he indicates that running the process of removing the bias again is actually hurting the picture quality.
2. Another post points out that my DSLR (Canon T3i) has a feature to enable long exposure noise reduction automatically for each picture.  If this is enabled, I will not need to do some additional pre-processing steps.  His result was that now I would only need the cleaned up Lights and the Flats.
If this is the case, what are the processing steps to go through to pre-process my pics with only the Lights and Flats?

Thanks for any opinions on all of this, and if this is ok to try, what are the steps and in what order.

Rich

[Niall Saunders]

Hi Rich,

Firstly - of course it is "OK to try" - that is part of the fun. Your try something, get results that seem 'reasonable', then you try another approach and either improve or worsen the result. And, you learn from that experience.

Next, be wary of astro-evangelists (with due respect to all faiths and religions). Not all of them (myself included) have built up a big enough wealth of knowledge to be able to state a bunch of commandments that are even remotely adeqyate to suit every occasion - yet plenty of them are always happy to offer you 'tincture of snake oil' when you try to seek their advice!

Bottom line - cover your lens with a lens cap, and take the very shortest of exposures that the camera will allow, at the lowest ISO rating the camera will offer. In fact, to eliminate possible 'one-off' errors, take some 30-odd of these exposures, and then perform the very simplest of ImageIntegration Processes on them (Average Sum, No Normalise, No Noise Evaluation, No PixelRejection, etc.). Now open up the ImageStatistics Process, and select the integrated image.

What is the 'lowest value' (the Minimum) pixel in the image?
What is the 'highest value' (the Maximum) pixel?
What is the 'average value' (the Mean) of all the pixels?
What is the AvDev (the Standard Deviation) value?

If the Bias has been subtracted in the camera, then the lowest value ought to be '0', as should be the case for the average value. And, the standard deviation should also be a very small value as well. The maximum value would indicate that you have 'hot' or 'stuck' pixels.

In your post, step (1) is 'wrong' (remember, beware the evangelist). If the Bias had been successfully removed 'in-camera', then the Biases that you have just taken (and averaged) would be 'zero-value' images anyway. So, if you subtract the zero-value Biases from your in-camera-bias-corrected images, you wouldn;t be changing the data at all 

In the second point that you made - this would be a valid exercise - providing that the Noise Reduction was based on some 'statistical model' of the noise on the sensor. I would tend to suggest (without any detailed knowledge of your, or any other 'consumer' DSLR) that - at best - a single 'dark-frame' is used to noise-correct a single light-frame.

Of course, I may be wrong (I often am - it comes from still not being able to detect snake-oil myself!!). There may well be DSLR cameras out there that 'understand' the statistical nature of noise, especially in the field of astro-imaging, and these would then correctly compensate for the artefacts that I have discussed. However, given that even the most expensive astro-imaging cameras don't work this way, then I would think it highly unlikely that a DSLR that cannot even offer TEC-cooling would be remotely capable of making a good job of producing 'off-the-shelf' calibrated light frames.

There just aren't any shortcuts - there are better ways of acquiring your RAW frames (not least of which would be TEC cooling), but even without TEC you will start off with a better set of data if you take the time to acquire Darks, etc., yourself.

Hope this helps.

[rtemen]

Thanks for all of your info.  I will check your suggestions out.

More info to share with you...
I have received the following info from another PixInsightForum member:
----
He indicates that my Master Dark and MasterBias are debayered stretched RGBs. Therefore they are not usable
for proper calibration. 
Calibration has to be done on the raw mono CR2 files with mono master files. Not with stretched RGB calibration Master files.
The bias and darks are important because you have a very noisy camera with thousand of warm/hot pixels which need to be removed.
----
Wow, all of the tutorials that I have been following have never mentioned that. Again, like you say, how do you know who is correct?

Also, I have heard many opinions about half and half on either side where half the people say that the Biases are to be taken at the lowest ISO, and the other half says they have to match the ISO of your lights.

How do I tell which is correct?  Isn't there a PixInsight Guru that knows the right answers? I would think that the developers of the program would have to know the ultimate answer since they wrote the software to process the files.

The manual for my camera (Canon T3i) does say that the noise reduction is done internally, so there are not 2 files, rather each CR2 has already been cleaned up.  I did a test and made a 3 minute shot with no noise correction and another 3 minute shot with noise correction.
I opened both in PI and stretched them the same.  The one with the noise correction looks much, much better.

So, I will give it a try using IC with only my flats applied to my lights and see what result I get.

Rich

[Niall Saunders]

Hi Rich,

Let's look at some of the statements in your last post :

my Master Dark and MasterBias are debayered stretched RGBs. Therefore they are not usable
for proper calibration . . .

I would have to agree - when you are trying to calibrate images, you want to be working with 'un-processed' data, and certainly not with data that has been processed in-camera using algorithms that are not made available to the user.

. . . noisy camera with thousand of warm/hot pixels which need to be removed

There is nothing particularly 'noisy' about your specific DSLR camera - all cameras (including high-end, multi-$10K imagers) are 'noisy'. The difference comes with how the noise is treated after an exposure is made. In 'our world' of astro-imaging we learn how to categorise the noise and then, with that information, we learn to take the necessary steps to counteract the effects of the noise (note: I didn't say "eliminate the noise" - in fact, that isn't even our ultimate goal). But, your DSLR is not targetted towards astro-imaging, it is a 'simple device' (no matter how much it costs!!) that is targetted at a wide range of users and scenes - including the ability to take long exposures in low-light conditions. But, that ability does not even begin to get close to the long exposures and low-light conditins of an astro-image.

Your DSLR will try to do the best job that it can, but it will ultimately fail - leaving you with an image that still contains noise. The problem is that, now, it is no longer possible to categorise that noise and so treatment cannot proceed in the usual manner (Darks, Biases, Flats, etc.) - if at all.

Biases are to be taken at the lowest ISO . . . / . . . match the ISO of your lights

How to know which is correct? Simple - you have to run an experiment, and determine the answer emprically. Take two groups of 30 Biases, oerform the simplest ImageIntegration on each group, and then look at the resultant Statistical Information for each MasterBias. Of course, if you can figure out a way of getting the DSLR to simply give you the RAW image, that helps a lot!

You can also think of things this way - on many (most? / all?) astroimaging cameras, the user has the facility to be able to set Gain and Offset (personally, I wouldn't buy an imager that didn't give me access to these two crucial parameters). There is a well-established routine that allows a user to determine the optimum Gain and optimum Offset for their imager and, once correctly set, these should remain reasonably stable throughout the life of the imager (aklthough easy enough to check, perhaps tweak, at any time in the future). Once these two parameters have been set, all future images are acquired using these parameters - including all calibration frames.

Now, in simplistic terms, your ISO setting is the equivalent of the Gain parameter, but you have no equivalent (IMHO) of the Offset parameter (which, more or less, determines the base ADU value for a Bias frame). So, in trying to answer your question, I might suggest that you select a fixed ISO (not necessarily the 'highest' figure that your camera can achieve - you can still use the algorithm that CCD imagers rely on) and use that for all of your imaging work, including calibration frame acquisition.

In my earlier post, I said :

at the lowest ISO rating the camera will offer

In my defence, that statement was to try and establish what the DSLR actually used as its 'base level' (I leave you to determine just how much snake-oil I am trying to sell with this statement  )

. . . the noise reduction is done internally, so there are not 2 files . . .

That is also entirely plausible - after all PixInsight itself has a multitude of 'noise-removal' processes in its armoury. But, we come back to the same thing - if you only take a single long-exposure in low-light conditions, then this one-off image might well be improved by letting the DSLR work its magic. However, you then gain nothing by taking further, similar, images with the intention of stacking these using ImageIntegration. Well, that's not strictly true - you do gain something - more noise 

Why? Well it all comes down to understanding how the (tried and trusted) calibration process works - at a statistical  level. There are plenty of good sources of information on that subject - and not too many snake-oil dealers. In summary though, because you have allowed the DSLR to 'change' the source image, you no longer have the raw data that is required to be able to reliably use the normal calibration process.

Again, I hope this helps, and I am happy to stand corrected on any or all of the comments or explanations that I have made.