DSLR calibration question

[Cheyenne]

Finally was able to get some photons over the weekend with some breaks in the clouds.  Now trying to do a full workflow for processing the images. 

So .. just to make sure that I have the right "gist" of what I need to do, here are some questions.

First, DSS is not an option (using Linux), so I have to use PI to do the full calibration process.

So lets start with what I have.
Canon 20Da raw images (.cr2)

2 Bias frames (1/8000th of a second) - b1 and b2 
2 Dark frames (10 minutes)  - d1 and d2
6 Image frames (10 minutes) - i1...i6
2 flat frames (twilight sky) (1/4 of a second - histogram nicely in the center) f1 and f2

All at ISO 400

I forgot to grab dark frames for the flats (oh well..), and I know that in the long run, more frames of each are really needed (I was dodging clouds Saturday

So..

I assume the following is what I need to do from a mathematical standpoint

b = avg(b1,b2)             - create master bias
d = avg(d1,d2)    - b     - create master dark
f = median(f1,f2) - b     - create master flat (I know that there should be a flat "dark" as well)

for i in {i1...i6}
    i = (i-b-d) / f

As I said I assume that's the math

The question that I have however is what options should I be using to load the actual .cr2 images (i.e. the dcraw options).  Do I need to be concerned with the bayer mask, and if so any hints on how to handle the bayer mask stuff?

for the rest of the processing I assume that running the star alignment against  i images then doing an avg of the registered images to create the "final" calibrated and stacked image.

[Niall Saunders]

Hi Cheyenne,

First - at only 1/4" exposure for your flats, you probably don't actually need FlatDarks - but you still would need to eliminate the 'bias offset' component in the flats, so use your Bias frames as FlatDarks, and see what the result is.

My suggestion would then be :-

Median(b1,b2) to give a (simulated) MasterFlatDark (well, average() might actually be better, because median() really benefits from a lot more statistical data than just two images !!)

Subtract (PixelMath) your MasterFlatDark from each Flat, keeping the resultant individually corrected images

Average all the corrected Flats to give you a MasterFlat

'Normalise' this MasterFlat (using PixelMath again) such that you 'multiply' the MasterFlat image by the constant ( 1.0000 / med(MasterFlat) ) --- this will set the 'middle' of the image range to '1.000', ready for the MasterFlat to be divided into your lights. HOWEVER, you must NOT 'enable' the 'rescale' facility. You NEED the image to contain values that are both 'above' (i.e. 'brighter') AND 'below' (i.e. 'dimmer') than the 'median' value of 1.00000.

Median (d1,d2) to give a MasterDark (again, you would be better off using average() with only two darks to play with)

Subtract the MasterDark (again, using PixelMath) from each of your six Lights, keeping the six results

Use PixelMath to DIVIDE each of your dark-subtracted Lights by the common NormalisedMasterFlat you created a few steps ago

Finally (assuming I haven't 'lost' you all together  ) take the six dark-subtracted and flat-divided Lights and deBayer them - this (IMHO) is the earliest stage of the process that your images should be converted from their 'RAW' format. If you deBayer any ealier in the process you are just introducing more and more unecessary 'noise' to your data, simply because EVERY time that you deBayer, you ARE 'making assumptions', and assumptions (or guessing) === noise.

Once deBayered, the six (now RGB) images should be 'StarAligned' ready for stacking. If you StarAlign any earlier in the process, you WILL lose the critical alignment between your image data and the Bayer array - perhaps not completely, but enough to introduce more noise than is necessary.

And then you can just apply a simple 'average combine' to your 6, aligned, RGB, images - hopefully resulting in a final image ready for the rest of the wonder of PI to be applied 

Remember, this is just how 'I' would tackle the workflow - others may offer different methods, and - like you - I will be happy to be shown where I might be going wrong, and to have my error explained.

HTH

[Cheyenne]

I understand that statistically I need more data.. (next time I'm out , and you did not loose me in your description

But I still have a question on the RAW format options that I should use when loading the CR2 images and then the final de-bayer step.

So I assume that in the RAW Format preferences (via Format Explorer) I need to set the "Create RAW Bayer picture (no interpolation, no debayerization)" and do not select any other options.

Do all the above "fancy math stuff"

Then use the debayer process with the final image?

[georg.viehoever]

Hi Cheyenne,

did you try DeepSkyStacker http://deepskystacker.free.fr/english/index.html ? It will do most of what you need automatically, works well with PI, and is free.


Georg

[Niall Saunders]

Hi again Cheyenne,

Sorry - I will have to let others step in to help you. I have never used DSLR images with PI, raw or otherwise, so it wouldn't be very helpful for me to try and comment on a process that I am not familiar with.

===========
Although, that said, I am now editing my initial post after having re-read your question several times. I think that what you are proposing is actually what I would have been looking to try myself.

Basically, if you can get your 'raw' image loaded such that it remains as a 'greyscale' (monochrome) image, then it has NOT been deBayered. The image should look as if you are viewing it through a 'mesh' - this is the effect of the 'CFA grid' that you are seeing. If you do nothing else at this point, you chould be able to manually apply the deBayering process and recover the colour RGB data from the RAW image. If that works, then yes, all you would need to do is to apply the steps that I described - and hopefully get a useable result.
===========

However, Georg's comments about DSS are also very valid - many of us are using DSS wjilst a similar process is developed to work inside PI itself.

Cheers,

[georg.viehoever]

Cheyenne,

if you absolutely want to do this in PI (some have reported results that are superior to DeepSkyStacker):

- I am not sure if the Format Explorer option "Create RAW Bayer picture (no interpolation, no debayerization)"  will really do what you want. It still splits the image into Red, Green and Blue pixels, but with the original Bayer pattern.
- I believe what you would need is to load the image is the dcraw option "-D", which unfortunately is not (yet) accessible from PI. "-D" would give a a one channel image what your CCD chip sees. I believe that Juan has plans to make it available with the next release.


Georg

[Juan Conejero]

Hi Georg,

I believe what you would need is to load the image is the dcraw option "-D", which unfortunately is not (yet) accessible from PI. "-D" would give a a one channel image what your CCD chip sees. I believe that Juan has plans to make it available with the next release.

I'll try to add this option to the next version of the DSLR_RAW module. It isn't as easy at it seems though, because it breaks some assumptions made in the module (e.g., that DSLR raw images are always RGB images), but of course it is doable.

[Cheyenne]

DSS is not an option for me.  I run only linux and DSS does not function under linux.  Wine is missing the GDI Plus functionality that is required by DSS.  I have tried (every now and then I will load a new version of Wine to see if it works... but no success yet).

I "discovered" that the debayer process requires a monochrome image.  

So here were the steps that I did:

• Set the RAW format option to not debayer --> this did result in a "RGB" image (looking at the process console output dcraw was invoked with the -D option
• did all the "fancy math stuff" of calibrating the images
• converted the calibrated images to greyscale (and wondering how much data I just lost or messed up)
• debayer process
• star alignment
• avg combined the images
• and finally did all the other fun image processing that PI provides

So.. a couple of "final" questions (ha!)

In all the pixmath steps, should I be rescaling the results, I know that to create the MasterFlat to not rescale, but I'm just wondering about the other steps.

Should I convert the images  to greyscale as the very very first step and should I do that for all files (the flats, darks, etc.) being loaded?  (Yes -- having an option to the RAW format to not debayer the image and produce a monochrome image would be helpful)

Anyway... attached is the output of a quick process on M13 (in all it's lossly jpeg 148k glory).  I've been focusing more on the calibration workflow and didn't spend too much time with cleaning up the image (yet).
 

[Cheyenne]

Hi Georg,

I believe what you would need is to load the image is the dcraw option "-D", which unfortunately is not (yet) accessible from PI. "-D" would give a a one channel image what your CCD chip sees. I believe that Juan has plans to make it available with the next release.

I'll try to add this option to the next version of the DSLR_RAW module. It isn't as easy at it seems though, because it breaks some assumptions made in the module (e.g., that DSLR raw images are always RGB images), but of course it is doable.

Having that option would be very helpful.  dcraw is being invoked with the -D option (verified by looking at the process console output) when the "Create RAW Bayer picture (no interpolation, no debayerization)" selection is created.

Basically, if you can get your 'raw' image loaded such that it remains as a 'greyscale' (monochrome) image, then it has NOT been deBayered. The image should look as if you are viewing it through a 'mesh' - this is the effect of the 'CFA grid' that you are seeing. If you do nothing else at this point, you chould be able to manually apply the deBayering process and recover the colour RGB data from the RAW image. If that works, then yes, all you would need to do is to apply the steps that I described - and hopefully get a useable result.

Exactly what I'm looking for...

[mmirot]

Cheyenne,


Search for some of the free professional academic soft for Linux.  It might be easier to used than manually doing all cal operation in PI.
Calibration is often called image reduction. You need to generated debayered fits images 1st.

Here is a start

http://iraf.noao.edu/

Max

[Cheyenne]

I've played around with iraf.  Biggest problem I have with iraf is getting the initial "ah ha" moment in getting images from raw format into the proper format that iraf wants (I know it wants fits format).

Anyway -- the PI procedure hasn't been too bad to work with. 

Once I get things figured out, I can create a script for doing all the "hard work"

[mmirot]

Once I get things figured out, I can create a script for doing all the "hard work"

Good to have the skills.
Hopefully we will get a real calibration module in a few versions.
PI does a great job with integration and a registration.
 It make sense to me to complete the tool set.

Max

[Cheyenne]

Okay... I think I got the "right" process.  The part that has been tripping me up was reading the RAW files directly with PI.  Colors just were not coming out right.

Anyway.. my "process" is to natively run dcraw against all images, and during all the pixmath to not rescale the results.

So.. here are my notes

for i in *.cr2; do dcraw -T -4 -D $i; done  convert to 16 bit tiff

In PI,

• reset tiff and fits file preferences to default
• read all the biases, median combine in pixmath (do not rescale), create a new image bias
• read all the flat darks, pixmath subtract the bias and median combine them (median( fd1-bias,...,fdn-bias)), do not rescale, create new image flatdark
• read all the flats, pixmath subtract the bias and the flatdark and median combine them (median( f1-bias-flatdark,...,fn-bias-flatdark)), do not rescale, create new image flat
• read all the darks, pixmath subtract the bias and median combine them (median( d1-bias,...,dn-bias)), do not rescale, create new image dark
• read all images, pixmath subtract bias and dark, divide by normalized flat ( ($T-bias-dark)/( flat * (1.0000/med(flat))) )
• debayer and save each image as a fits file in calibrated directory

Use staralignment tool and image integration tool to pull in the images and process using the many examples

[Cheyenne]

So here is M13 again in all of it's jpeg lossy compression...  I'm fairly happy with it considering the data that I was working with.

Canon 20Da, 4 x 5 minute, ISO 400, Takahashi TOA130


http://picasaweb.google.com/lh/photo/Sbg9FaJlC-Te3x8uwDSgCg?feat=directlink



(changed to link to larger image)

[Harry page]

Hi

Well done on getting your work out, hope you have your head round it now as you lost me ages ago

Hdr wavelets will do wonders for your m13 and reveal many more stars


Harry