Image Calibration - best practice? And a question

[joelshort]

Still learning PI and up until now I've been using the BatchPreProcessing script to calibrate images, then ImageIntegration to create the master light channels.  One note, since I'm using a rather low noise camera I have not been using darks but rather CosmeticCorrection within the BPP script (more on this later). 

It has recently been suggested to me to create master Bias, Dark, and Flat images and then use those for ImageCalibration.  One question I have is, what's the point of the BatchPreProcessing script if the best practice is to calibrate and stack using different scripts? 

Another question I have concerns "dark optimization" vs. CosmeticCorrection.  As mentioned I have been using the BPP script without darks but using a CosmeticCorrection process icon for light calibration.  If I use a master dark instead and select "optimize dark frames" does that essentially do the same thing as CosmeticCorrection?  If I understand "optimization" correctly, that means the BPP script or the ImageCalibration script will scale the master dark frame to match the light (or flat) frame exposure. i.e if the master dark is 20min and the light exposures are 10min "optimization" means that the master dark will be scaled.

[Dimitris Platis]

I seriously doubt the Cosmetic correction does the same thing as Dark subtraction. In fact I am sure.....
Use Darks, Bias and Flats....and u will see a huge difference in your ability to extract info from ur image.
Secondly, I use suggest against using optimized Darks. Make a Dark library with he appropriate dark and use that. Dont optimize...use the real thing

[joelshort]

This:
http://www.pixinsight.com/tutorials/master-frames/index.html

would seem to suggest the opposite regarding dark scaling.  The reason I was using cosmeticcorrection is because I am using a low noise camera and using darks can actually add noise to the images, or so I thought.

[Dimitris Platis]

This:
http://www.pixinsight.com/tutorials/master-frames/index.html

would seem to suggest the opposite regarding dark scaling.  The reason I was using cosmeticcorrection is because I am using a low noise camera and using darks can actually add noise to the images, or so I thought.

First of all, Darks might add noise to the image but they also subtract noise...a lot of it. You can compare the average ADU before and after Dark subtraction. In combination with dithering they are a lethal weapon. If you have set your camera to low temp (-20C) you cab forgo with Darks...but use dithering.
Now, about scaling....there is no algorithm in the world that can do dark scaling perfectly....so unless u cant be bothered with making darks of the same exposure time as ur lights....dont do darks at all. I mean....u really need probably 5 and 10min darks. Make a library once every 4-6 months and u r OK.
Documentation is one thing...but nothing beats experience.

[RickS]

Still learning PI and up until now I've been using the BatchPreProcessing script to calibrate images, then ImageIntegration to create the master light channels.  One note, since I'm using a rather low noise camera I have not been using darks but rather CosmeticCorrection within the BPP script (more on this later). 

It has recently been suggested to me to create master Bias, Dark, and Flat images and then use those for ImageCalibration.  One question I have is, what's the point of the BatchPreProcessing script if the best practice is to calibrate and stack using different scripts? 

I don't understand why you'd be using different scripts?  Why not create the masters with BPP and the reuse them with future BPP invocations.  That's what I do.

Another question I have concerns "dark optimization" vs. CosmeticCorrection.  As mentioned I have been using the BPP script without darks but using a CosmeticCorrection process icon for light calibration.  If I use a master dark instead and select "optimize dark frames" does that essentially do the same thing as CosmeticCorrection?  If I understand "optimization" correctly, that means the BPP script or the ImageCalibration script will scale the master dark frame to match the light (or flat) frame exposure. i.e if the master dark is 20min and the light exposures are 10min "optimization" means that the master dark will be scaled.

Dark optimization does scaling in PI but it's a bit more sophisticated than a simple time based ratio.  It scales the master dark in order to maximize SNR in the calibrated frames.  It has generally worked well for me.  The only exception that I have found is with RBI flooding where it was necessary to turn optimization off to get pattern noise from the pre-flash to calibrate out.  I normally do CosmeticCorrection as well as full calibration because I have a noisy sensor and the hot pixels don't seem to behave predictably.

Documentation is one thing...but nothing beats experience.

Well, Physics and Mathematics are pretty powerful too 

[Dimitris Platis]

Physics and Mathematics in this case do not include the various peculiarities of different cameras.....that is why experience is more important

[joelshort]

I don't understand why you'd be using different scripts?  Why not create the masters with BPP and the reuse them with future BPP invocations.  That's what I do.

That's the whole point of this thread Rick.  I've got different people telling me different things and I'm trying to learn what the best practice is (in general or for my system specifically). 

One set of people tell me that BPP is just a quick and dirty process to give an idea of what the data is like, but it shouldn't be used for actual creation of masters because it doesn't give full control and other scripts give more control.  Another set of people tell me that they use BPP for everything including image integration, which I find curious because the BPP script even has a warning about not using it for image integration.

In the link provided above, the PI document written by Vincent shows how to create master bias, flat and dark images - separately using ImageIntegration and ImageCalibration.  Is that document obsolete, written before the BPP script was available? 

Here's my summary of what the "don't use BPP" camp says:
1.   ImageIntegration - Create master Bias if necessary
2.   ImageIntegration - Create master Dark if necessary
3.   ImageCalibration & ImageIntegration - Create master Flats (using master bias and dark)
4.   ImageCalibration - Calibrate light frames with calibration masters
5.   StarAlignment - Choose the same reference image for all channels
6.   ImageIntegration - Stack calibrated light frames
        a.  Choose best image within each channel for proper normalization
        b.  Adjust rejection parameters based on number of images

[chris.bailey]

Here's my summary of what the "don't use BPP" camp says:
1.   ImageIntegration - Create master Bias if necessary
2.   ImageIntegration - Create master Dark if necessary
3.   ImageCalibration & ImageIntegration - Create master Flats (using master bias and dark)
4.   ImageCalibration - Calibrate light frames with calibration masters
5.   StarAlignment - Choose the same reference image for all channels
6.   ImageIntegration - Stack calibrated light frames
        a.  Choose best image within each channel for proper normalization
        b.  Adjust rejection parameters based on number of images

I would add to that

[4a) Defect Map - One of my Mono CCD's has some clusters of hot pixels and a defect map is the best way of dealing with them]

4b) Cosmetic Correction (In my case I have a stubborn column defect that calibration does not consistently deal with).

I believe that is pretty much what the BPP script does but under control of a script. I still do things manually step by step -

1) because thats a habit I have gotten into and prefer to review a few images at each stage

2) By the time I set up all the parameters of the BPP script, I could have done it manually

3) I prefer a different system of filing the images at various stages

Regarding Dark Scaling/Optimisation, I find it works pretty well for my Mono CCD camera images but not so well with OSC cameras.

Best practice will always be Calibrate, Align and Stack but there are subtle variations to that which will depend on the camera used and how images are acquired. If I don't dither I throw every tool available at images prior to alignment

[RickS]

I don't understand why you'd be using different scripts?  Why not create the masters with BPP and the reuse them with future BPP invocations.  That's what I do.

That's the whole point of this thread Rick.  I've got different people telling me different things and I'm trying to learn what the best practice is (in general or for my system specifically). 

One set of people tell me that BPP is just a quick and dirty process to give an idea of what the data is like, but it shouldn't be used for actual creation of masters because it doesn't give full control and other scripts give more control.  Another set of people tell me that they use BPP for everything including image integration, which I find curious because the BPP script even has a warning about not using it for image integration.

In the link provided above, the PI document written by Vincent shows how to create master bias, flat and dark images - separately using ImageIntegration and ImageCalibration.  Is that document obsolete, written before the BPP script was available? 

Joel, wrt making final master lights then I agree that you'd only do this in BPP as a quick & dirty first approximation.  I spend a lot of time after using BPP for calibration and registration doing test integrations to get the best combination of rejection algorithm and parameters to optimize the SNR of my master lights.  For making master bias, darks and flats and doing calibration and registration I find BPP perfectly adequate.

The article by Vicent predates the BPP script.  I used to do my calibration manually with a workflow based on that article.  I get results just as good and with less effort using BPP for the grunt work.

Physics and Mathematics in this case do not include the various peculiarities of different cameras.....that is why experience is more important

You can measure all the important parameters of your specific camera quite easily.  There's value in experience and experimentation but it is enhanced greatly by understanding a little theory and doing some simple calculations.

[CharlesW]

David Ault just produced this extremely thorough manual integration tutorial. It might take you a second to figure out the "weighting" in Subframe Selector but everything else is very simple. Follow it line by line and you should be okay. http://trappedphotons.com/blog/?p=693

[RickS]

David Ault just produced this extremely thorough manual integration tutorial.

Some useful stuff there although I think some of it is a bit masochistic (as I mentioned earlier, I used to do this all manually myself and BPP does take out much of the drudgery.)  There are a couple of important omissions.  I couldn't see that there was a process for picking a good quality sub as the alignment reference, but perhaps I missed that?  The big issue for me is that the integration process itself was just Winsorized Sigma Clipping with rejection parameters of 4 (low) and 3 (high.)  Best practice (IMO) is to pick appropriate rejection algorithm(s) based on the number of subs and then tune the rejection parameters to maximize SNR while rejecting the stuff you don't want.  That's a hugely important step that is currently missing from the tutorial.  It takes a lot of time and effort to capture the data.  It's worth spending a bit of time on the integration the optimize the result.

Cheers,
Rick.

[Geoff]

Best practice (IMO) is to pick appropriate rejection algorithm(s) based on the number of subs and then tune the rejection parameters to maximize SNR while rejecting the stuff you don't want.  That's a hugely important step that is currently missing from the tutorial.  It takes a lot of time and effort to capture the data.  It's worth spending a bit of time on the integration the optimize the result.

Cheers,
Rick.

All good advice from Rick. This is where you heed the BPP script warning about integrating your lights and switch to a more hands-on approach. Jordi Gallego has done a very nice power point presentation:
"Image integration techniques: Increasing SNR and outlier rejection with PixInsight" which can be downloaded here http://www.astrosurf.com/jordigallego/articles.html.  Well worth reading.
Geoff
Geoff

[joelshort]

OK, so it looks to me like the BPP script includes all the same parameters as the ImageIntegration script, with the exception of no normalization.  But normalization is not necessary for master bias and master dark creation.

However when creating master flats, according to David Ault above, normalization of flats is necessary and that's not possible to do with BPP. 

Another question I have regards the sigma rejection parameters.  As Rick mentioned the default values are 4 (low) and 3 (high).  David Ault mentions "I may alter the sigma rejection parameters based on the noise and number of dark frames I have."  My question is how does one know where to set these sigma values?  I always have 30 flats, 30 darks and 100 bias for calibration, and usually 12-25 lights.  So I know which rejection parameter to choose based on the number of subs, but that still leaves me in the dark regarding the sigma values based on the noise. 

I'll get to the PPT presentation on image integration after a bit.  Right now I'm just focusing on proper calibration frames, bias, flat and dark.

[RickS]

OK, so it looks to me like the BPP script includes all the same parameters as the ImageIntegration script, with the exception of no normalization.  But normalization is not necessary for master bias and master dark creation.

However when creating master flats, according to David Ault above, normalization of flats is necessary and that's not possible to do with BPP.

BPP doesn't expose the normalization options but it sets them appropriately for each type of frame.  Here's the source code.  Even if you don't read JavaScript it should be fairly obvious what it is doing.

   switch ( imageType )
   {
   case ImageType.LIGHT:
      II.normalization          = ImageIntegration.prototype.AdditiveWithScaling;
      II.rejectionNormalization = ImageIntegration.prototype.Scale;
      II.weightScale            = ImageIntegration.prototype.WeightScale_IKSS;
      break;
   case ImageType.FLAT:
      II.normalization          = ImageIntegration.prototype.Multiplicative;
      II.rejectionNormalization = ImageIntegration.prototype.EqualizeFluxes;
      II.weightScale            = ImageIntegration.prototype.WeightScale_IKSS;
      break;
   default:
      II.normalization          = ImageIntegration.prototype.NoNormalization;
      II.rejectionNormalization = ImageIntegration.prototype.NoRejectionNormalization;
      II.weightScale            = ImageIntegration.prototype.WeightScale_MAD;
      break;

Another question I have regards the sigma rejection parameters.  As Rick mentioned the default values are 4 (low) and 3 (high).  David Ault mentions "I may alter the sigma rejection parameters based on the noise and number of dark frames I have."  My question is how does one know where to set these sigma values?  I always have 30 flats, 30 darks and 100 bias for calibration, and usually 12-25 lights.  So I know which rejection parameter to choose based on the number of subs, but that still leaves me in the dark regarding the sigma values based on the noise. 

I've always found the defaults adequate for calibration frames but I check the rejection percentages.  I like to see there is some rejection going on (and know that outliers are being removed) and I don't like to see too much (which will be reducing SNR.)  I also inspect the rejection maps.  For bias and dark frames you'd expect to see a random speckle with maybe the odd artifact from a cosmic ray.

For integrating lights I use a method roughly as described in the PPT that Geoff shared.  I do an integration with no rejection to determine the best noise reduction possible (see the median noise reduction value printed on the process console by ImageIntegration.)  Then I pick one or more suitable integration algorithms depending on how many subs I have, e.g. percentile clipping if it is only a few and Winsorized and Linear Fit if I have many.  Using those algorithms I adjust the rejection parameters and inspect the results carefully trying to maximize the median noise reduction without leaving behind any artifacts that should be rejected.  Sometimes it's possible to get an improvement by trying a few different scale estimators as well (K sigma, MAD and AAD).  It's a bit painstaking, but it's usually possible to get within a percent or two of the maximum SNR and still have a clean integration.

Cheers,
Rick.

[joelshort]

That's all good to know Rick (and others!).  Thanks very much.  I think it's starting to come together in my head.   

Now another question: within the BPP script, is it better to use master bias, master dark and master flat for calibration, or use all the raw bias, dark and flats?  Why or why not, or doesn't it matter?  Obviously I can create the master bias and master dark and keep those on hand for future BPP runs and I assume that BPP will calibrate faster this way, but is it best practice to use masters instead of raw subs?