Author Topic: Requesting comments or suggestions on my calibration workflow  (Read 3126 times)

Offline Duncan

  • Member
  • *
  • Posts: 53
Requesting comments or suggestions on the calibration workflow that I’m using right now. I’ve tried a few different process variants, based on reading a number of different threads here and some tutorials in various places. I didn’t keep notes on where I got everything from, unfortunately. What I have right now seems to work very well; I get some very clean frames with no apparent residual vignetting. Nonetheless, I’d really appreciate any commentary on my current methodology and whether there are any improvements I should consider.

The basic process is this:
Integrate bias sets
Integrate dark sets
Integrate flat sets
(I’m applying no calibration to darks or flats prior to integration)
Calibrate lights using the masters created above with (mostly) all calibration/optimization checkboxes checked.

My light frames use either 1x1 binning (for L) or 2x2 binning (for RGB) so I have 2 sets of each of bias and darks, and the flats for each filter have the corresponding binning. My primary camera is a QSI583 which has a very stable regulated TEC, and I standardize the temperature set point at -10C which I can reliably hit under all conditions.

For integration I use Average combining with no weighting and no normalization. For each set, I do a first run with no pixel exclusion and save it. I then do a second run with an exclusion algorithm. Winsorized clipping seems to work well (I have usually 50 frames in each set). The dark frames are 10 minutes each, so I can get a pretty large number of cosmic ray artifacts. I take a pretty good look at the high exclusion image after integration to see that I’m getting rid of the cosmic ray artifacts and as little as possible else. I’ve also been using the blink tool to compare the excluded version with the saved non-excluded first run, which seems to give a nice indication of whether I have the exclusion set up right. A sigma of 4.0 seems to work most of the time, but I’ve used 3.5 on some sets.

For the calibration of lights, I set calibration off for the bias frames (there’s no overscan region set). I set calibration and optimization on for darks, and calibration on for flats.

The result of all this looks really good (I do subsequently run cosmetic correction on lights also).

I do usually see a warning that no dark subtraction was performed on the flats due to no correlation of noise evaluation. I’m assuming this is ok; the flats are taken using a flat box as a source which is quite bright and the exposures therefore quite short, so I’m assuming that any dark noise type artifacts are just being overwhelmed by shot noise so there’s nothing useful to correct for with dark subtraction.

Long post, I know, but does this seem sensible? Anything I should consider changing? Any feedback/suggestions/constructive criticism very much appreciated…

Duncan

Offline pfile

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 4729
Re: Requesting comments or suggestions on my calibration workflow
« Reply #1 on: 2017 January 21 00:12:11 »
for the flats you should use multiplicative integration normalization and "equalize fluxes" for pixel rejection normalization.

i just calibrate my flat subs with a master bias since they are short. this avoids the 'no correlation' warning. if my flats are long (narrowband) then i either scale my 1800s master dark or get some darks that match the flat duration.

other than that, it sounds good to me. i always calibrated my dark subs with the master bias and then only tick 'optimize' in the dark area while calibrating lights but i think either way is fine. there may be some advantage to doing it how you are doing it if your darks are short and so ADU-wise are not much above the bias level. in that case subtracting the master bias from the dark subs may result in some pixels being clamped at 0 in the calibrated dark subs. i usually use 1800s darks though so the values are well above the bias even at -30C.

rob

Offline Duncan

  • Member
  • *
  • Posts: 53
Re: Requesting comments or suggestions on my calibration workflow
« Reply #2 on: 2017 January 21 00:30:01 »
for the flats you should use multiplicative integration normalization and "equalize fluxes" for pixel rejection normalization.

Thanks - I will definitely try that. I see that as recommended in the reference documentation, but I haven't done that so far. My more recent flats are captured with a flat box that has good power regulation, so may have been masking any issues by my using no normalization.

Quote
i just calibrate my flat subs with a master bias since they are short. this avoids the 'no correlation' warning. if my flats are long (narrowband) then i either scale my 1800s master dark or get some darks that match the flat duration.

Ok, that makes sense.

Quote
other than that, it sounds good to me. i always calibrated my dark subs with the master bias and then only tick 'optimize' in the dark area while calibrating lights but i think either way is fine. there may be some advantage to doing it how you are doing it if your darks are short and so ADU-wise are not much above the bias level. in that case subtracting the master bias from the dark subs may result in some pixels being clamped at 0 in the calibrated dark subs. i usually use 1800s darks though so the values are well above the bias even at -30C.

rob

Great - thanks for taking the time to respond. That's all really useful information

Duncan

Offline Duncan

  • Member
  • *
  • Posts: 53
Re: Requesting comments or suggestions on my calibration workflow
« Reply #3 on: 2017 January 21 09:45:04 »
One additional quick question:

If I use multiplicative normalization during flat integration, then I need to calibrate the flat subs first to remove the bias pedestal, yes?

Many thanks for the help

Duncan

Offline Duncan

  • Member
  • *
  • Posts: 53
Re: Requesting comments or suggestions on my calibration workflow
« Reply #4 on: 2017 January 23 23:02:22 »
So I've gone through a few data sets with various options, and I think I understand how everything works. Not that I didn't have a workflow that was working right before, but I want to make sure I'm not missing anything...

Since my flats are from a regulated flat-box, the brightness is very even across the set, and it turns out not to matter if I use normalization during the flat combine or not.

I ran two different methodologies on my data: one just integrating each of the bias, darks and flats, and then doing light frame calibration with calibration boxes checked for both darks and flats, and the second pre-calibrating flat subs, doing flat integration with multiplicative normalization & equalize fluxes for rejection normalization, and then calibrating lights with no flat calibration.

Results: I couldn't see any difference between the two in the resulting light frames. I used the blink tool to compare corresponding frames and couldn't see any difference at all (and I spent quite a bit of time looking).

After that I used pixel math to subtract the corresponding pairs of frames from each other (adding in a small pedestal value to make sure the values didn't go negative). The resulting difference was a *very* low level of random noise. So there was a numerical difference between the two methodologies, but nothing that was meaningful.

This is probably going to be different when I have some flats that are more variable (and I have some in my archived data that are - I'll try one of those sets next). But I'm happy that I have a better understanding of how the processes work as a result of some tinkering... thanks to pfile/rob for the pointers.

Duncan

Offline MikeOates

  • PixInsight Addict
  • ***
  • Posts: 278
Re: Requesting comments or suggestions on my calibration workflow
« Reply #5 on: 2017 January 25 07:05:11 »
Duncan,


If I use multiplicative normalization during flat integration, then I need to calibrate the flat subs first to remove the bias pedestal, yes?

Yes, well that's how I do it.

I highly recommend looking at the following pages regarding settings for calibration and general ImageIntegration…

Master Calibration Frames: Acquisition and Processing
http://www.pixinsight.com/tutorials/master-frames/index.html

This second link is more detailed and it helps if you are good at math, which I am not. :(

ImageIntegration
http://www.pixinsight.com/doc/tools/ImageIntegration/ImageIntegration.html

Hope they help,

Mike

Offline Geoff

  • PixInsight Padawan
  • ****
  • Posts: 908
Re: Requesting comments or suggestions on my calibration workflow
« Reply #6 on: 2017 January 25 17:42:38 »
Why not just use the BatchPrepocessing script until you are more familiar with things.  It is very simple to use.
Don't panic! (Douglas Adams)
Astrobin page at http://www.astrobin.com/users/Geoff/
Webpage (under construction) http://geoffsastro.smugmug.com/

Offline Duncan

  • Member
  • *
  • Posts: 53
Re: Requesting comments or suggestions on my calibration workflow
« Reply #7 on: 2017 January 26 14:53:35 »
Duncan,


If I use multiplicative normalization during flat integration, then I need to calibrate the flat subs first to remove the bias pedestal, yes?

Yes, well that's how I do it.

thanks!


Quote
I highly recommend looking at the following pages regarding settings for calibration and general ImageIntegration…

Master Calibration Frames: Acquisition and Processing
http://www.pixinsight.com/tutorials/master-frames/index.html

This second link is more detailed and it helps if you are good at math, which I am not. :(

ImageIntegration
http://www.pixinsight.com/doc/tools/ImageIntegration/ImageIntegration.html

Hope they help,

Mike

I think I read Vincent's tutorial a while back, and then have been experimenting with a few tweaks. It looks like I've pretty much ended up circling back to where I started...

Duncan

Offline Duncan

  • Member
  • *
  • Posts: 53
Re: Requesting comments or suggestions on my calibration workflow
« Reply #8 on: 2017 January 26 14:55:52 »
Why not just use the BatchPrepocessing script until you are more familiar with things.  It is very simple to use.

Thanks for the response.

I could do that, but I'm digging more into the details of the process. It's not that I'm looking for simplification, rather I'm trying to make sure that I'm getting the best possible result by improving my understanding of the algorithm details.

Duncan