Author Topic: Image Calibration Workflow  (Read 31029 times)

Offline jeffweiss9

  • PixInsight Old Hand
  • ****
  • Posts: 339
Image Calibration Workflow
« on: 2010 November 20 20:35:38 »
I'm enjoying PI tremendously but one area I've not even attempted is image reduction, by which I mean image calibration, registration and stacking. Therefore, I have continued to do my image reduction in AIP4WIN which is familiar to me and, I believe, does an adequate job, but I'd like to switch entirely to PI.   Although I've used the StarAlignment and ImageIntegration tools successfully on occasion , I've been unable to gather a coherent picture of how to do image calibration.  Can someone just give an overview of their workflow for calibration of their subs with PI?  I think that would give me a place to start.
Thanks very much.
-Jeff
APM LZOS 130/780 f/6 LW CNC II APO, Riccardi 1.0 FF or 0.75 FF/FR, Tak EM-200 Temma2, FLI Microline ML-16200, Astrodon E Gen 2 filters and 5nm Ha, Orion 50mm Guider & Starlight Xpress Lodestar X2.

Offline dsnay

  • PixInsight Addict
  • ***
  • Posts: 100
Re: Image Calibration Workflow
« Reply #1 on: 2010 November 22 06:27:22 »
Jeff,

I don't have the answer to this for you, but it is the one area I feel would be of greatest benefit to new users like myself. I'm very interested in the answer to this question as well.

Dave

Offline vicent_peris

  • PTeam Member
  • PixInsight Padawan
  • ****
  • Posts: 988
    • http://www.astrofoto.es/
Re: Image Calibration Workflow
« Reply #2 on: 2010 November 22 10:14:05 »
I'm enjoying PI tremendously but one area I've not even attempted is image reduction, by which I mean image calibration, registration and stacking. Therefore, I have continued to do my image reduction in AIP4WIN which is familiar to me and, I believe, does an adequate job, but I'd like to switch entirely to PI.   Although I've used the StarAlignment and ImageIntegration tools successfully on occasion , I've been unable to gather a coherent picture of how to do image calibration.  Can someone just give an overview of their workflow for calibration of their subs with PI?  I think that would give me a place to start.
Thanks very much.
-Jeff

Hi Jeff,

let supose you have bias, dark, flat and light frames. In PixInsight, calibration is done manually for the moment, so first you must generate master calibration frames, and then you will calibrate you light frames.

Step 1: Generate master bias. Simply integrate them in ImageIntegration.
Step 2: Calibrate dark and flat frames by subtracting the master bias in ImageCalibration.
Step 3: Generate master dark and master flat frames by integrating each of them in ImageIntegration.
Step 4: Calibrate your light frames in ImageCalibration with your master bias, dark and flat frames. Take into account that the "Calibrate" checkboxes in ImageCalibration must be disabled, because you already subtracted the master bias.

If your camera has temperature control, and you do flats under a very controlled environment (no sky flats, but dome flats or something similar), you can simplify things:

Step 1: Generate master bias. Simply integrate them in ImageIntegration.
Step 2: Generate master dark. Simply integrate them in ImageIntegration.
Step 3: Generate master flat. Simply integrate them in ImageIntegration.
Step 4: Calibrate your light frames in ImageCalibration with your master bias, dark and flat frames. This time you *must* check the "Calibrate" checkboxes in ImageCalibration to subtract the bias from the master dark and flat frames.

Please read also my article on calibration:

http://pixinsight.com/tutorials/master-frames/en.html


Best regards,
Vicent.

Offline jeffweiss9

  • PixInsight Old Hand
  • ****
  • Posts: 339
Re: Image Calibration Workflow
« Reply #3 on: 2010 November 23 07:43:15 »
Thanks very much, Vicent, for laying that out.
 I use a light box that gives quite regular flats and I have a temperature-controlled camera so I'd hope to use the simpler method.  However, occasionally I am calibrating data obtained at a different temperature from my darks (say lights at -15 deg while I have darks at -10,-20 deg).  Would the second method still work? 
Thanks again.
-Jeff
APM LZOS 130/780 f/6 LW CNC II APO, Riccardi 1.0 FF or 0.75 FF/FR, Tak EM-200 Temma2, FLI Microline ML-16200, Astrodon E Gen 2 filters and 5nm Ha, Orion 50mm Guider & Starlight Xpress Lodestar X2.

Offline vicent_peris

  • PTeam Member
  • PixInsight Padawan
  • ****
  • Posts: 988
    • http://www.astrofoto.es/
Re: Image Calibration Workflow
« Reply #4 on: 2010 November 23 10:28:39 »
Thanks very much, Vicent, for laying that out.
 I use a light box that gives quite regular flats and I have a temperature-controlled camera so I'd hope to use the simpler method.  However, occasionally I am calibrating data obtained at a different temperature from my darks (say lights at -15 deg while I have darks at -10,-20 deg).  Would the second method still work? 
Thanks again.
-Jeff

Hmmm... it depends. Some cameras have varying bias pedestals with temperature. Check this.


Regards,
Vicent.

Offline Niall Saunders

  • PTeam Member
  • PixInsight Jedi Knight
  • *****
  • Posts: 1456
  • We have cookies? Where ?
Re: Image Calibration Workflow
« Reply #5 on: 2010 November 29 16:07:19 »
Quote
However, occasionally I am calibrating data obtained at a different temperature from my darks (say lights at -15 deg while I have darks at -10,-20 deg)

Jeff,

Surely. with a TEC camera, it is not that difficult to just acquire the appropriate Darks at the corresponding temperature and exposure - even AFTER you acquired the Lights?

Doing this would just significantly reduce any 'supposition' and 'inerpolation' needed because you had imaged at a temperature/exposure combination that was 'different' from what you already had calibration files for.

And, if you are going to post acquire Darks, then why not also acquire a set of new Bias frames at the same time - and avoid any issues due to 'old' (i.e. non-matching) bias frames.

Finally, if your imager has Gain/Offset controls, remember that ALL of your calibration frames really need to be based on the SAME settings for Gain and Offset.
Cheers,
Niall Saunders
Clinterty Observatories
Aberdeen, UK

Altair Astro GSO 10" f/8 Ritchey Chrétien CF OTA on EQ8 mount with homebrew 3D Balance and Pier
Moonfish ED80 APO & Celestron Omni XLT 120
QHY10 CCD & QHY5L-II Colour
9mm TS-OAG and Meade DSI-IIC

Offline jeffweiss9

  • PixInsight Old Hand
  • ****
  • Posts: 339
Re: Image Calibration Workflow
« Reply #6 on: 2010 December 02 21:44:00 »

 Good point. I should be building up my dark library now that I won't have trouble in December cooling down at home to the temperatures I usually image at up in the hills around here.  But now I'm confused about the acquisition of the bias frames.  I've been taking them at the same temperature as the lights and flat frames (and flat darks).  I think I'm hearing/reading that bias frames should actually be taken at the temperature of the dark frames, if there is a difference.  Also, how much of a temperature difference can be tolerated in the workflow Vicent described above?  That is, what is the limit on differing temperatures between dark and light frames as described below in the IC write-up?

IC is optimized to work with calibration libraries: don't worry about differing temperatures and exposure times between dark and light frames. IC will always rescale the dark noise to match every light frame.

 Thanks.
-Jeff
APM LZOS 130/780 f/6 LW CNC II APO, Riccardi 1.0 FF or 0.75 FF/FR, Tak EM-200 Temma2, FLI Microline ML-16200, Astrodon E Gen 2 filters and 5nm Ha, Orion 50mm Guider & Starlight Xpress Lodestar X2.

Offline Niall Saunders

  • PTeam Member
  • PixInsight Jedi Knight
  • *****
  • Posts: 1456
  • We have cookies? Where ?
Re: Image Calibration Workflow
« Reply #7 on: 2010 December 09 07:18:19 »
Hi Jeff,

I believe that Biases, for an 'ideal match', should be temperature matched to Darks and Lights. That will at least 'statistically eliminate' the temperature variable - because, after all, you have kept it 'constant'.

When it comes to Flats (a different kind of 'Lights', in reality), then I would apply the same logic. Flats, Darks and Biases all acquired at the same temperature AS EACH OTHER.

Of course - if you have a TEC camera, then it is sensible to figure out what is a SUSTAINABLE temperature for ALL of your data acquisition - and don't push this temperature setting down too low. For example, on my new QHY10 CCD, I have run trials at -15C, and using exposure times up to 30mins. It seems that -15C is an easily achievable CONSTANT temperature, even working inside the house at an ambient temperature of +20C - monitoring the TEC system, I was able to observe that the PWM drive signal never exceeded 60% of maximum, so I would have been able to handle even higher ambient temperatures, if needed (although HIGHLY unlikely here in the UK!!).

But, I would also have to consider a very 'cold' observing session - where the camera might be running in an AMBIENT environment of -15C, requiring the TEC system to run (effectively) at 0% - in other words with no real control whatsoever. Not a good mode of operation - but easily avoided because 'I' won't be out if it is THAT cold ;)

So, in my case, I feel ready to work at -15C, and will acquire Darks and Biases at -15C. That means I will also need to acquire Lights and Flats at -15C as well. So far, no problems.

Now, my typical exposure times for Lights will (hopefully) be between 1 and 5 minutes (this may get longer if my new OAG system eliminates my current 'flexure' problems), and I tend to just expsose in 'multiples' of 60 seconds - because I don't know of any reason to do otherwise. So, what I can do (because my TEC runs so well) is simply acquire a Dark frames library of 30-odd Darks each, for each exposure time I am likely to work with. And I have already done that for 60 through to 300s, 30 frames each at -15C, with another 30 Biases acquired in between each Dark frame expsoure session. Total of 300 frames. Took about ten hours. I slept most of that, and then went to work. Easy job!!!

So, now I have 5 MasterDarks, and a 'superMasterBias' created from 150 acquired Biases - all of which appeared (statistically) 'identical', throughout the ten-hour acquisition period. Next task is to 'prove to myself' that - if necessary - temperature-equivalent Darks can be 'scaled' for exposure, just in case I decide to acquire Lights at some intermediate exposure time.

For example, I will be looking to see if a 'scaled 2s MasterDark' is statistically equivalent to an 'actual 2s MasterDark'. If this IS the case, then I don't need to go out of my way to also acquire FlatDarks, at 2 second exposure times, to 'match' my 2s Flats (assuming that 2s exposures are what I will need to achieve 50% ADU levels with the new camera and my existing LightBox).

I know that there has been a lot of discussion regarding ImageCalibration, that suggests that 'all' Darks can be fed into the calibration process - even if they were NOT 'temperature correlated' to the Lights (or Flats) that they are being used to calibrate. Whilst this may well be the case (becasue of the inherent 'power' of the IC process in PI), I see no real reason 'for' adopting this approach, when - assuming a well-controlled TEC Camera - there is no reason NOT to be using temperature-correlated exposures in the first place.

And your last question, regarding the possible range of temperature difference that could be tolerated by the IC process - I hope to tackle this, by experiment, over the next few days - by simply recording Darks and Biases at a 'fixed' exposure time (probably 120s, for this experiment) and looking at how different the resultant Bias-Calibrated MasterDarks actually are. By varying the TEC temperature plus or minus a degree at a time, I will be able to see what kind of effect this actually has on the data.

Of course, if the sky clears up, and I can actually manage to get through the snow to the observatory, and can get the roof to open, and find that there isn't an eight-foot snow drift inside, I may actually just end up imaging the heavens - in which case all these experiments will have to wait until the next cloudy night!!
Cheers,
Niall Saunders
Clinterty Observatories
Aberdeen, UK

Altair Astro GSO 10" f/8 Ritchey Chrétien CF OTA on EQ8 mount with homebrew 3D Balance and Pier
Moonfish ED80 APO & Celestron Omni XLT 120
QHY10 CCD & QHY5L-II Colour
9mm TS-OAG and Meade DSI-IIC

Offline jeffweiss9

  • PixInsight Old Hand
  • ****
  • Posts: 339
Re: Image Calibration Workflow
« Reply #8 on: 2010 December 11 17:26:23 »
Thanks, Niall-
 Good luck on the experiments and the clear skies. I just heard our local forecast and it was off-and-on clouds forever.
-Jeff
APM LZOS 130/780 f/6 LW CNC II APO, Riccardi 1.0 FF or 0.75 FF/FR, Tak EM-200 Temma2, FLI Microline ML-16200, Astrodon E Gen 2 filters and 5nm Ha, Orion 50mm Guider & Starlight Xpress Lodestar X2.

Offline varmint

  • Newcomer
  • Posts: 35
    • Varmint's Astrophoto's
Re: Image Calibration Workflow
« Reply #9 on: 2011 February 14 13:25:13 »

Hi Jeff,

let supose you have bias, dark, flat and light frames. In PixInsight, calibration is done manually for the moment, so first you must generate master calibration frames, and then you will calibrate you light frames.

Step 1: Generate master bias. Simply integrate them in ImageIntegration.
Step 2: Calibrate dark and flat frames by subtracting the master bias in ImageCalibration.
Step 3: Generate master dark and master flat frames by integrating each of them in ImageIntegration.
Step 4: Calibrate your light frames in ImageCalibration with your master bias, dark and flat frames. Take into account that the "Calibrate" checkboxes in ImageCalibration must be disabled, because you already subtracted the master bias.

If your camera has temperature control, and you do flats under a very controlled environment (no sky flats, but dome flats or something similar), you can simplify things:

Step 1: Generate master bias. Simply integrate them in ImageIntegration.
Step 2: Generate master dark. Simply integrate them in ImageIntegration.
Step 3: Generate master flat. Simply integrate them in ImageIntegration.
Step 4: Calibrate your light frames in ImageCalibration with your master bias, dark and flat frames. This time you *must* check the "Calibrate" checkboxes in ImageCalibration to subtract the bias from the master dark and flat frames.

Please read also my article on calibration:

http://pixinsight.com/tutorials/master-frames/en.html


Best regards,
Vicent.


I have a question about the Calibrate toggle.  I've found that if I did NOT toggle the Calibrate button on the Master Dark frame section that my light images appear to have their black levels clipped.  What does the clipping?  Does the algorithm not handle negative numbers and truncates them to zero?

And the only function of the Calibrate button is to subtract the Master Bias from either the Master Dark or Master Flat (depending on which Calibrate is enabled)?  What does the Calibrate toggle do on the Master Bias?

If I follow Vicent's workflow exactly I tend to not have issues, but I guess I forgot to Check the Calibrate this past weekend and ran into trouble.

Here is the example of my "clipped" (I processed it through integration) frames of H-alpha for the Flame/Horsehead Region:
http://www.flickr.com/photos/varmints_pics/5442275567/
Here is what I got when I didn't clip:
http://www.flickr.com/photos/varmints_pics/5443210517/


Clear Skies,

Jim
--"Do or do not.  There is no Try" --Jedi Master Yoda

Offline Sedat

  • Member
  • *
  • Posts: 63
  • Newbie
Re: Image Calibration Workflow
« Reply #10 on: 2012 February 02 03:46:12 »
Vicent et al,

I am a new convert to Pixinsight :) and I am very glad that I did the move.

I believe the stacking in PI is also superior to other software I have been using.

Now I want to do all the calibration in PI too. I have one challenge though.... most of my flat frames are shot at ISO100 whereas my light frames are shot at ISO800 or 1600. I am using two DSLR cameras for shooting, a Canon 450D and a 550D, both astro modified.

Calibrating the ISO100 flats with ISO1600 bias or dark frames is likely to cause problems. Can you please tell me how to proceed ?

Many thanks in advance.

Sedat
Sedat

Offline pfile

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 4729
Re: Image Calibration Workflow
« Reply #11 on: 2012 February 02 08:10:48 »
hi sedat,

you can just precalibrate the master flat with it's own ISO100 bias frames (or darks). then just don't check 'calibrate' in the flats box when you calibrate the lights.

Offline Sedat

  • Member
  • *
  • Posts: 63
  • Newbie
Re: Image Calibration Workflow
« Reply #12 on: 2012 February 02 08:31:51 »
Perfect! thank you so much.
Sedat

Offline vicent_peris

  • PTeam Member
  • PixInsight Padawan
  • ****
  • Posts: 988
    • http://www.astrofoto.es/
Re: Image Calibration Workflow
« Reply #13 on: 2012 February 08 00:02:42 »
Hi,

I can confirm that this workflow should work. The ISO of the flat frames is not relevant because a 50% vignetting will be the same at 100 ISO as well as at 1600 ISO. Just take care of calibrating your flats with the proper dark and bias masters.


Regards,
Vicent.

Offline Sedat

  • Member
  • *
  • Posts: 63
  • Newbie
Re: Image Calibration Workflow
« Reply #14 on: 2012 February 08 08:00:46 »
Thank you for confirming this Vicent. Hope to produce some nice photos with the help of PI soon.

Best wishes

Sedat
Sedat