Author Topic: Calibration and align batch operations  (Read 135090 times)

Offline mmirot

  • PixInsight Padawan
  • ****
  • Posts: 881
    • View Profile
Re: Calibration and align batch operations
« Reply #15 on: 2012 January 04 21:13:41 »
 Version 7a seems to work fine
 Max

Offline RobF2

  • PixInsight Addict
  • ***
  • Posts: 189
  • Rob
    • View Profile
    • Rob's Astropics
Re: Calibration and align batch operations
« Reply #16 on: 2012 January 05 04:42:54 »
Guys, would anyone care to comment on pros/cons off this latest cal script versus the older PreProcessingPipeline script?

Thanks,
Rob
FSQ106/8" Newt on NEQ6/HEQ5Pro via EQMOD | QHY9 | Guiding:  ZS80II/QHY5IIL | Canon 450D | DBK21 and other "stuff"
Rob's Astropics

Offline mmirot

  • PixInsight Padawan
  • ****
  • Posts: 881
    • View Profile
Re: Calibration and align batch operations
« Reply #17 on: 2012 January 05 06:37:22 »
The script is not yet as sophisticated as the pipeline.  It does not integrate the aligned lights currently.  Hopefully, this will be added shortly.

However, it is much easier to set up and use. I find the UI much more intuitive.

I is geared a more to those that use a monochrome camera right now and have the FITs key words Bin, exposure, and filter.
MaxIm does this for all monchrome cameras automatically as do other acquistion programs.

The nice part of the script is it will automatically sort out darks, flats, bias by these criteria make masters and calibrate matching lights. That is if you give a bias set with bin2, darks with bin2 &10min and flats with bin 2 with a red filter;
you can expect the script to create masters for bias bin2, Master dark bin2 & 10min and a bin 2 red flat.

If Red ,bin2, 10min image are in the light list they are then automatically matched to the appropiate masters, calibrated and then aligned. It outputs a folder with each type of image produced masters, calibrated and registered.

You can list as many combination of images as you like with various bins, exposure times and filters in it and it will figure out the matching calibration.

The masters can be reused for future calibrations with script to save time.

Max




Offline kwiechen

  • PixInsight Addict
  • ***
  • Posts: 186
    • View Profile
Re: Calibration and align batch operations
« Reply #18 on: 2012 January 05 10:51:33 »
Jack, Max,

thank you for this contribution and feedback!

@Jack: can I integrate this nice doc into the help label of the next script version?

@Max: I have sorted the wish list a little bit:


1.  Persistent setting and paths. They should remain the same next time script is run
< I will try to add this to the next version.


2. Single image or an image named master in fits key word automatically identified as a master. No need to check use master
<There should be no problem to add a new custom keyword MASTERFRAME or something like this with a corresponding value and reuse these files automatically.

3 Images that don't contain the key words. Most of us use MaxIm and other programs that contain the standard Fits key words for time, bin, and filter.  However, there will be a few with the odd acquistion program that does not put these in.
<For older data not containing the fits header keywords, I would prefer to extend the small BatchConvertImagetype script to set all keywords necessary to be handled by the CalibrateAlign
script.

4 Handle color camera images.
<I have never used PI to process OSC/DSLR data. Can you please give me some details regarding the difference when calibrating/integrating OSC images compared to monochrome data? Is it necessary to debayer/split these images into different channels before? If we have
DSLR images how can I get for example the exposure time? To handle images from color cameras I can provide a mode ignoring image metadata (binning is meaningless from my point of view,
filter information can be taken from the debayer pattern)

5 Dark flat option or scaling darks for longer duration flats.
<Dark frame scaling is already set by default in this script. The dark flats can be added to the dark tab and will be used.

6  Add integration. Unselect align or integration processes when desired too
<I can try to add this - the integration control is separated in the script code and it may be possible to add this dynamically for all filter settings to the Light tab.
This was removed from the initial version because I thought about using different sets of integration parameters for well sampled L data and more or less poorly sampled RGB data.

<But I am not convinced that this is a good solution. This excellent document (Jordi Gallego, Image integration techniques: Increasing SNR and outlier rejection with PixInsight, slides for a
presentation given at the VI Seminario de Astrofotografía de Cielo Profundo, Madrid, November 20th, 2010.) shows the importance of fine-tuning integration parameters

7: Add cosmetic correction module to script. It can be helpful to remove fixed hot columns and pixels after calibration.
I need less sigma rejection when I do this for my hot columns.
<The CosmeticCorrection module is not installed in the last version of PI (Windows 7, x64), so I can not test this.


8. Deselect individual fits files from list in addition to clear all
9. Select the reference image from the image list , double click or default to first file ( easier than set path)
<Points 8 and 9 will require some rewrite of the internal structure. The TreeBox is only updated to provide visual feedback to the user in the current state of the script.

10. I use a rotator so matching the fits key rotator ' xxx.xx  ' position in degrees between flats and lights would be vary helpful. ( in addition to filter)
<Is there a 'standard' to handle this? And here I will need some test data.


Best,


Kai



Offline Jack Harvey

  • PTeam Member
  • PixInsight Padawan
  • ****
  • Posts: 975
    • View Profile
    • PegasusAstronomy.com & Starshadows.com
Re: Calibration and align batch operations
« Reply #19 on: 2012 January 05 11:12:24 »
Yes feel free to add it to help
Jack Harvey, PTeam Member
Team Leader, SSRO/PROMPT Imaging Team, CTIO

Offline mmirot

  • PixInsight Padawan
  • ****
  • Posts: 881
    • View Profile
Re: Calibration and align batch operations
« Reply #20 on: 2012 January 05 15:12:06 »
Jack, Max,

thank you for this contribution and feedback!

@Jack: can I integrate this nice doc into the help label of the next script version?

@Max: I have sorted the wish list a little bit:


1.  Persistent setting and paths. They should remain the same next time script is run
< I will try to add this to the next version.

Excellent
2. Single image or an image named master in fits key word automatically identified as a master. No need to check use master
<There should be no problem to add a new custom keyword MASTERFRAME or something like this with a corresponding value and reuse these files automatically.
Great


3 Images that don't contain the key words. Most of us use MaxIm and other programs that contain the standard Fits key words for time, bin, and filter.  However, there will be a few with the odd acquistion program that does not put these in.
<For older data not containing the fits header keywords, I would prefer to extend the small BatchConvertImagetype script to set all keywords necessary to be handled by the CalibrateAlign
script.

Good Idea a batch script crossed my mind too
4 Handle color camera images.
<I have never used PI to process OSC/DSLR data. Can you please give me some details regarding the difference when calibrating/integrating OSC images compared to monochrome data? Is it necessary to debayer/split these images into different channels before? If we have
DSLR images how can I get for example the exposure time? To handle images from color cameras I can provide a mode ignoring image metadata (binning is meaningless from my point of view,
filter information can be taken from the debayer pattern)

I am clueless here. I used a color CCD only once in my life. We need help from the peanut gallery on this one
5 Dark flat option or scaling darks for longer duration flats.
<Dark frame scaling is already set by default in this script. The dark flats can be added to the dark tab and will be used.

Good.  I assumed it was just using a bias subtract on flats as a default.
You might want see about a use "bias" option. There can be calibration problems if the dark noise in the flat is too low.  See Juans coments on this from today. http://pixinsight.com/forum/index.php?topic=3677.15
6  Add integration. Unselect align or integration processes when desired too
<I can try to add this - the integration control is separated in the script code and it may be possible to add this dynamically for all filter settings to the Light tab.
This was removed from the initial version because I thought about using different sets of integration parameters for well sampled L data and more or less poorly sampled RGB data.

<But I am not convinced that this is a good solution. This excellent document (Jordi Gallego, Image integration techniques: Increasing SNR and outlier rejection with PixInsight, slides for a
presentation given at the VI Seminario de Astrofotografía de Cielo Profundo, Madrid, November 20th, 2010.) shows the importance of fine-tuning integration parameters

You are right to be concerned. I agree the best way to do this is to link the parameters to each filter.  Once the all setting are persistant then you might be able to use a ROI just like the integration module. I think the files may still be in the cache after the script is run.  You could just uncheck calibrate, and align funtions then run the integration part of the script again with tweeking the ROI and settings the same as the module.   ::)

7: Add cosmetic correction module to script. It can be helpful to remove fixed hot columns and pixels after calibration.
I need less sigma rejection when I do this for my hot columns.
<The CosmeticCorrection module is not installed in the last version of PI (Windows 7, x64), so I can not test this.

This is way down on my wish list    ;)

8. Deselect individual fits files from list in addition to clear all
9. Select the reference image from the image list , double click or default to first file ( easier than set path)
<Points 8 and 9 will require some rewrite of the internal structure. The TreeBox is only updated to provide visual feedback to the user in the current state of the script.



10. I use a rotator so matching the fits key rotator ' xxx.xx  ' position in degrees between flats and lights would be vary helpful. ( in addition to filter)
<Is there a 'standard' to handle this? And here I will need some test data.
I will send a couple FITs overs soon

Max

Offline mmirot

  • PixInsight Padawan
  • ****
  • Posts: 881
    • View Profile
Re: Calibration and align batch operations
« Reply #21 on: 2012 January 07 10:08:47 »
Any programmers out there?    May be someone can provide Kai support on these issues ???

Path and setting persistence.

Calibration of color cameras

The TreeBox

Max

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7040
    • View Profile
    • http://pixinsight.com/
Re: Calibration and align batch operations
« Reply #22 on: 2012 January 07 10:20:15 »
Quote
Any programmers out there?

If I can be considered a programmer, I'm starting to take a look at Kai's script. First let me get acquainted with the script and I'll try to improve it and provide support on those topics.
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline mmirot

  • PixInsight Padawan
  • ****
  • Posts: 881
    • View Profile
Re: Calibration and align batch operations
« Reply #23 on: 2012 January 07 10:45:24 »
Thanks for the attention Juan!
--------------------------------------------------------------------------------------------------------------------------------------------------
BTW, Kai I for correction of hot columns we should be using the defect map which is a module
Attached is  a Fits header with MaxIM's Rotator entry. In the this it is at 313.00 degrees.

Max

Offline mmirot

  • PixInsight Padawan
  • ****
  • Posts: 881
    • View Profile
Re: Calibration and align batch operations
« Reply #24 on: 2012 January 09 08:05:40 »
Thanks for the attention Juan!
--------------------------------------------------------------------------------------------------------------------------------------------------
BTW, Kai I for correction of hot columns we should be using the defect map which is a module
Attached is  a Fits header with MaxIM's Rotator entry. In the this it is at 313.00 degrees.

Max

Thanks Juan

Offline kwiechen

  • PixInsight Addict
  • ***
  • Posts: 186
    • View Profile
Re: Calibration and align batch operations
« Reply #25 on: 2012 January 11 12:05:29 »
007b adds some kind of persistence. The code was splitted for better maintenance.

Offline mmirot

  • PixInsight Padawan
  • ****
  • Posts: 881
    • View Profile
Re: Calibration and align batch operations
« Reply #26 on: 2012 January 12 21:50:19 »
Seems to work the same. I can't tell what is persistant.

Note: I you check master ; the file is removed if it was selected first.


Max

Offline kwiechen

  • PixInsight Addict
  • ***
  • Posts: 186
    • View Profile
Re: Calibration and align batch operations
« Reply #27 on: 2012 January 12 23:19:59 »
The persistence settings save the state and values of controls - but not the file lists. If you check 'use master' the file list
in the current tab will be removed (tooltip), because the current state of the script is not able to recognize master frames from their fits headers.

Kai


Offline mmirot

  • PixInsight Padawan
  • ****
  • Posts: 881
    • View Profile
Re: Calibration and align batch operations
« Reply #28 on: 2012 January 18 11:53:42 »
I am looking forward to the next version of the script. I have new image at last.

Max

Offline mmirot

  • PixInsight Padawan
  • ****
  • Posts: 881
    • View Profile
Re: Calibration and align batch operations
« Reply #29 on: 2012 January 19 06:35:04 »
Small note.
I was trying to calbrate and align a few images from the west side of the mount. Most were obtained on the east side. So I set up to align and calibrate the westerly light frames, six green images and two reds to the reference frame.  The script stopped before calibrating the two red frames. The process window said it could not continue because there were less than three images.
I realise three will be required to integrate a master etc, but calibrate and alignment sure go ok.

Sure would help to recogize the rotator angle in the fits for calibration. I would not have to run the scipt twice.

Max