Author Topic: Calibration and align batch operations  (Read 135693 times)

Offline mmirot

  • PixInsight Padawan
  • ****
  • Posts: 881
    • View Profile
Re: Calibration and align batch operations
« Reply #330 on: 2012 April 27 14:22:33 »
Automatic typing also works by looking at name patterns (instead of FITS keyword)

You mean if the file name is      IC443_light_Red_600sec_-35_Westpierside_serial001.fit

That will get sorted somehow?

Explain how that works. I missed that.


Max

Offline jeffweiss9

  • PixInsight Old Hand
  • ****
  • Posts: 339
    • View Profile
Re: Calibration and align batch operations
« Reply #331 on: 2012 April 27 19:40:38 »
Thanks, Harry-
Trial and Error.  Where have I heard that before?
Clear skies,
-Jeff
APM LZOS 130/780 f/6 LW CNC II APO, Riccardi 1.0 FF or 0.75 FF/FR, Tak EM-200 Temma2, FLI Microline ML-16200, Astrodon E Gen 2 filters and 5nm Ha, Orion 50mm Guider & Starlight Xpress Lodestar X2.

Offline Harry page

  • PTeam Member
  • PixInsight Jedi Knight
  • *****
  • Posts: 1457
    • View Profile
    • http://www.harrysastroshed.com
Re: Calibration and align batch operations
« Reply #332 on: 2012 April 27 23:57:37 »
Thanks, Harry-
Trial and Error.  Where have I heard that before?
Clear skies,
-Jeff

EEErrrr    don't know as I will " Play with the settings " in future  >:D

But If there was a magic setting for every tool there would be no need for the user , be boring then  8)
Harry Page

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7089
    • View Profile
    • http://pixinsight.com/
Re: Calibration and align batch operations
« Reply #333 on: 2012 April 28 02:50:29 »
Hi Max,

Quote
Here Juan.......................

Wow, thank you so much for this nice description. You have saved me the work to write something like this to send a massive email to all PI users. I'll send a formal announcement of BatchPreprocessing today using your description.
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7089
    • View Profile
    • http://pixinsight.com/
Re: Calibration and align batch operations
« Reply #334 on: 2012 April 28 03:36:45 »
Quote
Thank you Juan for your hard work. What can we look forward to? Could you tell us a little bit?

Version 1.8 will be more focused on developers than on users. These are some of my goals for PI 1.8 as a development platform:

- Interprocess communication will work for processes (it now works only for file formats). This means that any module will be able to manage instances of any installed process, irrespective of which module implements it. A module will be able to create a process instance, populate it with parameters, execute it, and communicate with it. This will extend the development capabilities of PixInsight considerably, in ways that we probably cannot anticipate.

- Interprocess communication will also work for process interfaces. A module will be able to communicate with any existing tool window on the platform, even with windows defined by other modules.

- Core networking functionality exposed to modules. Thanks to this, a module can perform download/upload network operations without requiring third-party libraries, just as the JavaScript runtime can do now.

- (Still under evaluation) The entire PJSR exposed to modules. This means that a module can define and execute a JavaScript script. This functionality has some important security implications that I am still evaluating, so it perhaps will not be available in version 1.8.0. It will be available during the 1.8 cycle.

- Extended PJSR functionality with more core objects such as Workspace, Icon, ProcessIcon, ImageIcon, ProcessInterface, etc. This will allow scripts to fully manage the entire PixInsight workspace.

- PCL 2.0. This is actually unrelated to PI version 1.8, but I also want to have it ready before Summer. This new version will introduce lots of changes and improvements. (e.g. pixel iterators, a much more powerful ImageVariant class, improved container classes, optional C++11 support, etc.) This deserves a separate thorough description.

From the user's perspective, version 1.8 will be more focused on robustifying and stabilizing the platform than on new features. There will be some niceties though:

- Improved Process Explorer with more flexible and customizable favorite process collections.

- Improved web browser with file download capabilities.

- New image navigation floating window. It will allow you to see a reduced version of the current image with navigation capabilities, similar to the Blink tool.

- Better and more powerful options to organize and manage icons.

- New workspace "present windows" and "hide windows" features, similar to some Mac OS X, KDE and Windows 7 features. Present windows will temporarily show all image windows organized as a grid. Hide windows will temporarily hide all image windows in the current workspace to allow you access and organize your hidden icons.

- New and improved console commands.

- An installer application for Linux and FreeBSD. PixInsight 1.8 will integrate with most X11 desktops (automatic generation of menu items, workspace launcher icons, etc.). The new PI installer application with run in superuser mode to install PI on the standard /opt directory by default (user configurable). The PI updater application will use standard setuid bits on UNIX to run in privileged mode (similar to Windows UAC technology, but much simpler and more secure).

- Lots of performance improvements and bug fixes.

- More :)

Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline mmirot

  • PixInsight Padawan
  • ****
  • Posts: 881
    • View Profile
Re: Calibration and align batch operations
« Reply #335 on: 2012 April 28 04:25:21 »
Hi Max,

Quote
Here Juan.......................

Wow, thank you so much for this nice description. You have saved me the work to write something like this to send a massive email to all PI users. I'll send a formal announcement of BatchPreprocessing today using your description.

Your welcome Juan,


It was best contribution that I could offer besides beta testing. Glad you like the note . I think PI users will love the script. I do.

Max


Offline mmirot

  • PixInsight Padawan
  • ****
  • Posts: 881
    • View Profile
Re: Calibration and align batch operations
« Reply #336 on: 2012 April 28 20:36:26 »

My second try- lots of spellllllling mistakes fixed and some editing :surprised:
I did the last one from memory.
I was away from my usual CPU and did not have PI to look at the script.

Max
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Batch Preprocessing Script V1.0
Designed by Kia Wiechen and Juan Conejero.

Release notes.

The script allows for automatic sequential calibration, alignment and initial integration of image sets.
It is able to handle multiple simultaneous image sets with various combinations of binning and filters. 
It does so by accessing the already robust modules contained in the PixInsight Platform.
It eliminates much of the confusion with calibrating step by step.  Master frames are created and can be reused. 
Most of the important alignment and integration settings are available through the script. 

Other features include:
 * Cosmetic correction application module to remove defects such as hot/cold pixels and columns
*  Calibration of color one shot color CCD images
*  Direct calibration of DSLR raw files.
*  Neatly sorts the output into a directory and subfolders * 
*  automatic recognition of image types when used with images obtained with MaxIm DL
 
Images types and parameters such as light, bias, dark, flat, filter, and bin can be recognized automatically
through their FITS header when acquired through MaxIM DL. 

This done the by selecting the first add files button. Images will be automatically identified and sorted as light, dark, bias, flat, bin and filter.

Images sets can also be easily be entered into the script with individual "add" buttons for bias, dark, light and flat.

The "add custom" can be used to assign key words used in calibration for matching flats to lights.

Perhaps no key words for each filters has been created in the FITS header by your acquisition software.
No problem enter add  "Red" to your list of lights and than the flats and the Red images will be matched for calibration.
 Repeat the action for each filter as needed. 

A similar action could be used when obtaining images using a camera rotator.   
Typically an object will have 180 degree rotation to reacquire the guide star after a crossing the meridian on a GEM mount.
In this case just add a key word such as "West" to the flats and light acquired on the west pier side.

An instance of script can be saved by dragging the arrow and creating an icon. This is very useful since it saves all the paths and parameter for reuse.

The cosmetic correction script will be performed after image calibration when checked.

An icon has to be generated first from the cosmetic correction module.
This is done by setting up the module and dragging the arrow to save an instance of the CC module.
The icon can then be selected by the script. 

Don't forget to save the icon for future use. 
It can be used again next time you run the script using the same camera.

There is check box for limiting the script to calibration only. The alignment and integration are skipped.

Integration Settings
Settings parallel those on the integration module
Different settings can be applied for each category of master and for the light frames

Options Settings



 CFA. Stands for Color Filter Array.  Check color cameras
 Optimize dark- this the dark optimization option available in the calibration module
 Rejection maps - check this when you desire the map obtain from integrating the masters frames or the integration of the lights
 Export calibration files- files names exported as generated instances
 Up Bottom FITS order - Should be checked for most popular acquisition software packages.
 Use master bias, Use master dark, Use master flat, these should be checked when the image loaded is a master that was produced previously.

Registration Reference Image - select the path of a reference image to register all images in the data set.
 
Output Directory- Select or create an directory to write processed images from this script.

Diagnostics - Check for problems before running the script. It will spot things you forgot.

Run- Runs script

Exit- Closes script


Max

Offline RobF2

  • PixInsight Addict
  • ***
  • Posts: 189
  • Rob
    • View Profile
    • Rob's Astropics
Re: Calibration and align batch operations
« Reply #337 on: 2012 April 29 03:09:27 »
Help - I've been trying to get to know the script with a simple scenario of 5 binned bias, 5 binned darks, a few flats for R,G and B and a few lights for R,G and B.
When the script gets to the "Begin calibration of flat frames" section, it loads the master bias, then the master dark

Then it says "Loading master calibration frames:"
Then "Loading calibration frame image:"  (followed by name of the master bias frame)

Then gives error:
(name of master bias): Multiple images cannont be used as calibration frames



I get this whether I feed in multiple bias frames and let if make a master, OR just select the master the script made previously and tick the "Use Master Bias" and "Use Master Dark" boxes.

Can anyone help shed some light on what I'm doing wring?
FSQ106/8" Newt on NEQ6/HEQ5Pro via EQMOD | QHY9 | Guiding:  ZS80II/QHY5IIL | Canon 450D | DBK21 and other "stuff"
Rob's Astropics

Offline mmirot

  • PixInsight Padawan
  • ****
  • Posts: 881
    • View Profile
Re: Calibration and align batch operations
« Reply #338 on: 2012 April 29 03:56:21 »
  You are generating masters so don't check use master
Check diagnostic before going to run

Max

astropixel

  • Guest
Re: Calibration and align batch operations
« Reply #339 on: 2012 April 29 04:46:19 »
Brilliant Max! This makes the DSLR_RAW work flow redundant... great stuff!

I have just processed a set of RAW files in very quick time, far quicker than following the DSLR_RAW workflow. The only thing is, I have a very nice monochrome image. Where do I go from here or have I missed something - I was expecting an RGB.

Interestingly, dark frames that produced very high scaling factors and poor results with optimization checked, processed without issue using the script and optimization selected. The encouraging thing, is that there were only 10 darks (cooled dslr).

Regards,

Rowland

Offline RobF2

  • PixInsight Addict
  • ***
  • Posts: 189
  • Rob
    • View Profile
    • Rob's Astropics
Re: Calibration and align batch operations
« Reply #340 on: 2012 April 29 04:53:30 »
  You are generating masters so don't check use master
Check diagnostic before going to run

Max

I always end up with this error though Max if I start with raw frames and no master options ticked, OR if I use the masters previously generated and have master bias and dark ticked. 

What is this error message supposed to be telling me guys?  Any assistance appreciated.
FSQ106/8" Newt on NEQ6/HEQ5Pro via EQMOD | QHY9 | Guiding:  ZS80II/QHY5IIL | Canon 450D | DBK21 and other "stuff"
Rob's Astropics

Offline mmirot

  • PixInsight Padawan
  • ****
  • Posts: 881
    • View Profile
Re: Calibration and align batch operations
« Reply #341 on: 2012 April 29 08:41:39 »
Brilliant Max! This makes the DSLR_RAW work flow redundant... great stuff!

I have just processed a set of RAW files in very quick time, far quicker than following the DSLR_RAW workflow. The only thing is, I have a very nice monochrome image. Where do I go from here or have I missed something - I was expecting an RGB.

Interestingly, dark frames that produced very high scaling factors and poor results with optimization checked, processed without issue using the script and optimization selected. The encouraging thing, is that there were only 10 darks (cooled dslr).

Regards,

Rowland

Thanks but I am just the messenger. Kia and Juan wrote the script.

It debayers  the image so it should generate R G B calibrated and registered images.  Check the folders for the R G B output . I don't have color camera. May be someone else can help

Max

Offline mmirot

  • PixInsight Padawan
  • ****
  • Posts: 881
    • View Profile
Re: Calibration and align batch operations
« Reply #342 on: 2012 April 29 08:46:34 »
  You are generating masters so don't check use master
Check diagnostic before going to run

Max

I always end up with this error though Max if I start with raw frames and no master options ticked, OR if I use the masters previously generated and have master bias and dark ticked. 

What is this error message supposed to be telling me guys?  Any assistance appreciated.

Sounds ok to me . I have not seen this myself perhaps someone else has an answer


Max

Offline Luigi

  • PixInsight Addict
  • ***
  • Posts: 140
    • View Profile
Re: Calibration and align batch operations
« Reply #343 on: 2012 April 29 12:55:47 »
I just wanted to say thanks very much for this script! Excellent work! It took the script less than 20 minutes to do what would have taken me much longer, and, I didn't have do much except enjoy a nice Rioja and watch the show  ;D

Usage questions:

1) My 'mix' of lights includes exposures of differing lengths as we have no debloomer  tool >:D If I 'pile' all of the lights into the tool, it creates a single master light frame. Since my intention is to use HDR Composition,  do I just need to run the tool separately for each exposure length for which I want to create a master light? The tool does create a separate dark master for each integration time, so maybe I have missed a setting?

2) Since I have exposures of different lengths, the script generated dark masters for each of my exposure times. This is not a related to the tool per se, but am I wasting my time capturing these darks of different lengths?

Between this tool and the solver/annotater script, it's like Christmas in April!
Regards,
Luigi Marchesi

Offline kwalker

  • PixInsight Enthusiast
  • **
  • Posts: 92
    • View Profile
Re: Calibration and align batch operations
« Reply #344 on: 2012 April 29 13:21:44 »
Fantastic new script.
you don't know how long I've been waiting to ditch deepsky stacker :)
Handled my DSLR lights,darks ,flats ,and bias files and didn't kill the hard work collecting the data
Thanks guys