Author Topic: Calibration and align batch operations  (Read 135701 times)

Offline kwiechen

  • PixInsight Addict
  • ***
  • Posts: 186
    • View Profile
Re: Calibration and align batch operations
« Reply #105 on: 2012 March 16 16:43:00 »
Hi,

some additional code to use files without FITS keywords will be added.   

Kai


Offline JamieInCLT

  • Newcomer
  • Posts: 45
    • View Profile
    • My Astrobin
Re: Calibration and align batch operations
« Reply #106 on: 2012 March 16 16:51:01 »
Very cool, please make it work directly with RAW files, too :D, and not just fits

Offline georg.viehoever

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2132
    • View Profile
Re: Calibration and align batch operations
« Reply #107 on: 2012 March 16 17:47:25 »
Very cool, please make it work directly with RAW files, too :D, and not just fits
+1
Georg (6 inch Newton, unmodified Canon EOS40D+80D, unguided EQ5 mount)

Offline mmirot

  • PixInsight Padawan
  • ****
  • Posts: 881
    • View Profile
Re: Calibration and align batch operations
« Reply #108 on: 2012 March 16 18:43:10 »
It is great to see such interest in Kai's project.  8)

Max

Offline troypiggo

  • PixInsight Addict
  • ***
  • Posts: 258
    • View Profile
Re: Calibration and align batch operations
« Reply #109 on: 2012 March 17 14:18:10 »
I'm finding that if I use this script to generate master frames only, no lights, the next time I run it I get an error:

Code: [Select]
Processing script file: /Users/troy/Applications/pixinsight/CalibrateAlign.js
using config file /tmp/CalibrateAlign.cfg
*** Error [000]: /Users/troy/Applications/pixinsight/CalibrateAlign-settings.js, line 11: Error: File I/O Error: Unexpected end of file: /tmp/CalibrateAlign.cfg

I can manually delete the /tmp/CalibrateAlign.cfg but it's not really the desirable behaviour.

astropixel

  • Guest
Re: Calibration and align batch operations
« Reply #110 on: 2012 March 17 16:10:21 »
I second requests for handling raw files.

Offline kwiechen

  • PixInsight Addict
  • ***
  • Posts: 186
    • View Profile
Re: Calibration and align batch operations
« Reply #111 on: 2012 March 19 00:55:41 »
I have added some lines to make the script ready to use FITS files without keywords and OSC FITS. I will need some time to test and to debug this. Is there any difference when calibrating flat/light frames from OSC cameras compared to a monochrome CCD (detect CFA ??)?

Kai

 


Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7089
    • View Profile
    • http://pixinsight.com/
Re: Calibration and align batch operations
« Reply #112 on: 2012 March 19 03:02:29 »
Hi all,

I have made a first revision of Kai's script. You'll find it attached to this post.

First of all I want to say a big thanks to Kai for his nice initiative with this script.  His work is excellent and the script is clear and well designed and structured. It has been a pleasure working on it.

The first change you'll notice is a new name: Batch Preprocessing. I think this script has the potential to become a complete preprocessing pipeline for PixInsight, so a more general name was required in my opinion.

The revised version of the script does not rely so heavily on FITS headers. Now you can manually add bias, dark, flat and light frames that the script will load as such without further questions (you are supposed to know what you are doing in such case). Of course, the "Add Files" button is still there and works by selecting frame types automatically, based on standard FITS keywords. Both "Bias Frame" and "BIAS" keyword flavors are now supported.

The independence on FITS headers should make the script much more flexible. It should work now with raw frames generated by applications that don't store frame type keywords (which is a very bad practice IMO) and opens the door to support DSLR and OSC raw frames with little effort.

A change that all of you will love for sure is that the script can now generate instances. I have replaced Kai's settings system based on configuration files (which was efficient and very well designed by the way) with a standard import/export system based on the Parameters PJSR object. This means that the script integrates tightly with the platform, so you can generate process icons with your preprocessing pipelines and store them in XPSM icon files and XOSM project files.

The script now disables ImageIntegration's file cache for integration of bias, dark and flat frames. This greatly improves efficiency, since it prevents calculation of unnecessary noise estimates.

I have simplified the image integration parameters section. Integration and rejection normalization cannot be selected now. This is because these parameters are always fixed as either "no normalization" and "equalize fluxes", depending on frame types (the latter for flat frames), and changing them makes no sense. I have removed CCD Noise Model rejection because it is only of academic interest. I have added support for min/max rejection because it can be necessary sometimes. Finally, I have removed the image weighting parameter because calibration raw frames are never weighted. If you think that one of these parameters should be restored, please argument why, and we'll put it back.

Another change is the way master frames are managed from the GUI. Previously when you selected one of the "Use as master" options the corresponding list of files was cleared. This no longer happens, so you are free to enable or disable these options. This is more friendly in my opinion, and although it increases the risk of doing things wrong, will lead to increased flexibility when we implement a more complete management of file lists (e.g., when we can remove selected frames).

Finally I have made a GUI revamp with new buttons, some reorganization, more graphical elements and extensive use of Qt CSS styling. During this process I have also discovered a couple of minor bugs in the PixInsight Core application (some faulting PJSR routines), for  which I have found good workarounds, fortunately.

The script is perfectly usable right now. It is rather stable and robust, and yields excellent results as far as I have tested it. A lot of work remains to be done, however. These are just a few pending tasks:

* URGENT: Decide on a software license to release this script as official, via the update system. Our policy is to never release source code without a clearly stated license. I suggest using our PCL License v1.0, but this is of course Kai's decision.

* Add support for DSLR and OSC Bayer CFA raw frames. Not difficult now that the script doesn't depend on FITS headers.

* Add more file management features: remove selected frames, use enabled/disabled frame states, manage selections, etc.

* Improve diagnostics and error information.

* Make the script more robust against failed processes.

Hope you like it!
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline erikgu

  • PixInsight Enthusiast
  • **
  • Posts: 96
    • View Profile
Re: Calibration and align batch operations
« Reply #113 on: 2012 March 19 04:26:47 »
Hi !

Thank's Juan. Realy good news.  :D

Erik G.

Offline kwiechen

  • PixInsight Addict
  • ***
  • Posts: 186
    • View Profile
Re: Calibration and align batch operations
« Reply #114 on: 2012 March 19 05:16:50 »
Juan,

thank you very much for your work!

Quote
* URGENT: Decide on a software license to release this script as official, via the update system. Our policy is to never release source code without a clearly stated license. I suggest using our PCL License v1.0, but this is of course Kai's decision.

Please take the PCL License as suggested.

Best,

Kai


Offline mmirot

  • PixInsight Padawan
  • ****
  • Posts: 881
    • View Profile
Re: Calibration and align batch operations
« Reply #115 on: 2012 March 19 08:18:33 »
Great can wait to try it out. Sounds good.

Max

Offline georg.viehoever

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2132
    • View Profile
Re: Calibration and align batch operations
« Reply #116 on: 2012 March 19 09:16:44 »
A dream becomes true  8)
Georg (6 inch Newton, unmodified Canon EOS40D+80D, unguided EQ5 mount)

ruediger

  • Guest
Re: Calibration and align batch operations
« Reply #117 on: 2012 March 19 09:39:26 »
Very good! Thanks for the great work! Once debayering is included, this script will really speedup a typical dslr workflow.

One minor thing: the new csv file type as reference image type is not included. I have not really tried out this new alignment feature, but it looks very helpful if you have mosaic generation in mind.

RĂ¼diger

Offline bitli

  • PTeam Member
  • PixInsight Guru
  • ****
  • Posts: 511
    • View Profile
Re: Calibration and align batch operations
« Reply #118 on: 2012 March 19 11:29:26 »
The greatest think since slice bread.  Combined with the Blink process, this makes PI very useable.  I will no longer have to choose between spending a night doing manual process or having to switch to another program.
Thanks you very much.
-- bitli

Offline Harry page

  • PTeam Member
  • PixInsight Jedi Knight
  • *****
  • Posts: 1457
    • View Profile
    • http://www.harrysastroshed.com
Re: Calibration and align batch operations
« Reply #119 on: 2012 March 19 11:32:27 »
Hi

If It works I will make a vid , will stop all the moaning hopefully

Harry
Harry Page