Author Topic: Calibration and align batch operations  (Read 135810 times)

Offline Alejandro Tombolini

  • PTeam Member
  • PixInsight Jedi
  • *****
  • Posts: 1267
    • View Profile
    • Próxima Sur
Re: Calibration and align batch operations
« Reply #390 on: 2012 May 09 07:11:20 »
(and possibly already stacked) frames. 

CBR works perfect in stacked images (Canon 450 and 5D at least), it is only a matter of find the correct values, specially 1/sigma factor when are bright areas. No need to be applied in individual frames.
Saludos, Alejandro.

Offline georg.viehoever

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2132
    • View Profile
Re: Calibration and align batch operations
« Reply #391 on: 2012 May 09 08:04:56 »
Sorry, my statement probably was misleading :'( . What I wanted to say is that the current CBR implementation needs debayered images, not RAW images. The earliest point where the BatchProcessing process  produces debayered images is after calibration, after registration, and after integration. So you currently cannot use CBR earlier in the process.

Also, the benefit of doing it at an earlier stage currently is unclear, see the links.

Georg
Georg (6 inch Newton, unmodified Canon EOS40D+80D, unguided EQ5 mount)

Offline mmirot

  • PixInsight Padawan
  • ****
  • Posts: 881
    • View Profile
Re: Calibration and align batch operations
« Reply #392 on: 2012 May 09 09:29:48 »

Sorry did you say something  >:D :-*

Harry


I'm used to be being ignored :o

I am still waiting to hear if we add overscan to the script. I
 know NKV and Vicent would like this too.

Max

Offline erikgu

  • PixInsight Enthusiast
  • **
  • Posts: 96
    • View Profile
Re: Calibration and align batch operations
« Reply #393 on: 2012 May 09 21:31:46 »
Hi, sorry for wanting CBR on a wrong stage in the processing order. I think from the test i did that running the script before registration gave a better result than on the stacked image. Might be because i stacked images taken over several days with not exactly the same camera orientation. Result of the the registration would then give not exactly horizontal and vertical bandings, correct ?

Erik G

Offline pfile

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 4693
    • View Profile
Re: Calibration and align batch operations
« Reply #394 on: 2012 May 10 08:46:10 »
Hi, sorry for wanting CBR on a wrong stage in the processing order. I think from the test i did that running the script before registration gave a better result than on the stacked image. Might be because i stacked images taken over several days with not exactly the same camera orientation. Result of the the registration would then give not exactly horizontal and vertical bandings, correct ?

Erik G

yes, that's right. if you have enough nights each with their own subtly different PA the banding can get all crazy. but if you have enough different PAs you can make stacks of each night, do CBR, and then stack the stacks...

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7096
    • View Profile
    • http://pixinsight.com/
Re: Calibration and align batch operations
« Reply #395 on: 2012 May 11 04:41:05 »
Hi everybody,

I have just released version 1.05 as an update (for developers, the full source code is also attached to this post). This is a quick bugfix release while we prepare a more substantial update during the next days:

- FITS files with the .fts extension were not properly recognized in previous versions due to a bug in the File.loadFITSKeywords() helper routine. This has been fixed.

- Fixed a bug in the Add Custom Frames dialog: When no explicit selection was done in the Image type combo box, an invalid image type was selected.

Thanks to Andrew (atheelen) for uploading a raw data sample which I have used to fix the .fts bug.

Thanks to everybody for your patience and help with this script.

A few ideas that we want to implement in the next version:

* Support for multiple objects in a single instance of the BatchPreprocessing script. Some professional astronomers have asked us to add this feature.

* Optional automatic removal of intermediate calibrated images (e.g. calibrated bias, dark and flat frames).

* Automatic calculation of the required disk space and check available space during the diagnostics process.

* When one of the files being calibrated is selected as registration reference, use the corresponding calibrated/deBayered frame instead of the original frame.

Enjoy!
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline georg.viehoever

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2132
    • View Profile
Re: Calibration and align batch operations
« Reply #396 on: 2012 May 11 04:45:46 »
Hi everybody,

I have just released version 1.05 as an update (full source code attached to this post). This is a quick bugfix release while we prepare a more substantial update during the next days:
...
Great! Has the updated cosmetic correction module also been released?
http://pixinsight.com/forum/index.php?topic=3483.msg29800#msg29800

...
* Support for multiple objects in a single instance of the BatchPreprocessing script. Some professional astronomers have asked us to add this feature.
...
Professional astronomers are using  a script intended for newbies?  >:D
Georg
Georg (6 inch Newton, unmodified Canon EOS40D+80D, unguided EQ5 mount)

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7096
    • View Profile
    • http://pixinsight.com/
Re: Calibration and align batch operations
« Reply #397 on: 2012 May 11 05:00:07 »
Quote
Great! Has the updated cosmetic correction module also been released?

Yes, I have released it also this morning. Sorry I forgot to mention it.

Quote
Professional astronomers are using  a script intended for newbies?

They are very interested in an automatized solution to the whole preprocessing task. Having IRAF as the only alternative, our BPP script is seen as a heaven gift :)

Usually astrometry and photometry programs generate data for many objects during the same night. So they really need the script to support all the data from multiple objects in a single batch.
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline mmirot

  • PixInsight Padawan
  • ****
  • Posts: 881
    • View Profile
Re: Calibration and align batch operations
« Reply #398 on: 2012 May 11 09:00:11 »

Quote
Professional astronomers are using  a script intended for newbies?

They are very interested in an automatized solution to the whole preprocessing task. Having IRAF as the only alternative, our BPP script is seen as a heaven gift :)

Usually astrometry and photometry programs generate data for many objects during the same night. So they really need the script to support all the data from multiple objects in a single batch.
[/quote]

They must be asking of bias oversscan too?

Max

Offline mmirot

  • PixInsight Padawan
  • ****
  • Posts: 881
    • View Profile
Re: Calibration and align batch operations
« Reply #399 on: 2012 May 11 09:02:45 »
Quote
Great! Has the updated cosmetic correction module also been released?

Yes, I have released it also this morning. Sorry I forgot to mention it.

Quote
Professional astronomers are using  a script intended for newbies?

They are very interested in an automatized solution to the whole preprocessing task. Having IRAF as the only alternative, our BPP script is seen as a heaven gift :)

Usually astrometry and photometry programs generate data for many objects during the same night. So they really need the script to support all the data from multiple objects in a single batch.

They must asking for bias overscan too?

Max

Offline Harry page

  • PTeam Member
  • PixInsight Jedi Knight
  • *****
  • Posts: 1457
    • View Profile
    • http://www.harrysastroshed.com
Re: Calibration and align batch operations
« Reply #400 on: 2012 May 11 12:13:59 »
Hi Harry,

The latest version seems to have lost the ability to save any image  files when you save a "new instance"

Does this happen consistently? I have just checked the script and it saves/restores all file items normally.

Hi
Right Have found this bug , if you uncheck the " export calibration files " then saving the settings with new instance does not work and forgets the selected files

Harry

P.s.

I do wish you would look at the Debayer settings not right in my mind  >:D

Harry Page

Offline cs_pixinsight

  • PixInsight Addict
  • ***
  • Posts: 156
    • View Profile
Re: Calibration and align batch operations
« Reply #401 on: 2012 May 11 12:48:57 »
Is there a reason debayering and noise estimation is a sequential operation in the current version?  The total processing time would decrease quite a bit if it could occur in parallel like CosmeticCorrection is currently doing.  Only using 1 of 8 cores seems like a waste of resources  ;)

Craig

Juan, any chance more multi-threading can be added to these areas?

Craig

Offline cs_pixinsight

  • PixInsight Addict
  • ***
  • Posts: 156
    • View Profile
Re: Calibration and align batch operations
« Reply #402 on: 2012 May 11 12:51:57 »
Hi Harry,

The latest version seems to have lost the ability to save any image  files when you save a "new instance"

Does this happen consistently? I have just checked the script and it saves/restores all file items normally.

Hi
Right Have found this bug , if you uncheck the " export calibration files " then saving the settings with new instance does not work and forgets the selected files

Harry

P.s.

I do wish you would look at the Debayer settings not right in my mind  >:D

Harry, I reported this same problem here:  http://pixinsight.com/forum/index.php?topic=3654.msg28997#msg28997

It was fixed at one point, so I'm not sure how it has returned.

Craig

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7096
    • View Profile
    • http://pixinsight.com/
Re: Calibration and align batch operations
« Reply #403 on: 2012 May 12 02:38:29 »
Right Have found this bug , if you uncheck the " export calibration files " then saving the settings with new instance does not work and forgets the selected files

This is precisely the purpose of the "Export calibration files" option. When enabled (default state), newly created instances will contain all file paths selected in all frame groups. When this option is disabled, new instances will contain no file paths but only calibration, integration, registration and deBayering parameters.

Maybe the tool tip information is not clear enough for this check box?
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7096
    • View Profile
    • http://pixinsight.com/
Re: Calibration and align batch operations
« Reply #404 on: 2012 May 12 03:06:58 »
Is there a reason debayering and noise estimation is a sequential operation in the current version?  The total processing time would decrease quite a bit if it could occur in parallel like CosmeticCorrection is currently doing.  Only using 1 of 8 cores seems like a waste of resources  ;)

Craig

Juan, any chance more multi-threading can be added to these areas?

Craig

Hi Craig,

All processes currently executed by the BatchPreprocessing script are parallelized. However, not all of them are parallelized in the same way, neither all of them are using multithreading with the same efficiency for large batch tasks.

The ImageCalibration and CosmeticCorrection tasks use high-level parallelization. Basically, to process N files with P worker threads, each worker thread runs independently for a sublist of N/P files. There is an additional coordination thread for worker thread supervision and file reading/writing tasks. For large file lists, this is usually the most efficient implementation. ImageIntegration also uses a very efficient multithreading scheme.

Other processes use low-level parallelization, which in some cases is not appropriate for the BPP script. For example, some parts of the StarAlignment task are parallelized, such as the star matching and RANSAC routines. This is very efficient to process a few images (e.g. for mosaic construction, or to align less images than the number of processors available), but rather inefficient to align large sets of disk files. A future version of the StarAlignment process will use a high-level parallelization scheme, which will improve the performance of the BPP script considerably. This is near the top of the to-do list.

The whole PixInsight platform will be improved significantly for multithreaded execution in future versions. There is still a lot of work to do in this regard.
Juan Conejero
PixInsight Development Team
http://pixinsight.com/