Calibration and align batch operations

[Alejandro Tombolini]

(and possibly already stacked) frames. 

CBR works perfect in stacked images (Canon 450 and 5D at least), it is only a matter of find the correct values, specially 1/sigma factor when are bright areas. No need to be applied in individual frames.
Saludos, Alejandro.

[georg.viehoever]

Sorry, my statement probably was misleading . What I wanted to say is that the current CBR implementation needs debayered images, not RAW images. The earliest point where the BatchProcessing process  produces debayered images is after calibration, after registration, and after integration. So you currently cannot use CBR earlier in the process.

Also, the benefit of doing it at an earlier stage currently is unclear, see the links.

Georg

[mmirot]

Sorry did you say something 

Harry

I'm used to be being ignored

I am still waiting to hear if we add overscan to the script. I
 know NKV and Vicent would like this too.

Max

[erikgu]

Hi, sorry for wanting CBR on a wrong stage in the processing order. I think from the test i did that running the script before registration gave a better result than on the stacked image. Might be because i stacked images taken over several days with not exactly the same camera orientation. Result of the the registration would then give not exactly horizontal and vertical bandings, correct ?

Erik G

[pfile]

Hi, sorry for wanting CBR on a wrong stage in the processing order. I think from the test i did that running the script before registration gave a better result than on the stacked image. Might be because i stacked images taken over several days with not exactly the same camera orientation. Result of the the registration would then give not exactly horizontal and vertical bandings, correct ?

Erik G

yes, that's right. if you have enough nights each with their own subtly different PA the banding can get all crazy. but if you have enough different PAs you can make stacks of each night, do CBR, and then stack the stacks...

[Juan Conejero]

Hi everybody,

I have just released version 1.05 as an update (for developers, the full source code is also attached to this post). This is a quick bugfix release while we prepare a more substantial update during the next days:

- FITS files with the .fts extension were not properly recognized in previous versions due to a bug in the File.loadFITSKeywords() helper routine. This has been fixed.

- Fixed a bug in the Add Custom Frames dialog: When no explicit selection was done in the Image type combo box, an invalid image type was selected.

Thanks to Andrew (atheelen) for uploading a raw data sample which I have used to fix the .fts bug.

Thanks to everybody for your patience and help with this script.

A few ideas that we want to implement in the next version:

* Support for multiple objects in a single instance of the BatchPreprocessing script. Some professional astronomers have asked us to add this feature.

* Optional automatic removal of intermediate calibrated images (e.g. calibrated bias, dark and flat frames).

* Automatic calculation of the required disk space and check available space during the diagnostics process.

* When one of the files being calibrated is selected as registration reference, use the corresponding calibrated/deBayered frame instead of the original frame.

Enjoy!

[georg.viehoever]

Hi everybody,

I have just released version 1.05 as an update (full source code attached to this post). This is a quick bugfix release while we prepare a more substantial update during the next days:
...

Great! Has the updated cosmetic correction module also been released?
http://pixinsight.com/forum/index.php?topic=3483.msg29800#msg29800

...
* Support for multiple objects in a single instance of the BatchPreprocessing script. Some professional astronomers have asked us to add this feature.
...

Professional astronomers are using  a script intended for newbies? 
Georg

[Juan Conejero]

Great! Has the updated cosmetic correction module also been released?

Yes, I have released it also this morning. Sorry I forgot to mention it.

Professional astronomers are using  a script intended for newbies?

They are very interested in an automatized solution to the whole preprocessing task. Having IRAF as the only alternative, our BPP script is seen as a heaven gift

Usually astrometry and photometry programs generate data for many objects during the same night. So they really need the script to support all the data from multiple objects in a single batch.

[mmirot]

Professional astronomers are using  a script intended for newbies?

They are very interested in an automatized solution to the whole preprocessing task. Having IRAF as the only alternative, our BPP script is seen as a heaven gift

Usually astrometry and photometry programs generate data for many objects during the same night. So they really need the script to support all the data from multiple objects in a single batch.
[/quote]

They must be asking of bias oversscan too?

Max

[mmirot]

Great! Has the updated cosmetic correction module also been released?

Yes, I have released it also this morning. Sorry I forgot to mention it.

Professional astronomers are using  a script intended for newbies?

They are very interested in an automatized solution to the whole preprocessing task. Having IRAF as the only alternative, our BPP script is seen as a heaven gift

Usually astrometry and photometry programs generate data for many objects during the same night. So they really need the script to support all the data from multiple objects in a single batch.

They must asking for bias overscan too?

Max

[Harry page]

Hi Harry,

The latest version seems to have lost the ability to save any image  files when you save a "new instance"

Does this happen consistently? I have just checked the script and it saves/restores all file items normally.

Hi
Right Have found this bug , if you uncheck the " export calibration files " then saving the settings with new instance does not work and forgets the selected files

Harry

P.s.

I do wish you would look at the Debayer settings not right in my mind 

[cs_pixinsight]

Is there a reason debayering and noise estimation is a sequential operation in the current version?  The total processing time would decrease quite a bit if it could occur in parallel like CosmeticCorrection is currently doing.  Only using 1 of 8 cores seems like a waste of resources 

Craig

Juan, any chance more multi-threading can be added to these areas?

Craig

[cs_pixinsight]

Hi Harry,

The latest version seems to have lost the ability to save any image  files when you save a "new instance"

Does this happen consistently? I have just checked the script and it saves/restores all file items normally.

Hi
Right Have found this bug , if you uncheck the " export calibration files " then saving the settings with new instance does not work and forgets the selected files

Harry

P.s.

I do wish you would look at the Debayer settings not right in my mind 

Harry, I reported this same problem here:  http://pixinsight.com/forum/index.php?topic=3654.msg28997#msg28997

It was fixed at one point, so I'm not sure how it has returned.

Craig

[Juan Conejero]

Right Have found this bug , if you uncheck the " export calibration files " then saving the settings with new instance does not work and forgets the selected files

This is precisely the purpose of the "Export calibration files" option. When enabled (default state), newly created instances will contain all file paths selected in all frame groups. When this option is disabled, new instances will contain no file paths but only calibration, integration, registration and deBayering parameters.

Maybe the tool tip information is not clear enough for this check box?

[Juan Conejero]

Is there a reason debayering and noise estimation is a sequential operation in the current version?  The total processing time would decrease quite a bit if it could occur in parallel like CosmeticCorrection is currently doing.  Only using 1 of 8 cores seems like a waste of resources 

Craig

Juan, any chance more multi-threading can be added to these areas?

Craig

Hi Craig,

All processes currently executed by the BatchPreprocessing script are parallelized. However, not all of them are parallelized in the same way, neither all of them are using multithreading with the same efficiency for large batch tasks.

The ImageCalibration and CosmeticCorrection tasks use high-level parallelization. Basically, to process N files with P worker threads, each worker thread runs independently for a sublist of N/P files. There is an additional coordination thread for worker thread supervision and file reading/writing tasks. For large file lists, this is usually the most efficient implementation. ImageIntegration also uses a very efficient multithreading scheme.

Other processes use low-level parallelization, which in some cases is not appropriate for the BPP script. For example, some parts of the StarAlignment task are parallelized, such as the star matching and RANSAC routines. This is very efficient to process a few images (e.g. for mosaic construction, or to align less images than the number of processors available), but rather inefficient to align large sets of disk files. A future version of the StarAlignment process will use a high-level parallelization scheme, which will improve the performance of the BPP script considerably. This is near the top of the to-do list.

The whole PixInsight platform will be improved significantly for multithreaded execution in future versions. There is still a lot of work to do in this regard.