More parallelization

[Xplode]

I'm using BPP for calibration of my images and it would be nice to see it work in parallel on several images, it seems to be stuck to around 16% CPU usage with my 3950x.
There's probably other parts of PI that is time consuming that could get a high speed boost from parallelization too.

[pfile]

a good deal of calibration/registration is disk-bound, lots of reading and writing of files. so i'm not surprised to see low CPU utilization on average. juan did rejigger things a few years back to make the registration process a little better at hiding file IO latency.

most of the processes are parallelized and do see good speedup on multicore machines.

rob

[Xplode]

I'm using NVME disks with 3Gb/s + read/write so hdd speed shouldn't be a problem.
Registration is for sure working on multiple files at the same time with 70%+ cpu usage, but calibration is only working on a single file, this is also easy to see when watching the process console.

The increase in speed with some of the parallelization added in the last years has been awesome, seems to be that calibration is the last missing part for the puzzle to get even higher speedup

[Juan Conejero]

Please give a read to this post:

https://pixinsight.com/forum/index.php?topic=10588.0

where I describe the state of parallelization in current versions of PixInsight, and provide important information on the repercussion of I/O wait states for data and processor intensive tasks.

As you see in the article above, all image preprocessing tasks are already using high-level parallelization in current versions of PixInsight. Take into account that there are very important performance differences among operating systems. In descending order of performance: Linux, FreeBSD, macOS and Windows.

Instead of BPP, which is now obsolete and maintained only for compatibility with existing projects, you should use the new WeightedBatchPreprocessing script (WBPP). The image weighting task in this script requires high-level parallelization, just as implemented in the SubframeSelector tool. I am working on it, time permitting.

[Juan Conejero]

calibration is only working on a single file

This is strange, since ImageCalibration uses high-level parallelization. It does since its first version, released many years ago. The information given on console is slightly different from other tools such as SA or II (because of IC's age), but if you search for it, you'll see a 'Waiting for running tasks to terminate' message.

[Xplode]

By looking closer at what happens i see that i'm wrong about calibration not using parallelization, but it doesn't change the fact that there's a bottleneck somehwere since it uses just 17-18% cpu at max (for PI alone)
During registration cpu usage jumps to 100% for the full system 

I will investigate further and see i can find something that makes a difference to the cpu usage.

I'm working with 500+ 60mpix files from the ASI6200 so being able to speedup calibration 3-4x would be a hugh time saving

[Juan Conejero]

Two questions:

- Are you using Windows?

- Are you using FITS files?

If yes to both, then I guess I know where the problem is. As you probably know we use NASA's CFITSIO library as the back-end for our FITS support module. The Windows implementation of CFITSIO is not thread-safe for disk I/O operations (we have had many stability issues in the past because of this problem), and hence we have disabled parallel file accesses for the Windows build of our FITS module. The Linux, FreeBSD and macOS versions don't have this problem.

This means that although the calibration process is fully parallelized on all platforms, loading a FITS file is a thread-locking task on Windows. This problem does not happen during the rest of the preprocessing pipeline because the XISF format implementation is of course fully thread-safe in every aspect.

[Xplode]

Yes i'm on Windows and the files are of course FITS before i do anything with them, i'd love to be able to convert all files easily to XISF, but it's a lot of work to do this in PI every day and of course time consuming.
I've asked for a standalone program before that could handle the conversion to XISF and delete the FITS files automatically, it would have to scan through folders instead if having to select the files manually.
Being able to run a software like this would be a great help, being able to compress them at the same time would also be a great help in saving some space for arhciving files.

I will try to push more for capturing softwares to support XISF, but seems it's very slow to get it added

[Juan Conejero]

On revising ImageCalibration's source code, I see that we are actually reading input data from the main thread, that is, the file loading process is not running in parallel. This is a reminiscence of the old times of rotational disks. So we have some room for improvement here. However, note that this won't change the situation I have described above for FITS files on Windows.

It's clear that our ImageCalibration tool is very old and requires a revamp. The performance improvement will be modest but still significant. I'll implement it as soon as possible, thank you for pointing this out.

[Juan Conejero]

I will try to push more for capturing softwares to support XISF, but seems it's very slow to get it added

Thank you very much for your support. With a few but important exceptions, the lack of interest in XISF is general, although has always been foreseeable. So any help in this regard is always welcome, not as a way to support PixInsight, but to improve interoperability and efficiency for everybody.

I'm going to review current versions of CFITSIO in terms of thread safety on Windows. Maybe we can make a step forward here.

[Xplode]

Thanks for looking into this.
It will be interesting to test the revamped ImageCalibration 

[Xplode]

Could you also add parallelization to BatchFormatConversion? It's painfully slow since it works with 1 image