Processing speed

R

Riccardo A. Ballerini

Well-known member
Hi,
huge cmos cameras are becoming popular and some models come with tiny pixels, like the asi 2600.
Fast ƒ ratio and increased quantum efficiency lead to big numbers of manymegapixel frames.
I understand that java allows Multiplatform releases (an as a OSX user I could not be more grateful of PI existence) but it seems at performance cost...

here is the console log of WBPP for 69 asi2600 frames :(

1 hour and 24 minutes
MacBookPro 4core 2.6MHz i7 16GB SSD 10.13
the cpu was running at 103°C with fans at almost 6k rpm

I have a7.1 MacPro 12core 64GB as well ...and the results are not very different :\


* End integration of Light frames
************************************************************
* Writing master Light frame:

/Users/riccardoangeloballerini/Pictures/Astro/Squid/master/masterLight-BINNING_1-FILTER_NoFilter-EXPTIME_300.xisf
Writing image 'integration': w=6248 h=4176 n=3 RGB Float32
594 FITS keyword(s) embedded.
7 image properties embedded.

* WeightedBatchPreprocessing: 01:24:22.5


Is there a chance that some time intensive tasks will be rewritten in a more efficient way anytime soon?
...otherwise I'll be sadly forced to find a different solution

best regards
Riccardo
 
your premise is that most of the CPU cycles that PI are expending on the WBPP task are wasted? do you have evidence for that claim?

Java has nothing to do with PI and has nothing to do with the cross-platform nature of PI. PI does of course include a JavaScript interpreter but Java and JavaScript have no relationship except the string "Java" in both names, which was clearly a mistake.

while WBPP itself is a javascript program, the bulk of the work is done by the various PI processes. these processes are written in C++ and are compiled to x86 instructions on all 3 platforms. the speed of ImageIntegration, StarAlignment, Debayer, etc. are exactly as fast when called from WBPP as they are when the user directly executes them. you can prove this to yourself by looking at your CPU utilization during WBPP. any parts of the process that are driven by the JavaScript interpreter are single-threaded. if you see more than one core doing something, that is when one of the processes is running.

maybe the problem is simply a huge input dataset. macbooks (and i have several) are widely known to have terrible thermals. the mac pro does not though and i'm sure it can sustain full load without such high CPU temperatures.

you can limit the number of CPU cores PI is allowed to use in Edit > Global Preferences > Parallel Processing and Threads. untick "allow using all available processors" and set the number to maybe 60-80% of the number of cores you have. that should mitigate the overheating but will of course increase your runtime. but you should rest assured that the performance of PI is not affected by its cross-platform nature.
 
Riccardo A. Ballerini said:
I have a7.1 MacPro 12core 64GB as well ...and the results are not very different :\
Click to expand...
This may indicate that your storage performance is the bottleneck. If the storage technology of both computers is similar (e.g., SSDs with similar read/write speeds), it is expected to perform similarly for tasks involving large datasets, despite the differences in CPU and RAM performance.

It is very common to use two languages like Javascript for scripting and C++ for the heavy-lifting. A very popular commercial raw photo manipulation software :censored: is 40% written in the scripting language Lua, while the rest is C++ (according to Wikipedia). These days Javascript isn't that bad, performance-wise. And along with C++, is one of the most popular languages.
 
well the 'problem' with the PI javascript interpreter is that it is single threaded. i don't know if the same is true for the Lua interpreter in LR. hopefully this will change; regardless javascript is not the bottleneck in the problem riccardo is describing.

and you are right about the SSD - WBPP is very disk intensive and it's certainly possible for that to be the actual bottleneck in preprocessing.
 
thank you all for your prompt reply!
I think that SSD at 6 Gigabit sata is quite a standard
The info that the actual processing speed is as fast as can be isn't a good news to me...
I'm using a RASA consequently exposures are not that long ...and it's not unusual doing 100 60" 16bit rgb at 6248x4176 frames!
I'm no programmer expert, I was just hoping in a reasonable processing times with this my new setup
Sorry for barking at the wrong tree :)

btw there are problems debayering as well
the correct rggb matrix leads to wrong colors - grbg works (and yes I tried both up-bottom and bottom-up fit management - tried acquiring straight in xisf as well (with cloudmakers' astroimager ))

Greetings from Rome
Riccardo
 
NVMe SSDs are much faster than SATA. don't both machines have NVMe drives?

perhaps there is a way to speed up the disk writes that ImageCalibration and StarAlignment do - SSDs can accept multiple queued writes if the right interface is used. however a few years back i think Juan refactored all this code to maximize thruput so it could be as fast as it's going to get.

rob
 
i guess the 7,1 has SATA SSDs which is... amazing. they should have used NVMe. i wonder if an external thunderbolt enclosure with an NVMe SSD would be faster.

rob
 
You must log in or register to reply here.
Back
Top