Speed up Local Normalization
- Thread starter bugfood
- Start date
fredvanner
Well-known member
If you find a way, please let me know...
Hi „bugfood“
unfortunately I don’t have the answer, I have the same problem and posted a similar question a few days ago.
Do you observe the same behavior I described in my posting?
https://pixinsight.com/forum/index....rformance-for-a-large-number-of-images.20827/
I wonder if there is a kind of bug, if system resources are filling up, causing the processing speed to drop steadily?
Best
Christof
unfortunately I don’t have the answer, I have the same problem and posted a similar question a few days ago.
Do you observe the same behavior I described in my posting?
https://pixinsight.com/forum/index....rformance-for-a-large-number-of-images.20827/
I wonder if there is a kind of bug, if system resources are filling up, causing the processing speed to drop steadily?
Best
Christof
Kvastronomer
Well-known member
Check this old thread: https://pixinsight.com/forum/index.php?threads/image-integration-efficiency-with-huge-stacks.17306/
There is still no solution. You learn to be patient and stop and restart WBPP multiple times until you get final result.
WBPP occasionally slows down on my PC (usually at integration step) if I have more than 1000 frames. Once you notice that PI is not using 100% CPU (30-60% instead) you can abort the process. If you have doubts that there is a problem, measure how many minutes it takes to change execution percentage counter by 1-2% and estimate the execution time. In my case I usually get answer like - I need 10-20 hours to integrate just one channel. So I stop WBPP and continue manually step by step.
I'm not sure 100% but it seems like this issue only relevant for AMD CPU's. Correct me if I'm wrong.
There is still no solution. You learn to be patient and stop and restart WBPP multiple times until you get final result.
WBPP occasionally slows down on my PC (usually at integration step) if I have more than 1000 frames. Once you notice that PI is not using 100% CPU (30-60% instead) you can abort the process. If you have doubts that there is a problem, measure how many minutes it takes to change execution percentage counter by 1-2% and estimate the execution time. In my case I usually get answer like - I need 10-20 hours to integrate just one channel. So I stop WBPP and continue manually step by step.
I'm not sure 100% but it seems like this issue only relevant for AMD CPU's. Correct me if I'm wrong.
This has been with respect to image integration, I don’t know if this is the same root cause we have here.
Also this has been 2021 and CPUs got more and more cores in the past and will probably get more in future, especially when it comes to high end PCs. With CMOS cameras the use case that we users of PI have to process larger and larger image stacks this becomes even more of a problem. We have to use CPUs with more cores to get our data processed.
Is there any feedback from the PI team?
I have been working in a software company processing large 3D image data sets. Large in this case means 128 GB and more per data set.
We ran into many issues with larger and larger core numbers. Windows has a limitation of 64 cores / threads programmers have to work around if they want to utilize more. We changed our codes years ago to overcome limitations. Please don‘t start the WIN vs Linux discussion as all users have their preferences and luckily PI gives us the freedom to use the OS we prefer. This was the same in my company, we also supported the three OS. We invested in all three to run at the same performance if the underlying hardware and software layers allowed it.
In teams of multi threading we spend a lot of time to optimize code, e.g. in terms of memory allocation, concurring data access, etc. to allow users to get the maximum performance out of their hardware. My guess is that for LN there is an issue in the code slowing the process down for image stacks with hundreds or thousands of files.
Is the PI team still not able to reproduce this behavior? If not please let me know. I’m happy to support you localizing this. This was part of my job in my company.
Best
Christof
Also this has been 2021 and CPUs got more and more cores in the past and will probably get more in future, especially when it comes to high end PCs. With CMOS cameras the use case that we users of PI have to process larger and larger image stacks this becomes even more of a problem. We have to use CPUs with more cores to get our data processed.
Is there any feedback from the PI team?
I have been working in a software company processing large 3D image data sets. Large in this case means 128 GB and more per data set.
We ran into many issues with larger and larger core numbers. Windows has a limitation of 64 cores / threads programmers have to work around if they want to utilize more. We changed our codes years ago to overcome limitations. Please don‘t start the WIN vs Linux discussion as all users have their preferences and luckily PI gives us the freedom to use the OS we prefer. This was the same in my company, we also supported the three OS. We invested in all three to run at the same performance if the underlying hardware and software layers allowed it.
In teams of multi threading we spend a lot of time to optimize code, e.g. in terms of memory allocation, concurring data access, etc. to allow users to get the maximum performance out of their hardware. My guess is that for LN there is an issue in the code slowing the process down for image stacks with hundreds or thousands of files.
Is the PI team still not able to reproduce this behavior? If not please let me know. I’m happy to support you localizing this. This was part of my job in my company.
Best
Christof