fixed 2nd CPU not recognized/utilized

cray2mpx

cray2mpx

New member
I am running PixInsight 1.8.8-12 on my HP G4-Z8 workstation (dual Xeon Gold 6154, 36 (2x18) cores, 72 (2x36) logical processors, Windows 10 x64 Workstation Edition).
It appears that PixInsight uses only 36 logical processors (or only one CPU) in any multi-threaded task.
Also when running the PixInsight Benchmark it reports 36 logical processors only and the benchmark results are in line with a single CPU system.

There is an older bug report in the forum from 2018 which can be found here:

2nd CPU not recognized | PixInsight Forum

Juan then explained why this bug would exist and that this would get fixed in future Windows releases.

Could it be that this was not addressed since 2018?

Installing Linux on my workstation is not an option for me.
 
Just as a data point, I have a dual Xeon setup (2680v4) with 56 logical processors and PixInsight is able to use all of them. Running Win10x64 Pro, latest version of PixInsight.
 
Of course, the problem described in the 2018's thread that you mention was fixed. The code that retrieves the total number of logical processors on Windows is correct (I've just checked it) and there is no reason for it to fail on your machine. In fact this problem has not been reported again since 2018. Could this be a system hardware/software configuration issue?
 
No, my computer is also configured with two Xeon E5-2699A CPUs with 44 cores and 88 threads. But I found that when PixInsight was processing, only 22 core 44 threads of one CPU were working, while the other CPU was idle all the time!!
 

Attachments

  • 截图20221104005011.png
    截图20221104005011.png
    184.1 KB · Views: 183
I now upgrade the operating system to Windows11 22H2 and PixInsight to 1.8.9-1, but the problems I reported still exist.
But I found that when PixInsight was processing, only 22 core 44 threads of one CPU were working, while the other CPU was idle all the time!!
 
An an FYI: I have the identical problem running Dual E5-2699 V4 Xeons with 44 cores total (22 per chip). Pixinsight only only utilizes Numa Node 0.

Interesting thing is when I ran Dual E5-2658 V3 Xeons with 24 cores total (12 per chip) pixinsight utilized both Numa Node 0 and Node 1 (utilized all cores). Everything was identical with the 2658 V3 chips...same OS, Motherboard, etc...I literally swapped out the 2658 V3 for the the Dual 2699 V4 and Pixinsight started only utilizing the single Numa Node.


Current System Information:
OS Name Microsoft Windows Server 2019
Version 10.0.17763 Build 17763
Processors Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz, 2201 Mhz, 22 Core(s), 44 Logical Processor(s)
Installed Physical Memory (RAM) 256GB
 
A little more context. Rolled back to the E5-2658 and pixinsight utilized all cores. Then swapped back to a single E5-2699 and it used all cores on that chip. Added the 2nd E5-2699 and it continued to use only the cores on a single chip, but did move to Numa Node 1.

Also,
In Preferences->Parallel Processing and Threads is set to the following

1668125181161.png
 
Astro-Not said:
A little more context. Rolled back to the E5-2658 and pixinsight utilized all cores. Then swapped back to a single E5-2699 and it used all cores on that chip. Added the 2nd E5-2699 and it continued to use only the cores on a single chip, but did move to Numa Node 1.

Also,
In Preferences->Parallel Processing and Threads is set to the following
Click to expand...

Astro-Not.

You state that you are running PixInsight on Windows Server 2019.

AFAIK PixInsight is not tested, certified or advertised as being compatible with Windows Server 2019.

What level of Windows Server 2019 license(s) do you have?

Assuming you have a legitimate Windows Server 2019 license, the standard license only allows a maximum of 16 cores to be utilised at any one time, irrespective of how many physical CPU's / Cores are installed.

To access more CPU's / cores requires the purchase of Windows Server 2019 extension licenses, this used to be quite expensive, from Microsoft Server resellers about ~75 USD per core, I have no idea what is available today.

If your Windows Server 2019 license is only the standard version then even though apps such as PixInsight may report that x number of cores are available (derived from reading the hardware tables) Windows Server 2019 will not allow any more than the total number of licensed cores to be accessed.

HTH.

Will.
 
Will,

This is a fully licensed version for 44 cores. It would not have ran the 24 core total for the dual E5-2658 V3s if it didn't have the appropriate number of core packs to exceed the 16 limit.

Having been a hardcore developer and owner of a company with numerous Windows Server licenses for 15 years I can tell you that I have never seen a program that will run on the consumer version of Windows that won't run on a server version unless your code starts touching enterprise features.

The vast majority of developers who write code to run on the server version write and test locally on a "Pro" consumer version prior to checking the code in for CI/CD to deploy to Staging, QA and/or Prod environments.

They literally run the same code.
 
As a quick test to show the behaviour I started WBPP to process some Heart Nebula pics until it hit a point of maxing all cores on a CPU. You can clear see it maxes out NUMA Node 0 which is 22 Cores. NUMA Node 1 was untouched by Pixinsight.

1668173695380.png
 
FYI: I reproduced this workload isolation to a single socket in C# by isolating to a processor group. But, now looking into it looks like PixInsight is written in C++ so will build a test in C++.
 
Astro-Not, have you been able to make a head way on this issue?

I ran across your thread today and realized I have the exact same issue, the 2nd CPU is NOT utilized at all.

Can anyone else confirm this issue where a second CPU is not utilized please?

Thanks.
 
Not yet jfletcher. Holiday workload has slowed me down. Planning on tackling a C++ POC early next year to see if I can reproduce.

What is the your CPUs and what brand and model is your Motherboard? Also does it switch which CPU it uses or is always the same one?
 
Screenshot from my machine showing the same issue that others have seen as well below... As you can see 52 cores are pegged at 100% and the other 52 (second CPU) are idle. The computer is a Dell 7920 Precision Workstation. Just watching the performance tab by eye I cannot see anytime where a process is using anything other then the first logical CPU.

Also I am running Win 10 Pro Workstation as well. I saw there was a comment earlier about OS licensing but I can confirm just like you answered that using other software to bench test like Passmark Performance or CPUIDz that these programs use ALL cores and ALL logical processors so something is a miss with PI when is comes to utilizing multiple CPU's. Not sure if it is something that can be addressed with settings inside PI or if its a code issue with PI.

Thanks.

1671476882545.png
 
Astro-Not, To be fair to PI and the developers, I just saw something that questions the theory of a 2nd CPU not being utilized, at least not 100% of the time... Apparently it does happen, screenshot below from a different Dell Precision 7920 workstation or 2nd almost identical machine that I have. This machine is very similar to the first one except the CPU's are different (Xeon Gold 6226R's vs. Xeon Gold 6230R's) or (52 cores/104 logical vs. 32 cores/64 logical). The two machines act differently. Both screenshots were taken during the cosmetic correction process. The machine with 32 cores/64 logical is utilizing both CPU's. The machine with 52 cores/104 logical is NOT utilizing the 2nd CPU at all. I am testing with the exact same dataset (2100+ images @ 50Mb files) and settings configured in WBPP. Maybe this is more of something to do with the number of cores then then number of CPU's?

1671478600636.png
 
Thank jfletcher. I had similar experience in that one chipset, the lower core count one, did not exhibit this behaviour but the higher count one did. I don't think it is purely core count because if that were the case we would see partial utilization of the 2nd CPU till it hit its max core count in lieu of only using a single socket. As you have figured out the licensing means nothing...the previous poster is confused on how licensing works on Windows Servers. For a while I was concerned it was a memory addressing problem, but eventually found a data set that made PI use 230gbs of my 256 so that wasn't it either. I assume your PI benchmark only uses the single CPU as well?
 
Astro-not,
I need to stop looking at this issue, because it will drive me insane... You asked me this question the other day, if I ever saw the 2nd CPU being utilized if the first one was not and I reported what I had seen up to that point, but it can. So as you will see in the screenshot below, when only one of the CPU's is utilized it is not always the first CPU. Yet more mystery...

1671639684859.png
 
drizf,
There is no solution yet. Astro-Not will be writing some test code in C++ to validate the use of all cores and CPU's soon. I think that will be the next step in trying to figure this out.

Thanks.
 
You must log in or register to reply here.
Back
Top