[Question] Why are we getting the number of cores, rather than the number of logical processors?

[tannergooding]

In our build script, we are calling wmic cpu get NumberOfCores: https://github.com/dotnet/coreclr/blob/5a01d8a39576f0d07441e4d123a90138f4ae0735/build.cmd#L323

This seems like the wrong thing to be grabbing as it fails to return the proper number of cores for a machine with SMT (simultaneous multi-threading) support (machine will be under-utilized).

It will also report an incorrect number of cores if the user has disabled one or more cores via the UEFI/BIOS, or some other mechanism (machine will be over-utilized).

NumberOfCores should likely be used in conjunction with one or more of NumberOfEnabledCore, NumberOfLogicalProcessors, and ThreadCount.

This will ensure that the processor is properly utilized based on its actual configuration.

[4creators]

The code for optimizing native build has to tackle entanglement of MSBuild and CL parallelizm control to get any significant performance and memory usage improvements. By entanglement I understand that it is difficult to control native solution build parallelism in any other way than by /maxcpucount:x MSBuild command line parameter. In currently used solution this was disentangled by using in a first attempt CL_MPCount MSBuild property to control CL parallelism which was set to value of NumberOfEnabledCore and /maxcpucount:2 to enable MSBuild parallelism and theoretically limiting it to 2 MSBuild nodes (practically it gives 3 Tracker.exe nodes each running CL_MPCount CL processes). Usage of NumberOfEnabledCore was justified in original PR dotnet/coreclr#14509 just as you have described:

will also report an incorrect number of cores if the user has disabled one or more cores via the UEFI/BIOS

As it turns out NumberOfEnabledCore WMI CPU property is not supported on virtual machines and it resulted in filing an issue #9170. IMO it is a Windows bug and a reasonable way to workaround this issue is to use NumberOfCores property.

Why not to use other processor data: NumberOfLogicalProcessors or ThreadCount?

Current solution is suboptimal due to MSBuild handling of multiple vcxproj build processes and getting best results would require changes to MSBuild native build logic. It was optimized empirically on SMT processor 4 cores / 8 threads (see dotnet/coreclr#14509 (comment)) and it could be close to what can be achieved but if there are any other optimization data points indicating that we can do better there is no reason to stick to current properties or values. But in general I assume that it would be better investment to improve MSBuild than work on current solution.

IMO it should be possible to shave off additional 20% build time on top of earlier 11% if optimizations would be applied to native project build graph on MSBuild level. This perhaps would further reduce memory usage as well. I plan to look into this ... but don't know when I will find time for doing it.

[dotnet-policy-service]

Due to lack of recent activity, this issue has been marked as a candidate for backlog cleanup. It will be closed if no further activity occurs within 14 more days. Any new comment (by anyone, not necessarily the author) will undo this process.

This process is part of our issue cleanup automation.