From @|mon@urb@nek @end|ng |rom R-project@org  Tue Aug  8 03:17:55 2023
From: @|mon@urb@nek @end|ng |rom R-project@org (Simon Urbanek)
Date: Tue, 8 Aug 2023 13:17:55 +1200
Subject: [Rd] 
 Detecting physical CPUs in detectCores() on Linux platforms
In-Reply-To: <25809.34633.249825.74893@rob.eddelbuettel.com>
References: <CAObhMffcPOLHN_ko1VQEu-taHjCiuc5kPCA9ybS1O9r_UOn1GQ@mail.gmail.com>
 <25808.57266.337153.29417@rob.eddelbuettel.com>
 <20c86758cf57baa0c045aa39d99bc950a8cb8b31.camel@uni-jena.de>
 <CC836864-15F7-4D44-BED6-20CA02FCD8BA@R-project.org>
 <25809.34633.249825.74893@rob.eddelbuettel.com>
Message-ID: <0DCAA0B2-CE51-4518-B7A5-95B737D1281F@R-project.org>



> On 8/08/2023, at 12:07 PM, Dirk Eddelbuettel <edd at debian.org> wrote:
> 
> 
> On 8 August 2023 at 11:21, Simon Urbanek wrote:
> | First, detecting HT vs cores is not necessarily possible in general, Linux may assign core id to each HT depending on circumstances:
> | 
> | $ grep 'cpu cores' /proc/cpuinfo | uniq
> | cpu cores	: 32
> | $ grep 'model name' /proc/cpuinfo | uniq
> | model name	: Intel(R) Xeon(R) Gold 6142 CPU @ 2.60GHz
> | 
> | and you can look up that Xenon 6142 has 16 cores.
> | 
> | Second, instead of "awk"ward contortions it's easily done in R with something like
> | 
> | d=read.dcf("/proc/cpuinfo")
> | sum(as.integer(tapply(
> |   d[,grep("cpu cores",colnames(d))],
> |   d[,grep("physical id",colnames(d))], `[`, 1)))
> | 
> | which avoids subprocesses, quoting hell and all such issues...
> 
> Love the use of read.dcf("/proc/cpuinfo") !!
> 
> On my box a simpler
> 
>> d <- read.dcf("/proc/cpuinfo") 
>> as.integer(unique(d[, grep("cpu cores",colnames(d))]))
>  [1] 6
>> 
> 

I don't think that works on NUMA/SMP machines - you need to add all the cores for each CPU (that's why the above splits by physical id which is unique per cpu). On a dual-cpu machine:

> as.integer(unique(d[, grep("cpu cores",colnames(d))]))
[1] 32
> sum(as.integer(tapply(
 d[,grep("cpu cores",colnames(d))],
 d[,grep("physical id",colnames(d))], `[`, 1)))
[1] 64

Also things get quite fun on VMs as they can cobble together quite a few virtual CPUs regardless of the underlying hardware.

To be honest I think the motivation of this thread is dubious at best: it is a bad idea to use detectCore() blindly to specify parallelization and we explicitly say it's a bad idea - any sensible person will set it according to the demands, the hardware and the task. The number of cores is only partially relevant - e.g. if any I/O is involved you want to oversubscribe the CPU. If you have other users you want to only use a fraction etc. That doesn't mean that the we couldn't do a better job, but if you have to use detectCores() then you are already in trouble to start with.

Cheers,
Simon



[Rd] Detecting physical CPUs in detectCores() on Linux platforms
Simon Urbanek