From christian.krause at idiv.de  Thu Nov 24 08:30:52 2016
From: christian.krause at idiv.de (Christian Krause)
Date: Thu, 24 Nov 2016 08:30:52 +0100
Subject: [Rd] [parallel-package] feature request: set default cluster type
 via environment variable
Message-ID: <3458b75c-fbb7-f9c1-b05a-f06c399f4443@idiv.de>

Dear all,

I?m working as an administrator of a High-Performance Computing (HPC) Cluster which runs on Linux. A lot of people are using R on this Linux cluster and, of course, the *parallel* package to speed up their computations.

It has been our collective experience, that using |makeForkCluster| yields an overall better experience /on Linux/ than the |makePSOCKcluster|, for whatever definition of better. Let me just summarize that it works smoother. I believe, other people working with *parallel* on Linux can share this experience

Also, we did really welcome the environment variable |MC_CORES|, to be able to specify (in job submit scripts) the amount of CPU cores a user has been granted, most importantly for /dynamic resource requests/ (see below for an example).

What we would also appreciate - and now we finally get to the feature request - is another environment variable to choose the used cluster, as in:

|export MC_CLUSTER_TYPE=FORK |

Do you think something like this could be implemented in future releases?


      Parallel R job submit script

This works with the Univa Grid Engine and should work with other * Grid Engine products:

|#!/bin/bash # request a "parallel environment" with 2 to 20 cores #$ -pe smp 2-20 # set number of cores for the R cluster to the granted value (between 2 and 20) export MC_CORES=$NSLOTS # we want this: export MC_CLUSTER_TYPE=FORK Rscript /path/to/script.R |

Best Regards

?
-- 

Christian Krause

Scientific Computing Administration and Support

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Phone: +49 341 97 33144

Email: christian.krause at idiv.de

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

German Centre for Integrative Biodiversity Research (iDiv) Halle-Jena-Leipzig

Deutscher Platz 5e

04103 Leipzig

Germany

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

iDiv is a research centre of the DFG ? Deutsche Forschungsgemeinschaft

iDiv ist eine zentrale Einrichtung der Universit?t Leipzig im Sinne des ? 92 Abs. 1 S?chsHSFG und wird zusammen mit der Martin-Luther-Universit?t Halle-Wittenberg und der Friedrich-Schiller-Universit?t Jena betrieben sowie in Kooperation mit dem Helmholtz-Zentrum f?r Umweltforschung GmbH ? UFZ. Beteiligte Kooperationspartner sind die folgenden au?eruniversit?ren Forschungseinrichtungen: das Helmholtz-Zentrum f?r Umweltforschung GmbH - UFZ, das Max-Planck-Institut f?r Biogeochemie (MPI BGC), das Max-Planck-Institut f?r chemische ?kologie (MPI CE), das Max-Planck-Institut f?r evolution?re Anthropologie (MPI EVA), das Leibniz-Institut Deutsche Sammlung von Mikroorganismen und Zellkulturen (DSMZ), das Leibniz-Institut f?r Pflanzenbiochemie (IPB), das Leibniz-Institut f?r Pflanzengenetik und Kulturpflanzenforschung (IPK) und das Leibniz-Institut Senckenberg Museum f?r Naturkunde G?rlitz (SMNG). USt-IdNr. DE 141510383


	[[alternative HTML version deleted]]



[Rd] [parallel-package] feature request: set default cluster type via environment variable
Christian Krause