From maechler at stat.math.ethz.ch  Thu May 12 09:51:54 2016
From: maechler at stat.math.ethz.ch (Martin Maechler)
Date: Thu, 12 May 2016 09:51:54 +0200
Subject: [Rd] R process killed when allocating too large matrix (Mac OS
	X)
In-Reply-To: <5732F090.8020301@ivt.baug.ethz.ch>
References: <CAM3-KjYqc5CvDt4yKkOEPw--1Fq-c8O8x6UfFir5x==-UwFYvw@mail.gmail.com>
	<0e0160ec793649dda463b43d7e7d5c37@CONNHUB1.connect.uwaterloo.ca>
	<CAM3-Kja40deG=04U+PDNjLD=sHzQRiz_umia+wBEP77YyW5qQA@mail.gmail.com>
	<9b3da5c7758b455baad7209384abe7a7@CONNHUB4.connect.uwaterloo.ca>
	<CAM3-KjZ13TOz7x6DC+F1=9WsJFHbz4Ay0gf-fhQOv84oSRiE=w@mail.gmail.com>
	<308D9A3B-C40F-4AF7-AF1A-43DC8E1130EF@r-project.org>
	<572AA6AF.9040400@fredhutch.org>
	<3e8b8d2d46ac440892d2055c502b5be9@CONNHUB1.connect.uwaterloo.ca>
	<CAM3-KjbDejim0PSj9s5ognw4wfA=_BvD2UmfVDdmixqww2MkoA@mail.gmail.com>
	<2fa1c8e2-c692-1237-ea3c-572397038b26@statistik.tu-dortmund.de>
	<572B14D9.3050005@stats.ox.ac.uk>
	<CABFfbXtm5hOW+ah_bjtnP5QproRnGXBtPUsKDK54SXqGfQu=Qg@mail.gmail.com>
	<5732F090.8020301@ivt.baug.ethz.ch>
Message-ID: <22324.13850.608673.341820@stat.math.ethz.ch>

>>>>> Kirill M?ller <kirill.mueller at ivt.baug.ethz.ch>
>>>>>     on Wed, 11 May 2016 10:42:56 +0200 writes:

    > My ulimit package exposes this API ([1], should finally submit it to 
    > CRAN); unfortunately this very API seems to be unsupported on OS X 
    > [2,3]. Last time I looked into it, neither of the documented settings 
    > achieved the desired effect.

    > -Kirill

    > [1] http://krlmlr.github.io/ulimit
    > [2] 
    > http://stackoverflow.com/questions/3274385/how-to-limit-memory-of-a-os-x-program-ulimit-v-neither-m-are-working
    > [3] 
    > https://developer.apple.com/library/ios/documentation/System/Conceptual/ManPages_iPhoneOS/man2/getrlimit.2.html


    > On 10.05.2016 01:08, Jeroen Ooms wrote:
    >> On 05/05/2016 10:11, Uwe Ligges wrote:
    >>> Actually this also happens under Linux and I had my R processes killed
    >>> more than once (and much worse also other processes so that we had to
    >>> reboot a server, essentially).

I agree that Linux is not consistently fine here either.

    >> I found that setting RLIMIT_AS [1] works very well on Linux. But this
    >> requires that you cap memory to some fixed value.

conceivably one could set a default cap, using something
equivalent to the data in

		sfsmisc::Sys.meminfo()
or		sfsmisc::Sys.memGB()		

(very simple, > 10 year old interfaces, based on the  Linux-only (?)
 '/proc/*' filesystem).


In an ideal word, some of us,
    from R core, Jeroen, Kyrill, <one person knowing much about Windows>,
    maintainer("microbenchmark>, ...
would sit together and devise an R function interface (based on
low level platform specific interfaces, specifically for at least
Linux/POSIX-compliant, Mac, and Windows) which would allow
something  like your rlimit(..) calls below.

We'd really need something to work on all platforms ideally,
to be used by R package maintainers
and possibly even better by R itself at startup, setting a
reasonable memory cap - which the user could raise even to +Inf (or lower
even more).

Martin

---------
Final notes about RAppArmor, not relevant to the main thread topic :

Note: I'm working in pretty well maintained Fedora Linux env,
but Apparmor is not only not activated, but even not available.
OTOH, usting  RLIMIT / gerlimit on Linux is very generally
available.

As a consequence, the three last lines of

   > require(RAppArmor)
   Loading required package: RAppArmor
   Loading required package: tools
   Failed to lookup process confinement:
   AppArmor not supported on this system
   Have a look at: sudo aa-status

were very confusing to me:  My conclusion was I could not use
the RAppArmor package.

(But that's wrong: For the  rlimit*()  functions below, one do
 *NOT* need an AppArmor-enabled version of Linux !)
		  
    >>> library(RAppArmor)
    >>> rlimit_as(1e9)
    >>> rnorm(1e9)
    >> Error: cannot allocate vector of size 7.5 Gb
    >> 
    >> The RAppArmor package has many other utilities to protect your server
    >> such from a mis-behaving process such as limiting cpu time
    >> (RLIMIT_CPU), fork bombs (RLIMIT_NPROC) and file sizes (RLIMIT_FSIZE).
    >> 
    >> [1] http://linux.die.net/man/2/getrlimit

and from my current explorations I gather that all of these are
*not* Apparmor related... so could/should maybe rather migrate
into a lightweight package not mentioning AppArmor ?



[Rd] R process killed when allocating too large matrix (Mac OS X)
Martin Maechler