From radford at cs.toronto.edu  Sun Jul 26 20:14:02 2015
From: radford at cs.toronto.edu (Radford Neal)
Date: Sun, 26 Jul 2015 14:14:02 -0400
Subject: [Rd] R-devel Digest, Vol 149, Issue 22
In-Reply-To: <mailman.19.1437818404.17944.r-devel@r-project.org>
References: <mailman.19.1437818404.17944.r-devel@r-project.org>
Message-ID: <20150726181402.GA27570@cs.toronto.edu>

> From: Joshua Bradley <jgbradley1 at gmail.com>
> 
> I have been having issues using parallel::mclapply in a memory-efficient
> way and would like some guidance. I am using a 40 core machine with 96 GB
> of RAM. I've tried to run mclapply with 20, 30, and 40 mc.cores and it has
> practically brought the machine to a standstill each time to the point
> where I do a hard reset.

When mclapply forks to start a new process, the memory is initially
shared with the parent process.  However, a memory page has to be
copied whenever either process writes to it.  Unfortunately, R's
garbage collector writes to each object to mark and unmark it whenever
a full garbage collection is done, so it's quite possible that every R
object will be duplicated in each process, even though many of them
are not actually changed (from the point of view of the R programs).

One thing on my near-term to-do list for pqR is to re-implement R's
garbage collector in a way that will avoid this (as well as having
various other advantages, including less memory overhead per object).

   Radford Neal



[Rd] R-devel Digest, Vol 149, Issue 22
Radford Neal