From @|mon@urb@nek @end|ng |rom R-project@org  Sat Apr 13 16:16:22 2019
From: @|mon@urb@nek @end|ng |rom R-project@org (Simon Urbanek)
Date: Sat, 13 Apr 2019 10:16:22 -0400
Subject: [Rd] SUGGESTION: Settings to disable forked processing in R,
 e.g. parallel::mclapply()
In-Reply-To: <CALEXWq2NzM7tLJ4kNTF=D=wKdDFe2xu2nw6mXYbbFJgp9Q1q2g@mail.gmail.com>
References: <CAFDcVCSPzMYE9A9V==Gs2yYTuH+YDqWhT_q=WU4AU5JL0WzX=A@mail.gmail.com>
 <CALEXWq3RapiDCTOFUerzxfiawYBu3LFxpCQpYRwmL3MQBh-S4g@mail.gmail.com>
 <CAPLMX9E6rcPNR0MMmaRPRQOBbhopiC43EBDEaHE56tWFGu5ZTQ@mail.gmail.com>
 <CALEXWq2gkuDW48SmGT+YWjn-GmuEjW56bsXj28xTRO=E-tZ48Q@mail.gmail.com>
 <CAPLMX9Fxe1T+npHdkXAzugApuopXN9F+5HgRSk0MyosRngopsw@mail.gmail.com>
 <CAJXgQP1i0EGEmHj2UAqY-H3QnYfspsJ3BkhY8fGE6oOKCb9vnw@mail.gmail.com>
 <CALEXWq2NzM7tLJ4kNTF=D=wKdDFe2xu2nw6mXYbbFJgp9Q1q2g@mail.gmail.com>
Message-ID: <2DC90380-893E-4525-8406-C685AC6D82FE@R-project.org>

Sure, but that a completely bogus argument because in that case it would fail even more spectacularly with any other method like PSOCK because you would *have to* allocate n times as much memory so unlike mclapply it is guaranteed to fail. With mclapply it is simply much more efficient as it will share memory as long as possible. It is rather obvious that any new objects you create can no longer be shared as they now exist separately in each process.

Cheers,
Simon



> On Apr 13, 2019, at 06:05, I?aki Ucar <iucar at fedoraproject.org> wrote:
> 
> On Sat, 13 Apr 2019 at 03:51, Kevin Ushey <kevinushey at gmail.com> wrote:
>> 
>> I think it's worth saying that mclapply() works as documented
> 
> Mostly, yes. But it says nothing about fork's copy-on-write and memory
> overcommitment, and that this means that it may work nicely or fail
> spectacularly depending on whether, e.g., you operate on a long
> vector.
> 
> -- 
> I?aki ?car
> 
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
> 



[Rd] SUGGESTION: Settings to disable forked processing in R, e.g. parallel::mclapply()
Simon Urbanek