From @|mon@urb@nek @end|ng |rom R-project@org  Sun Apr 14 01:08:01 2019
From: @|mon@urb@nek @end|ng |rom R-project@org (Simon Urbanek)
Date: Sat, 13 Apr 2019 19:08:01 -0400
Subject: [Rd] SUGGESTION: Settings to disable forked processing in R,
 e.g. parallel::mclapply()
In-Reply-To: <CALEXWq0wHU5fs2CkswbfmneJ4QSMkZRtn6m5KOu3aWcbMertsQ@mail.gmail.com>
References: <CAFDcVCSPzMYE9A9V==Gs2yYTuH+YDqWhT_q=WU4AU5JL0WzX=A@mail.gmail.com>
 <CALEXWq3RapiDCTOFUerzxfiawYBu3LFxpCQpYRwmL3MQBh-S4g@mail.gmail.com>
 <CAPLMX9E6rcPNR0MMmaRPRQOBbhopiC43EBDEaHE56tWFGu5ZTQ@mail.gmail.com>
 <CALEXWq2gkuDW48SmGT+YWjn-GmuEjW56bsXj28xTRO=E-tZ48Q@mail.gmail.com>
 <CAPLMX9Fxe1T+npHdkXAzugApuopXN9F+5HgRSk0MyosRngopsw@mail.gmail.com>
 <CAJXgQP1i0EGEmHj2UAqY-H3QnYfspsJ3BkhY8fGE6oOKCb9vnw@mail.gmail.com>
 <CALEXWq2NzM7tLJ4kNTF=D=wKdDFe2xu2nw6mXYbbFJgp9Q1q2g@mail.gmail.com>
 <2DC90380-893E-4525-8406-C685AC6D82FE@R-project.org>
 <CALEXWq0wHU5fs2CkswbfmneJ4QSMkZRtn6m5KOu3aWcbMertsQ@mail.gmail.com>
Message-ID: <D94BADA2-3199-4E25-8363-C200FF4B9EBD@R-project.org>



> On Apr 13, 2019, at 16:56, I?aki Ucar <iucar at fedoraproject.org> wrote:
> 
> On Sat, 13 Apr 2019 at 18:41, Simon Urbanek <simon.urbanek at r-project.org> wrote:
>> 
>> Sure, but that a completely bogus argument because in that case it would fail even more spectacularly with any other method like PSOCK because you would *have to* allocate n times as much memory so unlike mclapply it is guaranteed to fail. With mclapply it is simply much more efficient as it will share memory as long as possible. It is rather obvious that any new objects you create can no longer be shared as they now exist separately in each process.
> 
> The point was that PSOCK fails and succeeds *consistently*,
> independently of what you do with the input in the function provided.
> I think that's a good property.
> 

So does parallel. It is consistent. If you do things that use too much memory you will consistently fail. That's a pretty universal rule, there is nothing probabilistic about it. It makes no difference if it's PSOCK, multicore, or anything else.



[Rd] SUGGESTION: Settings to disable forked processing in R, e.g. parallel::mclapply()
Simon Urbanek