From jeff.horner at vanderbilt.edu  Thu Jan  4 19:27:09 2007
From: jeff.horner at vanderbilt.edu (Jeffrey Horner)
Date: Thu, 04 Jan 2007 12:27:09 -0600
Subject: [Rd] How to execute R scripts simultaneously from
	multiple	threads
In-Reply-To: <200701041237.30054.vdergachev@rcgardis.com>
References: <459B6DBA.3050207@erik.prutser.cx>	<200701031337.45613.vdergachev@rcgardis.com>	<459CCEDB.5040201@erik.prutser.cx>
	<200701041237.30054.vdergachev@rcgardis.com>
Message-ID: <459D46FD.2060307@vanderbilt.edu>

Vladimir Dergachev wrote:
> On Thursday 04 January 2007 4:54 am, Erik van Zijst wrote:
>> Vladimir Dergachev wrote:
>>> On Wednesday 03 January 2007 3:47 am, Erik van Zijst wrote:
>>>> Appearantly the R C-API does not provide a mechanism for parallel
>>>> execution..
>>>>
>>>> It is preferred that the solution is not based on multi-processing (like
>>>> C/S), because that would introduce IPC overhead.
>>> One thing to keep in mind is that IPC is very fast in Linux. So unless
>>> you are making lots of calls to really tiny functions this should not be
>>> an issue.
>> Using pipes or shared memory to pass things around to other processes on
>> the same box is very fast indeed, but if we base our design around
>> something like RServe which uses TCP it could be significantly slower.
>> Our R-based system will be running scripts in response to high-volume
>> real-time stock exchange data, so we expect lots of calls to many tiny
>> functions indeed.
> 
> Very interesting :) 
> 
> If you are running RServe on the other box you will need to send data over 
> ethernet anyway (and will probably use TCP). If it is on the same box and you 
> use "localhost" the packets will go over loopback - which would be 
> significantly faster.

I haven't looked at RServe in awhile, but I think that it fires up an R 
interpreter in response to a client request and then sticks around for 
the same client to serve it additional requests. The question is how 
does it manage all the R interpreters with varying demand.

This issue is solved when you embed R into Apache (using the prefork 
MPM), as the pool of apache child processes (each with their own R 
interpreter) expands and contracts on demand. Using this with the 
loopback device would be a nice solution:

http://biostat.mc.vanderbilt.edu/RApacheProject

Jeff
-- 
http://biostat.mc.vanderbilt.edu/JeffreyHorner



[Rd] How to execute R scripts simultaneously from multiple threads
Jeffrey Horner