From ernest.turro at ic.ac.uk  Tue Feb 27 19:01:33 2007
From: ernest.turro at ic.ac.uk (Ernest Turro)
Date: Tue, 27 Feb 2007 18:01:33 +0000
Subject: [Rd] R/C++/memory leaks
In-Reply-To: <Pine.LNX.4.64.0702271137200.2965@nokomis.stat.uiowa.edu>
References: <39CFDB8F-918C-48FB-9A6D-D8055D828C13@ic.ac.uk>
	<20070225222144.GA16188@wheat.betterworld.us>
	<F3C922F3-4B4C-4DA9-84E1-F17B52302BCB@ic.ac.uk>
	<20070226064336.GL5871@wheat.betterworld.us>
	<A0FD96B2-2FCD-416E-9496-6D6192511357@ic.ac.uk>
	<1172529270.9347.164.camel@iron.psg.net>
	<B62E11C5-9122-495A-9032-37610A7288B4@ic.ac.uk>
	<Pine.LNX.4.64.0702271137200.2965@nokomis.stat.uiowa.edu>
Message-ID: <041F0BCD-653B-4309-85FA-82334DA2B50B@ic.ac.uk>


On 27 Feb 2007, at 17:45, Luke Tierney wrote:

> On Tue, 27 Feb 2007, Ernest Turro wrote:
>
>> Hi Ross,
>>
>> On 26 Feb 2007, at 22:34, Ross Boylan wrote:
>>
>>> On Mon, 2007-02-26 at 16:08 +0000, Ernest Turro wrote:
>>>> Thanks for your comments Ross. A couple more comments/queries  
>>>> below:
>>>>
>>>> On 26 Feb 2007, at 06:43, Ross Boylan wrote:
>>>>
>>>>> [details snipped]
>>>>>
>>>>> The use of the R api can be confined to a wrapper function.  But
>>>>> I can
>>>>> think of no reason that a change to the alternate approach I
>>>>> outlined
>>>>> would solve the apparent leaking you describe.
>>>>>
>>>>
>>>> I'm not sure I see how a wrapper function using the R API would
>>>> suffice. Example:
>>> It doesn't sound as if it would suffice.  I was responding to your
>>> original remark that
>>>
>>>> Since this is a standalone C++ program too, I'd rather use the R  
>>>> API
>>>> as little as possible... But I will look at your solution if I find
>>>> it is really necessary.. Thanks
>>>
>>> I thought that was expressing a concern about using the alternate
>>> approach I outlined because it would use the R API.  If you need to
>>> use
>>> that API for other reasons, you're still stuck with it :)
>>>>
>>>> During heavy computation in the C++ function I need to allow
>>>> interrupts from R. This means that R_CheckUserInterrupt needs to be
>>>> called during the computation. Therefore, use of the R API can't be
>>>> confined to just the wrapper function.
>>>>
>>>> In fact, I'm worried that some of the libraries I'm using are  
>>>> failing
>>>> to release memory after interrupt and that that is the problem. I
>>>> can't see what I could do about that... E.g.
>>>>
>>>> #include <valarray>
>>>>
>>>> valarray<double> foo; // I don't know 100% that the foo object  
>>>> hasn't
>>>> allocated some memory. if the program is interrupted it wouldn't be
>>>> released....
>>> That's certainly possible, but you seem to be overlooking the
>>> possibility that all the code is releasing memory appropriately,
>>> but the
>>> process's memory footprint isn't going down correspondingly.  In my
>>> experience that's fairly typical behavior.
>>>
>>
>> OK, but does this still explain why the footprint keeps increasing
>> indefinitely when i do run, interrupt, run, interrupt, run,
>> interrupt......?
>>
>>
>>> In that case, depending on your point of view, you either don't  
>>> have a
>>> problem or you have a hard problem.  If you really want the memory
>>> released back to the system, it's a hard problem.  If you don't
>>> care, as
>>> long as you have no leaks, all's well.
>>>
>>>>
>>>> I find it's very unfortunate that R_CheckUserInterrupt doesn't  
>>>> return
>>>> a value. If it did (e.g. if it returned true if an interrupt has
>>>> occurred), I could just branch off somewhere, clean up properly and
>>>> return to R.
>>>>
>>>> Any ideas on how this could be achieved?
>>> I can't tell from the info page what function gets called in R if
>>> there
>>> is an interrupt, but it sounds as you could do the following hack:
>>> The R interrupt handler gets a function that calls a C function of
>>> your
>>> devising.  The C function sets a flag meaning "interrupt requested".
>>> Then in your main code, you periodically call R_CheckUserInterrupt.
>>> When it returns you check the flag; if it's set, you cleanup and  
>>> exit.
>>> Ross
>>>
>>
>> If this is feasible, it's by far the best solution.
>>
>> in error.c:
>>
>> void R_CheckUserInterrupt(void)
>> {
>>     R_CheckStack();
>>     /* This is the point where GUI systems need to do enough event
>>        processing to determine whether there is a user interrupt  
>> event
>>        pending.  Need to be careful not to do too much event
>>        processing though: if event handlers written in R are allowed
>>        to run at this point then we end up with concurrent R
>>        evaluations and that can cause problems until we have proper
>>        concurrency support. LT */
>> #if  ( defined(HAVE_AQUA) || defined(Win32) )
>>     R_ProcessEvents();
>> #else
>>     if (R_interrupts_pending)
>>   onintr();
>> #endif /* Win32 */
>> }
>>
>> Leaving aside the HAVE_AQUA and Win32 cases, I would like to write a
>> new function:
>
> Unfortunately we can't leave those aside.  If standard unix where
> interrupts arrive as signals is all you care about then you can just
> save, replace and restore the R SIGINT handler around your code with
> one that sets a flag of your own.  Things are not so simple on GUI
> systems where detecting a user interrupt action requires event
> processing, which might result in errors and non-local exits in
> response to those.
>
> There is an internal mechanism for registering C level on.exit
> routines but this is not in a form that can be made public as it would
> tie down implementation decisions too much.  It is principle possible
> to build something around R_ToplevelExec, but that is not at this
> point part of the public API and so is subject to change.  We might
> consider providing a variant of R_CheckInterrupts that either just
> checks or that executes cleanup code sometime after 2.5 is released.
>

Yes, I think it would be great if you added a variant of  
R_CheckUserInterrupt() that actually returned something (e.g. a  
boolean) to the R API.

So in your view, is the best solution to pass a memory cleanup C  
function to on.exit() before the call? What about memory allocated in  
some of the external libraries that I use? E.g. as I mentioned above:

>>>>
>>>> #include <valarray>
>>>> // ...
>>>> valarray<double> foo; // I don't know 100% that the foo object  
>>>> hasn't
>>>> allocated some memory. if the program is interrupted it wouldn't be
>>>> released....


Thanks for your help,

Ernest


> Best,
>
> luke
>
>>
>> int R_CheckInterruptsPending(void)
>> {
>>     R_CheckStack();
>>     return R_interrupts_pending;
>> }
>>
>> and then in my C++ code:
>>
>> if(R_checkInterruptsPending) {
>> 	// clean up
>> 	// ...
>> 	R_CheckInterruptsPending();
>> }
>>
>> R_CheckStack() is declared in R_ext/Utils.h but the variable
>> R_interrupts_pending isn't, so how could I access it? In other words,
>> how can I extend error.c .....
>>
>>
>> Thanks,
>>
>> E
>>
>> ______________________________________________
>> R-devel at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-devel
>>
>
> -- 
> Luke Tierney
> Chair, Statistics and Actuarial Science
> Ralph E. Wareham Professor of Mathematical Sciences
> University of Iowa                  Phone:             319-335-3386
> Department of Statistics and        Fax:               319-335-3017
>    Actuarial Science
> 241 Schaeffer Hall                  email:      luke at stat.uiowa.edu
> Iowa City, IA 52242                 WWW:  http://www.stat.uiowa.edu



[Rd] R/C++/memory leaks
Ernest Turro