From luke at stat.uiowa.edu  Tue Feb 27 19:32:18 2007
From: luke at stat.uiowa.edu (Luke Tierney)
Date: Tue, 27 Feb 2007 12:32:18 -0600 (CST)
Subject: [Rd] R/C++/memory leaks
In-Reply-To: <041F0BCD-653B-4309-85FA-82334DA2B50B@ic.ac.uk>
References: <39CFDB8F-918C-48FB-9A6D-D8055D828C13@ic.ac.uk>
	<20070225222144.GA16188@wheat.betterworld.us>
	<F3C922F3-4B4C-4DA9-84E1-F17B52302BCB@ic.ac.uk>
	<20070226064336.GL5871@wheat.betterworld.us>
	<A0FD96B2-2FCD-416E-9496-6D6192511357@ic.ac.uk>
	<1172529270.9347.164.camel@iron.psg.net>
	<B62E11C5-9122-495A-9032-37610A7288B4@ic.ac.uk>
	<Pine.LNX.4.64.0702271137200.2965@nokomis.stat.uiowa.edu>
	<041F0BCD-653B-4309-85FA-82334DA2B50B@ic.ac.uk>
Message-ID: <Pine.LNX.4.64.0702271230430.2965@nokomis.stat.uiowa.edu>

On Tue, 27 Feb 2007, Ernest Turro wrote:

>
> On 27 Feb 2007, at 17:45, Luke Tierney wrote:
>
>> On Tue, 27 Feb 2007, Ernest Turro wrote:
>> 
>>> Hi Ross,
>>> 
>>> On 26 Feb 2007, at 22:34, Ross Boylan wrote:
>>> 
>>>> On Mon, 2007-02-26 at 16:08 +0000, Ernest Turro wrote:
>>>>> Thanks for your comments Ross. A couple more comments/queries below:
>>>>> 
>>>>> On 26 Feb 2007, at 06:43, Ross Boylan wrote:
>>>>> 
>>>>>> [details snipped]
>>>>>> 
>>>>>> The use of the R api can be confined to a wrapper function.  But
>>>>>> I can
>>>>>> think of no reason that a change to the alternate approach I
>>>>>> outlined
>>>>>> would solve the apparent leaking you describe.
>>>>>> 
>>>>> 
>>>>> I'm not sure I see how a wrapper function using the R API would
>>>>> suffice. Example:
>>>> It doesn't sound as if it would suffice.  I was responding to your
>>>> original remark that
>>>> 
>>>>> Since this is a standalone C++ program too, I'd rather use the R API
>>>>> as little as possible... But I will look at your solution if I find
>>>>> it is really necessary.. Thanks
>>>> 
>>>> I thought that was expressing a concern about using the alternate
>>>> approach I outlined because it would use the R API.  If you need to
>>>> use
>>>> that API for other reasons, you're still stuck with it :)
>>>>> 
>>>>> During heavy computation in the C++ function I need to allow
>>>>> interrupts from R. This means that R_CheckUserInterrupt needs to be
>>>>> called during the computation. Therefore, use of the R API can't be
>>>>> confined to just the wrapper function.
>>>>> 
>>>>> In fact, I'm worried that some of the libraries I'm using are failing
>>>>> to release memory after interrupt and that that is the problem. I
>>>>> can't see what I could do about that... E.g.
>>>>> 
>>>>> #include <valarray>
>>>>> 
>>>>> valarray<double> foo; // I don't know 100% that the foo object hasn't
>>>>> allocated some memory. if the program is interrupted it wouldn't be
>>>>> released....
>>>> That's certainly possible, but you seem to be overlooking the
>>>> possibility that all the code is releasing memory appropriately,
>>>> but the
>>>> process's memory footprint isn't going down correspondingly.  In my
>>>> experience that's fairly typical behavior.
>>>> 
>>> 
>>> OK, but does this still explain why the footprint keeps increasing
>>> indefinitely when i do run, interrupt, run, interrupt, run,
>>> interrupt......?
>>> 
>>> 
>>>> In that case, depending on your point of view, you either don't have a
>>>> problem or you have a hard problem.  If you really want the memory
>>>> released back to the system, it's a hard problem.  If you don't
>>>> care, as
>>>> long as you have no leaks, all's well.
>>>> 
>>>>> 
>>>>> I find it's very unfortunate that R_CheckUserInterrupt doesn't return
>>>>> a value. If it did (e.g. if it returned true if an interrupt has
>>>>> occurred), I could just branch off somewhere, clean up properly and
>>>>> return to R.
>>>>> 
>>>>> Any ideas on how this could be achieved?
>>>> I can't tell from the info page what function gets called in R if
>>>> there
>>>> is an interrupt, but it sounds as you could do the following hack:
>>>> The R interrupt handler gets a function that calls a C function of
>>>> your
>>>> devising.  The C function sets a flag meaning "interrupt requested".
>>>> Then in your main code, you periodically call R_CheckUserInterrupt.
>>>> When it returns you check the flag; if it's set, you cleanup and exit.
>>>> Ross
>>>> 
>>> 
>>> If this is feasible, it's by far the best solution.
>>> 
>>> in error.c:
>>> 
>>> void R_CheckUserInterrupt(void)
>>> {
>>>    R_CheckStack();
>>>    /* This is the point where GUI systems need to do enough event
>>>       processing to determine whether there is a user interrupt event
>>>       pending.  Need to be careful not to do too much event
>>>       processing though: if event handlers written in R are allowed
>>>       to run at this point then we end up with concurrent R
>>>       evaluations and that can cause problems until we have proper
>>>       concurrency support. LT */
>>> #if  ( defined(HAVE_AQUA) || defined(Win32) )
>>>    R_ProcessEvents();
>>> #else
>>>    if (R_interrupts_pending)
>>>  onintr();
>>> #endif /* Win32 */
>>> }
>>> 
>>> Leaving aside the HAVE_AQUA and Win32 cases, I would like to write a
>>> new function:
>> 
>> Unfortunately we can't leave those aside.  If standard unix where
>> interrupts arrive as signals is all you care about then you can just
>> save, replace and restore the R SIGINT handler around your code with
>> one that sets a flag of your own.  Things are not so simple on GUI
>> systems where detecting a user interrupt action requires event
>> processing, which might result in errors and non-local exits in
>> response to those.
>> 
>> There is an internal mechanism for registering C level on.exit
>> routines but this is not in a form that can be made public as it would
>> tie down implementation decisions too much.  It is principle possible
>> to build something around R_ToplevelExec, but that is not at this
>> point part of the public API and so is subject to change.  We might
>> consider providing a variant of R_CheckInterrupts that either just
>> checks or that executes cleanup code sometime after 2.5 is released.
>> 
>
> Yes, I think it would be great if you added a variant of 
> R_CheckUserInterrupt() that actually returned something (e.g. a boolean) to 
> the R API.
>
> So in your view, is the best solution to pass a memory cleanup C function to 
> on.exit() before the call? What about memory allocated in some of the 
> external libraries that I use? E.g. as I mentioned above:

Depends on the cleanup actions you have to take.

Best,

luke

>
>>>>> 
>>>>> #include <valarray>
>>>>> // ...
>>>>> valarray<double> foo; // I don't know 100% that the foo object hasn't
>>>>> allocated some memory. if the program is interrupted it wouldn't be
>>>>> released....
>
>
> Thanks for your help,
>
> Ernest
>
>
>> Best,
>> 
>> luke
>> 
>>> 
>>> int R_CheckInterruptsPending(void)
>>> {
>>>    R_CheckStack();
>>>    return R_interrupts_pending;
>>> }
>>> 
>>> and then in my C++ code:
>>> 
>>> if(R_checkInterruptsPending) {
>>> 	// clean up
>>> 	// ...
>>> 	R_CheckInterruptsPending();
>>> }
>>> 
>>> R_CheckStack() is declared in R_ext/Utils.h but the variable
>>> R_interrupts_pending isn't, so how could I access it? In other words,
>>> how can I extend error.c .....
>>> 
>>> 
>>> Thanks,
>>> 
>>> E
>>> 
>>> ______________________________________________
>>> R-devel at r-project.org mailing list
>>> https://stat.ethz.ch/mailman/listinfo/r-devel
>>> 
>> 
>> -- 
>> Luke Tierney
>> Chair, Statistics and Actuarial Science
>> Ralph E. Wareham Professor of Mathematical Sciences
>> University of Iowa                  Phone:             319-335-3386
>> Department of Statistics and        Fax:               319-335-3017
>>   Actuarial Science
>> 241 Schaeffer Hall                  email:      luke at stat.uiowa.edu
>> Iowa City, IA 52242                 WWW:  http://www.stat.uiowa.edu
>

-- 
Luke Tierney
Chair, Statistics and Actuarial Science
Ralph E. Wareham Professor of Mathematical Sciences
University of Iowa                  Phone:             319-335-3386
Department of Statistics and        Fax:               319-335-3017
    Actuarial Science
241 Schaeffer Hall                  email:      luke at stat.uiowa.edu
Iowa City, IA 52242                 WWW:  http://www.stat.uiowa.edu



[Rd] R/C++/memory leaks
Luke Tierney