From simon.urbanek at r-project.org  Fri Apr 20 15:51:30 2012
From: simon.urbanek at r-project.org (Simon Urbanek)
Date: Fri, 20 Apr 2012 09:51:30 -0400
Subject: [Rd] C - R integration: Memory Issues
In-Reply-To: <5f2eed30c28c417068a65dddf6388771@webmail.duth.gr>
References: <4eae997ce7a91027c9a116e74bfe0e62@webmail.duth.gr>
	<20368.824.429418.688401@max.nulle.part>
	<5f2eed30c28c417068a65dddf6388771@webmail.duth.gr>
Message-ID: <92217B5B-A391-487F-BBFF-D94EC59B54FB@r-project.org>


On Apr 20, 2012, at 5:06 AM, Nikolaos Bezirgiannidis wrote:

> On Thu, 19 Apr 2012 07:21:12 -0500, Dirk Eddelbuettel wrote:
>> On 18 April 2012 at 17:40, Nikolaos Bezirgiannidis wrote:
>> |  Hi all,
>> |
>> |  I am a PhD student and I am working on a C project that involves some
>> |  statistical calculations. So, I tried to embed R into C, in order to
>> |  call R functions from a C program. My program seems to get the correct
>> |  results from R. However, it appears to have a lot of memory allocation
>> |  issues, in contrast to the small amounts of memory that my code
>> |  allocates. Some additional info that might be useful: I have build R
>> |  from source with shared libraries enabled and the compiler I use is gcc
>> |  version 4.61 in a Ubuntu 11.10 linux machine.
>> 
>> [ Well I suspect "sudo apt-get install r-base" would have given you the same;
>>  see the README at $CRAN/src/bin/linux/ubuntu ]
>> 
>> |  This is my function:
>> |
>> |  static int prediction(double *berHistory, int berValues, double *ber)
>> |  {
>> |       SEXP e;
>> |       SEXP bers;
>> |       SEXP mean;
>> |       int     i;
>> |       int     errorOccurred;
>> |       static int init = 0;
>> |       char *argv[] = {"REmbeddedPostgres", "--gui=none", "--silent"};
>> |       int argc = sizeof(argv)/sizeof(argv[0]);
>> |
>> |       // Initialize Embedded R
>> |       if (init == 0)
>> |       {
>> |            Rf_initEmbeddedR(argc, argv);
>> |       }
>> |       init = 1;
>> |
>> |       // Allocate bers and copy values
>> |       PROTECT(bers = allocVector(REALSXP, berValues));
>> |
>> |       for (i = 0; i < berValues; i++)
>> |       {
>> |            REAL(bers)[i] = berHistory[i];
>> |       }
>> |
>> |       PROTECT(mean = allocVector(REALSXP, 1));
>> |       PROTECT(e = lang2(install("mean"), bers));
>> |       mean = R_tryEval(e, R_GlobalEnv, &errorOccurred);
>> |       if (errorOccurred)
>> |       {
>> |            printf("error occurred in mean\n");
>> |       }
>> |
>> |       for (i = 0; i < berValues; i++)
>> |       {
>> |            REAL(bers)[i] = REAL(bers)[i] / REAL(mean)[0];
>> |       }
>> |
>> |       *ber = REAL(mean)[0];
>> |
>> |       Rf_endEmbeddedR(0);
>> |       UNPROTECT(3);
>> |       return 0;
>> |  }
>> |
>> |  And these are the errors from Valgrind output:
>> |
>> |  HEAP SUMMARY:
>> |  ==2909==     in use at exit: 18,832,260 bytes in 6,791 blocks
>> |  ==2909==   total heap usage: 21,758 allocs, 14,967 frees, 30,803,476
>> |  bytes allocated
>> |  ==2909==
>> |  ==2909== 160 (40 direct, 120 indirect) bytes in 1 blocks are definitely
>> |  lost in loss record 179 of 1,398
>> |  ==2909==    at 0x4028876: malloc (vg_replace_malloc.c:236)
>> |  ==2909==    by 0x41B364C: nss_parse_service_list (nsswitch.c:626)
>> |  ==2909==    by 0x41B3C59: __nss_database_lookup (nsswitch.c:167)
>> |  ==2909==    by 0x59272F8: ???
>> |  ==2909==    by 0x5928CCC: ???
>> |  ==2909==    by 0x416ABA6: getpwuid_r@@GLIBC_2.1.2 (getXXbyYY_r.c:256)
>> |  ==2909==    by 0x416A4ED: getpwuid (getXXbyYY.c:117)
>> |  ==2909==    by 0x439CCB9: do_fileinfo (platform.c:944)
>> |  ==2909==    by 0x43289ED: bcEval (eval.c:4430)
>> |  ==2909==    by 0x4332CA4: Rf_eval (eval.c:397)
>> |  ==2909==    by 0x43377E0: Rf_applyClosure (eval.c:855)
>> |  ==2909==    by 0x432F17E: bcEval (eval.c:4410)
>> |  ==2909==
>> |  ==2909== 160 (40 direct, 120 indirect) bytes in 1 blocks are definitely
>> |  lost in loss record 180 of 1,398
>> |  ==2909==    at 0x4028876: malloc (vg_replace_malloc.c:236)
>> |  ==2909==    by 0x41B364C: nss_parse_service_list (nsswitch.c:626)
>> |  ==2909==    by 0x41B3C59: __nss_database_lookup (nsswitch.c:167)
>> |  ==2909==    by 0x5926148: ???
>> |  ==2909==    by 0x5926F3C: ???
>> |  ==2909==    by 0x41694A6: getgrgid_r@@GLIBC_2.1.2 (getXXbyYY_r.c:256)
>> |  ==2909==    by 0x4168CAD: getgrgid (getXXbyYY.c:117)
>> |  ==2909==    by 0x439CCEB: do_fileinfo (platform.c:947)
>> |  ==2909==    by 0x43289ED: bcEval (eval.c:4430)
>> |  ==2909==    by 0x4332CA4: Rf_eval (eval.c:397)
>> |  ==2909==    by 0x43377E0: Rf_applyClosure (eval.c:855)
>> |  ==2909==    by 0x432F17E: bcEval (eval.c:4410)
>> |  ==2909==
>> |  ==2909== LEAK SUMMARY:
>> |  ==2909==    definitely lost: 80 bytes in 2 blocks
>> |  ==2909==    indirectly lost: 240 bytes in 20 blocks
>> |  ==2909==      possibly lost: 0 bytes in 0 blocks
>> |  ==2909==    still reachable: 18,831,940 bytes in 6,769 blocks
>> 
>> Well I think there is no real bad error here. You lost 80 and 240 bytes. That
>> is nothing to worry about, and somewhat normal. R is a dynamic system,
>> valgrind measures "with some error".
>> 
>> You can compare the result to doing an allocation of a longer vector and not
>> freeing it.
>> 
>> You could see if freeing your 'mean' variable at the end makes a difference,
>> or using a pointer to a single (stack) instance instead of an allocation
>> makes a difference.  Likewise, you could returns bers as well. Or free
>> it. Right now I am not entirely what it is that your 'prediction' function is
>> trying to do.
> 
> It is actually a more complex function with many more allocations and R_tryEval calls; the simplified one that I included leads to exactly the same lost bytes, which means that it is not caused by allocVector, but probably by the  R_tryEval call. Is there any other, more proper way to evaluate R expressions from C?
> 
> In my function I only need to return ber, so I could easily free all R objects (bers and mean here). But C's free() has no result and I am not familiar with any R function that frees objects allocated by allocVector. I have also seen in some R tutorials and examples that objects allocated by allocVector are not freed in any way.
> 

R uses a garbage collector, so I think you're chasing a ghost here... 

But if you look at the output of lost blocks this is simply storage allocated by your glibc for non-reentrant calls so it has nothing to do with R.

Cheers,
Simon


>> 
>> |  Reachable error summary is far too long to include in this mail. The
>> |  interesting thing is that reachable errors are all caused by this small
>> |  function.
>> |
>> |  Any ideas? I would also appreciate any suggestions on how to improve
>> |  the R-C integration in my code.
>> 
>> Have you considered using R and C++ instead, and looked at Rcpp and
>> RInside?
>> 
>>     http://dirk.eddelbuettel.com/code/rcpp.html
>> 
>>     http://dirk.eddelbuettel.com/code/rinside.html
>> 
>> RInside in particular is a lot simpler, at least to me. But some people
>> really want plain C in which case you know which route to take
>> 
>> Dirk
> 
> No, unfortunately I cannot switch to C++, because my program is a part of a bigger C project.
> 
> Thanks for your reply and suggestions,
> Nikos
> 
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
> 
> 



[Rd] C - R integration: Memory Issues
Simon Urbanek