From nbezirgi at ee.duth.gr  Fri Apr 20 11:06:52 2012
From: nbezirgi at ee.duth.gr (Nikolaos Bezirgiannidis)
Date: Fri, 20 Apr 2012 09:06:52 +0000
Subject: [Rd] C - R integration: Memory Issues
In-Reply-To: <20368.824.429418.688401@max.nulle.part>
References: <4eae997ce7a91027c9a116e74bfe0e62@webmail.duth.gr>
	<20368.824.429418.688401@max.nulle.part>
Message-ID: <5f2eed30c28c417068a65dddf6388771@webmail.duth.gr>

 On Thu, 19 Apr 2012 07:21:12 -0500, Dirk Eddelbuettel wrote:
> On 18 April 2012 at 17:40, Nikolaos Bezirgiannidis wrote:
> |  Hi all,
> |
> |  I am a PhD student and I am working on a C project that involves 
> some
> |  statistical calculations. So, I tried to embed R into C, in order 
> to
> |  call R functions from a C program. My program seems to get the 
> correct
> |  results from R. However, it appears to have a lot of memory 
> allocation
> |  issues, in contrast to the small amounts of memory that my code
> |  allocates. Some additional info that might be useful: I have build 
> R
> |  from source with shared libraries enabled and the compiler I use 
> is gcc
> |  version 4.61 in a Ubuntu 11.10 linux machine.
>
> [ Well I suspect "sudo apt-get install r-base" would have given you 
> the same;
>   see the README at $CRAN/src/bin/linux/ubuntu ]
>
> |  This is my function:
> |
> |  static int prediction(double *berHistory, int berValues, double 
> *ber)
> |  {
> |       SEXP e;
> |       SEXP bers;
> |       SEXP mean;
> |       int     i;
> |       int     errorOccurred;
> |       static int init = 0;
> |       char *argv[] = {"REmbeddedPostgres", "--gui=none", 
> "--silent"};
> |       int argc = sizeof(argv)/sizeof(argv[0]);
> |
> |       // Initialize Embedded R
> |       if (init == 0)
> |       {
> |            Rf_initEmbeddedR(argc, argv);
> |       }
> |       init = 1;
> |
> |       // Allocate bers and copy values
> |       PROTECT(bers = allocVector(REALSXP, berValues));
> |
> |       for (i = 0; i < berValues; i++)
> |       {
> |            REAL(bers)[i] = berHistory[i];
> |       }
> |
> |       PROTECT(mean = allocVector(REALSXP, 1));
> |       PROTECT(e = lang2(install("mean"), bers));
> |       mean = R_tryEval(e, R_GlobalEnv, &errorOccurred);
> |       if (errorOccurred)
> |       {
> |            printf("error occurred in mean\n");
> |       }
> |
> |       for (i = 0; i < berValues; i++)
> |       {
> |            REAL(bers)[i] = REAL(bers)[i] / REAL(mean)[0];
> |       }
> |
> |       *ber = REAL(mean)[0];
> |
> |       Rf_endEmbeddedR(0);
> |       UNPROTECT(3);
> |       return 0;
> |  }
> |
> |  And these are the errors from Valgrind output:
> |
> |  HEAP SUMMARY:
> |  ==2909==     in use at exit: 18,832,260 bytes in 6,791 blocks
> |  ==2909==   total heap usage: 21,758 allocs, 14,967 frees, 
> 30,803,476
> |  bytes allocated
> |  ==2909==
> |  ==2909== 160 (40 direct, 120 indirect) bytes in 1 blocks are 
> definitely
> |  lost in loss record 179 of 1,398
> |  ==2909==    at 0x4028876: malloc (vg_replace_malloc.c:236)
> |  ==2909==    by 0x41B364C: nss_parse_service_list (nsswitch.c:626)
> |  ==2909==    by 0x41B3C59: __nss_database_lookup (nsswitch.c:167)
> |  ==2909==    by 0x59272F8: ???
> |  ==2909==    by 0x5928CCC: ???
> |  ==2909==    by 0x416ABA6: getpwuid_r@@GLIBC_2.1.2 
> (getXXbyYY_r.c:256)
> |  ==2909==    by 0x416A4ED: getpwuid (getXXbyYY.c:117)
> |  ==2909==    by 0x439CCB9: do_fileinfo (platform.c:944)
> |  ==2909==    by 0x43289ED: bcEval (eval.c:4430)
> |  ==2909==    by 0x4332CA4: Rf_eval (eval.c:397)
> |  ==2909==    by 0x43377E0: Rf_applyClosure (eval.c:855)
> |  ==2909==    by 0x432F17E: bcEval (eval.c:4410)
> |  ==2909==
> |  ==2909== 160 (40 direct, 120 indirect) bytes in 1 blocks are 
> definitely
> |  lost in loss record 180 of 1,398
> |  ==2909==    at 0x4028876: malloc (vg_replace_malloc.c:236)
> |  ==2909==    by 0x41B364C: nss_parse_service_list (nsswitch.c:626)
> |  ==2909==    by 0x41B3C59: __nss_database_lookup (nsswitch.c:167)
> |  ==2909==    by 0x5926148: ???
> |  ==2909==    by 0x5926F3C: ???
> |  ==2909==    by 0x41694A6: getgrgid_r@@GLIBC_2.1.2 
> (getXXbyYY_r.c:256)
> |  ==2909==    by 0x4168CAD: getgrgid (getXXbyYY.c:117)
> |  ==2909==    by 0x439CCEB: do_fileinfo (platform.c:947)
> |  ==2909==    by 0x43289ED: bcEval (eval.c:4430)
> |  ==2909==    by 0x4332CA4: Rf_eval (eval.c:397)
> |  ==2909==    by 0x43377E0: Rf_applyClosure (eval.c:855)
> |  ==2909==    by 0x432F17E: bcEval (eval.c:4410)
> |  ==2909==
> |  ==2909== LEAK SUMMARY:
> |  ==2909==    definitely lost: 80 bytes in 2 blocks
> |  ==2909==    indirectly lost: 240 bytes in 20 blocks
> |  ==2909==      possibly lost: 0 bytes in 0 blocks
> |  ==2909==    still reachable: 18,831,940 bytes in 6,769 blocks
>
> Well I think there is no real bad error here. You lost 80 and 240 
> bytes. That
> is nothing to worry about, and somewhat normal. R is a dynamic 
> system,
> valgrind measures "with some error".
>
> You can compare the result to doing an allocation of a longer vector 
> and not
> freeing it.
>
> You could see if freeing your 'mean' variable at the end makes a 
> difference,
> or using a pointer to a single (stack) instance instead of an 
> allocation
> makes a difference.  Likewise, you could returns bers as well. Or 
> free
> it. Right now I am not entirely what it is that your 'prediction' 
> function is
> trying to do.

 It is actually a more complex function with many more allocations and 
 R_tryEval calls; the simplified one that I included leads to exactly the 
 same lost bytes, which means that it is not caused by allocVector, but 
 probably by the  R_tryEval call. Is there any other, more proper way to 
 evaluate R expressions from C?

 In my function I only need to return ber, so I could easily free all R 
 objects (bers and mean here). But C's free() has no result and I am not 
 familiar with any R function that frees objects allocated by 
 allocVector. I have also seen in some R tutorials and examples that 
 objects allocated by allocVector are not freed in any way.

>
> |  Reachable error summary is far too long to include in this mail. 
> The
> |  interesting thing is that reachable errors are all caused by this 
> small
> |  function.
> |
> |  Any ideas? I would also appreciate any suggestions on how to 
> improve
> |  the R-C integration in my code.
>
> Have you considered using R and C++ instead, and looked at Rcpp and
> RInside?
>
>      http://dirk.eddelbuettel.com/code/rcpp.html
>
>      http://dirk.eddelbuettel.com/code/rinside.html
>
> RInside in particular is a lot simpler, at least to me. But some 
> people
> really want plain C in which case you know which route to take
>
> Dirk

 No, unfortunately I cannot switch to C++, because my program is a part 
 of a bigger C project.

 Thanks for your reply and suggestions,
 Nikos



[Rd] C - R integration: Memory Issues
Nikolaos Bezirgiannidis