From g.vegayon at gmail.com  Thu Nov  7 14:30:40 2013
From: g.vegayon at gmail.com (George Vega Yon)
Date: Thu, 7 Nov 2013 10:30:40 -0300
Subject: [Rd] Dynamic list creation (SEXP in C) returns error
 "unimplemented type (29) in 'duplicate'"
In-Reply-To: <527B92F3.7080706@r-enthusiasts.com>
References: <CAPJUxXXWvDjAuoHBp=QZmX4BV2PScNuAUYm5VujervZ_bXeZqA@mail.gmail.com>
	<CADwqtCNS8NMWz2y+88QCG6Q9cSy9=s7aw1RsF+iEvWz6hEYZvw@mail.gmail.com>
	<CAPJUxXWXFctds0veg3Ayy8bD3woWkA46Ss6JFWc_TFjRKDFTZA@mail.gmail.com>
	<CADwqtCPemzpwOjuhVgVmgeHq2zEkF1ViOOzUk6NXNaP3WYvVng@mail.gmail.com>
	<CAPJUxXUfJ5Q1yJOxOMmPLFHE93rj1BzCKZkzNJxVVn4+xt0Smg@mail.gmail.com>
	<CAPJUxXXF7103s06yh1BoHQbzv3Onkb1Ooj+AShZ=oykJnK=uig@mail.gmail.com>
	<527B92F3.7080706@r-enthusiasts.com>
Message-ID: <CAPJUxXW=ZEzcVUmMGctqHFQvdjXSd2X36H_LU4P4qZnju4HZUA@mail.gmail.com>

Romain,

Thanks for your quick response. I've already received that suggestion,
but, besides of haven't ever used C++, I wanted to understand first
what am I doing wrong. Still, would you give me a small example, in R
C++, of:

  - Creating a generic vector "L1" of size N
  - Creating a data.frame "D" and increasing its the number of rows of it
  - Storing the data.frame "D" in the first element of "L1"

I would be very gratefull if you can do that.

Thanks again!

George Vega Yon
+56 9 7 647 2552
http://ggvega.cl


2013/11/7 Romain Francois <romain at r-enthusiasts.com>:
> Hello,
>
> Any particular reason you're not using Rcpp? You would have access to nice
> abstraction instead of these MACROS all over the place.
>
> The cost of these abstractions is close to 0.
>
> Looping around and SET_LENGTH is going to be quite expensive. I would urge
> you to accumulate data in data structures that know how to grow efficiently,
> i.e. a std::vector and then convert that to an R vector when you're done
> with them.
>
> Romain
>
> Le 07/11/2013 14:03, George Vega Yon a ?crit :
>
>> Hi!
>>
>> I didn't wanted to do this but I think that this is the easiest way
>> for you to understand my problem (thanks again for all the comments
>> that you have made). Here is a copy of the function that I'm working
>> on. This may be tedious to analyze, so I understand if you don't feel
>> keen to give it a time. Having dedicated many hours to this (as a new
>> user of both C and R C API), I would be very pleased to know what am I
>> doing wrong here.
>>
>> G0 is a Nx2 matrix. The first column is a group id (can be shared with
>> several observations) and the second tells how many individuals are in
>> that group. This matrix can look something like this:
>>
>> id_group  nreps
>> 1  3
>> 1  3
>> 1  3
>> 2  1
>> 3  1
>> 4  2
>> 5  1
>> 6  1
>> 4  2
>> ...
>>
>> L0 is list of two column data.frames with different sizes. The first
>> column (id) are row indexes (with values 1 to N) and the second column
>> are real numbers. L0 can look something like this
>> [[1]]
>> id  lambda
>> 3  0.5
>> 15  0.3
>> 25  0.2
>> [[2]]
>> id  lambda
>> 15  0.8
>> 40  0.2
>> ...
>> [[N]]
>> id  lambda
>> 80  1
>>
>> TE0 is a int scalar in {0,1,2}
>>
>> T0 is a dichotomous vector of length N that can look something like this
>> [1] 0 1 0 1 1 1 0 ...
>> [N] 1
>>
>> L1 (the expected output) is a modified version of L0, that, for
>> instance can look something like this (note the rows marked with "*")
>>
>> [[1]]
>> id  lambda
>> 3  0.5
>> *15  0.15 (15 was in the same group of 50, so I added this new row and
>> divided the value of lambda by two)
>> 25  0.2
>> *50  0.15
>> [[2]]
>> id  lambda
>> 15  0.8
>> 40  0.2
>> ...
>> [[N]]
>> id  lambda
>> *80  0.333 (80 shared group id with 30 and 100, so lambda is divided by 3)
>> *30  0.333
>> *100 0.333
>>
>> That said, the function is as follows
>>
>> SEXP distribute_lambdas(
>>    SEXP G0,  // Groups ids (matrix of Nx2). First column = Group Id,
>> second column: Elements in the group
>>    SEXP L0,  // List of N two-column dataframes with different number of
>> rows
>>    SEXP TE0, // Treatment effect (int scalar): ATE(0) ATT(1) ATC(2)
>>    SEXP T0   // Treat var (bool vector, 0/1, of size N)
>> )
>> {
>>
>>    int i, j, l, m;
>>    const int *G = INTEGER_POINTER(PROTECT(G0 = AS_INTEGER(G0 )));
>>    const int *T = INTEGER_POINTER(PROTECT(T0 = AS_INTEGER(T0 )));
>>    const int *TE= INTEGER_POINTER(PROTECT(TE0= AS_INTEGER(TE0)));
>>    double *L, val;
>>    int *I, nlambdas, nreps;
>>
>>    const int n = length(T0);
>>
>>    PROTECT_INDEX pin0, pin1;
>>    SEXP L1;
>>    PROTECT(L1 = allocVector(VECSXP,n));
>>    SEXP id, lambda;
>>
>>    // Fixing size
>>    for(i=0;i<n;i++)
>>    {
>>      SET_VECTOR_ELT(L1, i, allocVector(VECSXP, 2));
>>    //  SET_VECTOR_ELT(VECTOR_ELT(L1,i), 0, NEW_INTEGER(100));
>>    //  SET_VECTOR_ELT(VECTOR_ELT(L1,i), 1, NEW_NUMERIC(100));
>>    }
>>
>>    // For over the list, i.e observations
>>    for(i=0;i<n;i++)
>>    {
>>
>>      R_CheckUserInterrupt();
>>
>>      // Checking if has to be analyzed.
>>      if (
>>        ((TE[0] == 1 & !T[i]) | (TE[0] == 2 & T[i])) |
>>        (length(VECTOR_ELT(L0,i)) != 2)
>>      )
>>      {
>>        SET_VECTOR_ELT(L1,i,R_NilValue);
>>        continue;
>>      }
>>
>>      // Checking how many rows does the i-th data.frame has
>>      nlambdas = length(VECTOR_ELT(VECTOR_ELT(L0,i),0));
>>
>>      // Pointing to the data.frame's origianl values
>>      I =
>> INTEGER_POINTER(AS_INTEGER(PROTECT(VECTOR_ELT(VECTOR_ELT(L0,i),0))));
>>      L =
>> NUMERIC_POINTER(AS_NUMERIC(PROTECT(VECTOR_ELT(VECTOR_ELT(L0,i),1))));
>>
>>      // Creating a copy of the pointed values
>>      PROTECT_WITH_INDEX(id   = duplicate(VECTOR_ELT(VECTOR_ELT(L0,i),0)),
>> &pin0);
>>      PROTECT_WITH_INDEX(lambda=duplicate(VECTOR_ELT(VECTOR_ELT(L0,i),1)),
>> &pin1);
>>
>>      // Over the rows of the i-th data.frame
>>      nreps=0;
>>      for(l=0;l<nlambdas;l++)
>>      {
>>        // If the current lambda id is repeated, ie ther are more
>> individuals
>>        // with the same covariates, then enter.
>>        if (G[n+I[l]-1] > 1)
>>        {
>>          /* Changing the length of the object */
>>          REPROTECT(SET_LENGTH(id,    length(lambda) + G[n+I[l]-1] -1),
>> pin0);
>>          REPROTECT(SET_LENGTH(lambda,length(lambda) + G[n+I[l]-1] -1),
>> pin1);
>>
>>          // Getting the new value
>>          val = L[l]/G[n+I[l] - 1];
>>          REAL(lambda)[l] = val;
>>
>>          // Looping over the full set of groups
>>          m = -1,j = -1;
>>          while(m < (G[n+I[l]-1] - 1))
>>          {
>>            // Looking for individuals in the same group
>>            if (G[++j] != G[I[l]-1]) continue;
>>
>>            // If it is the current lambda, then do not asign it
>>            if (j == (I[l] - 1)) continue;
>>
>>            INTEGER(id)[length(id) - (G[n+I[l]-1] - 1) + ++m] = j+1;
>>            REAL(lambda)[length(id) - (G[n+I[l]-1] - 1) + m] = val;
>>          }
>>
>>          nreps+=1;
>>        }
>>      }
>>
>>      if (nreps)
>>      {
>>        // Replacing elements from of the list (modified)
>>        SET_VECTOR_ELT(VECTOR_ELT(L1, i), 0, duplicate(id));
>>        SET_VECTOR_ELT(VECTOR_ELT(L1, i), 1, duplicate(lambda));
>>      }
>>      else {
>>        // Setting the list with the old elements
>>        SET_VECTOR_ELT(VECTOR_ELT(L1, i), 0,
>>          duplicate(VECTOR_ELT(VECTOR_ELT(L0,i),0)));
>>        SET_VECTOR_ELT(VECTOR_ELT(L1, i), 1,
>>          duplicate(VECTOR_ELT(VECTOR_ELT(L0,i),1)));
>>      }
>>
>>      // Unprotecting elements
>>      UNPROTECT(4);
>>    }
>>
>>    Rprintf("Exito\n") ;
>>    UNPROTECT(4);
>>
>>    return L1;
>> }
>>
>> Thanks again in advanced.
>>
>> George Vega Yon
>> +56 9 7 647 2552
>> http://ggvega.cl
>>
>> 2013/11/5 George Vega Yon <g.vegayon at gmail.com>:
>>>
>>> Either way, understanding that it may not be the best way of do it, is
>>> there anything wrong in what I'm doing??
>>> George Vega Yon
>>> +56 9 7 647 2552
>>> http://ggvega.cl
>>>
>>>
>>> 2013/11/5 Gabriel Becker <gmbecker at ucdavis.edu>:
>>>>
>>>> George,
>>>>
>>>> My point is you don't need to create them and then grow them....
>>>>
>>>>
>>>> for(i=0;i<n;i++)
>>>> {
>>>>    // Creating the "id" and "lambda" vectors. I do this in every
>>>> repetition
>>>> of
>>>>    // the loop.
>>>>
>>>>    // ... Some other instructions where I set the value of an integer
>>>>    // z, which tells how much do the vectors have to grow ...
>>>>
>>>> PROTECT(id=allocVector(INTSXP, 4 +z));
>>>> PROTECT(lambda=allocVector(REALSXP, 4 +z));
>>>>
>>>>
>>>>    // ... some lines where I fill the vectors ...
>>>>
>>>>    // Storing the new vectors at the i-th element of the list
>>>>    SET_VECTOR_ELT(VECTOR_ELT(L1, i), 0, duplicate(id));
>>>>    SET_VECTOR_ELT(VECTOR_ELT(L1, i), 1, duplicate(lambda));
>>>>
>>>>    // Unprotecting the "id" and "lambda" vectors
>>>>    UNPROTECT(2);
>>>> }
>>>>
>>>> ~G
>>>>
>>>>
>>>> On Tue, Nov 5, 2013 at 1:56 PM, George Vega Yon <g.vegayon at gmail.com>
>>>> wrote:
>>>>>
>>>>>
>>>>> Gabriel,
>>>>>
>>>>> While the length (in terms of number of SEXP elements it stores) of L1
>>>>> doesn't changes, the vectors within L1 do (sorry if I didn't explained
>>>>> it well before).
>>>>>
>>>>> The post was about a SEXP object that grows, in my case, every pair of
>>>>> vectors in L1 (id and lambda) can change lengths, this is why I need
>>>>> to reprotect them. I populate the i-th element of L1 by creating the
>>>>> vectors "id" and "lambda", setting the length of these according to
>>>>> some rule (that's the part where lengths change)... here is a reduced
>>>>> form of my code:
>>>>>
>>>>> //////////////////////////////////////// C
>>>>> ////////////////////////////////////////
>>>>> const int = length(L0);
>>>>> SEXP L1;
>>>>> PROTECT(L1 = allocVector(VECSXP,n));
>>>>> SEXP id, lambda;
>>>>>
>>>>> // Fixing size
>>>>> for(i=0;i<n;i++)
>>>>>    SET_VECTOR_ELT(L1, i, allocVector(VECSXP, 2));
>>>>>
>>>>> for(i=0;i<n;i++)
>>>>> {
>>>>>    // Creating the "id" and "lambda" vectors. I do this in every
>>>>> repetition
>>>>> of
>>>>>    // the loop.
>>>>>    PROTECT_WITH_INDEX(id=allocVector(INTSXP, 4), &ipx0);
>>>>>    PROTECT_WITH_INDEX(lambda=allocVector(REALSXP, 4), &ipx1);
>>>>>
>>>>>    // ... Some other instructions where I set the value of an integer
>>>>>    // z, which tells how much do the vectors have to grow ...
>>>>>
>>>>>    REPROTECT(SET_LENGTH(id,    length(lambda) + z), ipx0);
>>>>>    REPROTECT(SET_LENGTH(lambda,length(lambda) + z), ipx1);
>>>>>
>>>>>    // ... some lines where I fill the vectors ...
>>>>>
>>>>>    // Storing the new vectors at the i-th element of the list
>>>>>    SET_VECTOR_ELT(VECTOR_ELT(L1, i), 0, duplicate(id));
>>>>>    SET_VECTOR_ELT(VECTOR_ELT(L1, i), 1, duplicate(lambda));
>>>>>
>>>>>    // Unprotecting the "id" and "lambda" vectors
>>>>>    UNPROTECT(2);
>>>>> }
>>>>>
>>>>> UNPROTECT(1);
>>>>>
>>>>> return L1;
>>>>> //////////////////////////////////////// C
>>>>> ////////////////////////////////////////
>>>>>
>>>>> I can't set the length from the start because every pair of vectors in
>>>>> L1 have different lengths, lengths that I cannot tell before starting
>>>>> the loop.
>>>>>
>>>>> Thanks for your help,
>>>>>
>>>>> Regards,
>>>>>
>>>>> George Vega Yon
>>>>> +56 9 7 647 2552
>>>>> http://ggvega.cl
>>>>>
>>>>>
>>>>> 2013/11/5 Gabriel Becker <gmbecker at ucdavis.edu>:
>>>>>>
>>>>>> George,
>>>>>>
>>>>>> I don't see the relevance of the stackoverflow post you linked. In the
>>>>>> post,
>>>>>> the author wanted to change the length of an existing "mother list"
>>>>>> (matrix,
>>>>>> etc), while you specifically state that the length of L1 will not
>>>>>> change.
>>>>>>
>>>>>> You say that the child lists (vectors if they are INTSXP/REALSXP) are
>>>>>> variable, but that is not what the linked post was about unless I am
>>>>>> completely missing something.
>>>>>>
>>>>>> I can't really say more without knowing the details of how the vectors
>>>>>> are
>>>>>> being created and why they cannot just have the right length from the
>>>>>> start.
>>>>>>
>>>>>> As for the error, that is a weird one. I imagine it means that a SEXP
>>>>>> thinks
>>>>>> that it has a type other than ones defined in Rinternals. I can't
>>>>>> speak
>>>>>> to
>>>>>> how that could have happened from what you posted though.
>>>>>>
>>>>>> Sorry I can't be of more help,
>>>>>> ~G
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Nov 4, 2013 at 8:00 PM, George Vega Yon <g.vegayon at gmail.com>
>>>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>> Dear R-devel,
>>>>>>>
>>>>>>> A couple of weeks ago I started to use the R C API for package
>>>>>>> development. Without knowing much about C, I've been able to write
>>>>>>> some routines sucessfully... until now.
>>>>>>>
>>>>>>> My problem consists in dynamically creating a list ("L1") of lists
>>>>>>> using .Call, the tricky part is that each element of the "mother
>>>>>>> list"
>>>>>>> contains two vectors (INTSXP and REALEXP types) with varying sizes;
>>>>>>> sizes that I set while I'm looping over another list's ("L1")
>>>>>>> elements
>>>>>>>   (input list). The steps I've follow are:
>>>>>>>
>>>>>>> FIRST: Create the "mother list" of size "n=length(L0)" (doesn't
>>>>>>> change) and protect it as
>>>>>>>    PROTECT(L1=allocVector(VECEXP, length(L0)))
>>>>>>> and filling it with vectors of length two:
>>>>>>>    for(i=0;i<n;i++) SET_VECTOR_ELT(L1,i, allocVector(VECSXP, 2));
>>>>>>>
>>>>>>> then, for each element of the mother list:
>>>>>>>
>>>>>>>    for(i=0;i<n;i++) {
>>>>>>>
>>>>>>> SECOND: By reading this post in Stackoverflow
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> http://stackoverflow.com/questions/7458364/growing-an-r-matrix-inside-a-c-loop/7458516#7458516
>>>>>>> I understood that it was necesary to (1) create the "child lists" and
>>>>>>> protecting them with PROTECT_WITH_INDEX, and (2) changing its size
>>>>>>> using SETLENGTH (Rf_lengthgets) and REPROTECT ing the lists in order
>>>>>>> to tell the GC that the vectors had change.
>>>>>>>
>>>>>>> THIRD: Once my two vectors are done ("id" and "lambda"), assign them
>>>>>>> to the i-th element of the "mother list" L1 using
>>>>>>>    SET_VECTOR_ELT(VECTOR_ELT(L1,i), 0, duplicate(id));
>>>>>>>    SET_VECTOR_ELT(VECTOR_ELT(L1,i), 1, duplicate(lambda));
>>>>>>>
>>>>>>> and unprotecting the elements protected with index: UNPROTECT(2);
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> FOURTH: Unprotecting the "mother list" (L1) and return it to R
>>>>>>>
>>>>>>> With small datasets this works fine, but after trying with bigger
>>>>>>> ones
>>>>>>> R (my code) keeps failing and returning a strange error that I
>>>>>>> haven't
>>>>>>> been able to identify (or find in the web)
>>>>>>>
>>>>>>>    "unimplemented type (29) in 'duplicate'"
>>>>>>>
>>>>>>> This happens right after I try to use the returned list from my
>>>>>>> routine (trying to print it or building a data-frame).
>>>>>>>
>>>>>>> Does anyone have an idea of what am I doing wrong?
>>>>>>>
>>>>>>> Best regards,
>>>>>>>
>>>>>>> PS: I didn't wanted to copy the entire function... but if you need it
>>>>>>> I can do it.
>>>>>>>
>>>>>>> George Vega Yon
>>>>>>> +56 9 7 647 2552
>>>>>>> http://ggvega.cl
>>>>>>>
>>>>>>> ______________________________________________
>>>>>>> R-devel at r-project.org mailing list
>>>>>>> https://stat.ethz.ch/mailman/listinfo/r-devel
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Gabriel Becker
>>>>>> Graduate Student
>>>>>> Statistics Department
>>>>>> University of California, Davis
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Gabriel Becker
>>>> Graduate Student
>>>> Statistics Department
>>>> University of California, Davis
>>
>>
>> ______________________________________________
>> R-devel at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-devel
>>
>
>
> --
> Romain Francois
> Professional R Enthusiast
> +33(0) 6 28 91 30 30
>
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel



[Rd] Dynamic list creation (SEXP in C) returns error "unimplemented type (29) in 'duplicate'"
George Vega Yon