From simon.urbanek at r-project.org  Sat Oct 24 23:25:46 2009
From: simon.urbanek at r-project.org (Simon Urbanek)
Date: Sat, 24 Oct 2009 17:25:46 -0400
Subject: [Rd] Confusion regarding allocating Matrices.
In-Reply-To: <fc212e850910241158paee302csc5a75755d8efb56c@mail.gmail.com>
References: <fc212e850910230639r5038f080hd1aad609064cf5b2@mail.gmail.com>
	<40e66e0b0910230723n1cfca327kd8d9ba1127ae46f1@mail.gmail.com>
	<40e66e0b0910230727y4e1c5722p6989ec3e124af534@mail.gmail.com>
	<fc212e850910231202h36f9cb22u1baf2523f61ea274@mail.gmail.com>
	<40e66e0b0910241006m2e52bd87m765cee190ac5e540@mail.gmail.com>
	<fc212e850910241158paee302csc5a75755d8efb56c@mail.gmail.com>
Message-ID: <43D5C158-E52F-44AF-8C75-94610F193C8B@r-project.org>


On Oct 24, 2009, at 2:58 PM, Abhijit Bera wrote:

> Ok I get it. So everytime it does a alloc and copy.
>
> I haven't finished the design yet. I'm just thinking about how  
> randomly the data might arrive; its real time data. So I will  
> allocate a large chunk of memory and keep track of when it fills up,  
> once the data exceeds I will alloc and copy the data (provided the  
> size is  within system limits). In this manner I should be able to  
> reduce the number of expensive operations of allocing and copying.
>

Many smart people have thought about those things before you, it's  
worthwhile to read about it --- I would suggest reading a bit about  
data structures and programming in C. What you describe is usually  
tackled by a allocating additional (usually linked) buffers as you go  
since that means you don't have to copy anything (except for that last  
step where you create the R object). It's also very trivial to  
implement.

Cheers,
Simon


>
>
> Abhijit Bera
>
> On Sat, Oct 24, 2009 at 10:06 PM, Douglas Bates  
> <bates at stat.wisc.edu> wrote:
>
>> On Fri, Oct 23, 2009 at 2:02 PM, Abhijit Bera <abhibera at gmail.com>  
>> wrote:
>>> Sorry, I made a mistake while writing the code. The declaration of  
>>> Data
>>> should have been first.
>>
>>> I still have some doubts:
>>
>> Because you are making some sweeping and incorrect assumptions about
>> the way that the internals of R operate.  R allows for arrays to be
>> dynamically resized but this is accomplished internally by allocating
>> new storage, copying the current contents to this new location and
>> installing the values of the new elements.  It is an expensive
>> operation, which is why it is discouraged.
>>
>> Your design is deeply flawed.  Go back to the drawing board.
>>
>
>
>
>>> When you say calloc and realloc are you talking about R's C  
>>> interface
>> Calloc
>>> and Realloc or the regular calloc and realloc?
>>
>> Either one.
>>
>>> I want to feed data directly into a R matrix and grow it as  
>>> required. So
>> at
>>> one time I might have 100 rows coming in from a data source. The  
>>> next
>> time I
>>> might have 200 rows coming in from a data source. I want to be  
>>> able to
>>> expand the R-matrix instead of creating a regular C float matrix  
>>> and then
>>> make an R-matrix based on the new size. I just want to have one R  
>>> object
>> and
>>> be able to expand it's size dynamically.
>>
>> R stores floating-point numbers as the C data type double, not float.
>> It may seem pedantic to point out distinctions like that but not when
>> you are writing programs.  Compilers are the ultimate pedants - they
>> are real sticklers for getting the details right.
>>
>> As I said, it just doesn't work the way that you think it does.  The
>> fact that there is an R object with a certain name before and after  
>> an
>> operation doesn't mean it is the same R object.
>>
>>> I was reading the language specs. It says that one could declare an
>> object
>>> in R like this:
>>>
>>> m=matrix(nrows=10,ncols=10)
>>>
>>> and then one could assign
>>>
>>> m[101]=1.00
>>>
>>> to expand the object.
>>>
>>> but this has one problem when I do a
>>>
>>> dim(m)
>>>
>>> I get
>>>
>>> NULL instead of 10 10
>>>
>>> So what is happening here?
>>>
>>>
>>> I am aware that R matrices are stored in column major order.
>>>
>>> Thanks for the tip on using float *dat= REAL(Data);
>>>
>>> Regards
>>>
>>> Abhijit Bera
>>>
>>>
>>>
>>> On Fri, Oct 23, 2009 at 7:27 PM, Douglas Bates <bates at stat.wisc.edu>
>> wrote:
>>>>
>>>> On Fri, Oct 23, 2009 at 9:23 AM, Douglas Bates  
>>>> <bates at stat.wisc.edu>
>>>> wrote:
>>>>> On Fri, Oct 23, 2009 at 8:39 AM, Abhijit Bera <abhibera at gmail.com>
>>>>> wrote:
>>>>>> Hi
>>>>>>
>>>>>> I'm having slight confusion.
>>>>>
>>>>> Indeed.
>>>>>
>>>>>> I plan to grow/realloc a matrix depending on the data available  
>>>>>> in a
>> C
>>>>>> program.
>>>>>
>>>>>> Here is what I'm tried to do:
>>>>>
>>>>>> Data=allocMatrix(REALSXP,3,4);
>>>>>> SEXP Data;
>>>>>
>>>>> Those lines should be in the other order, shouldn't they?
>>>>>
>>>>> Also, you need to PROTECT Data or bad things will happen.
>>>>>
>>>>>> REAL(Data)[8]=0.001123;
>>>>>> REAL(Data)[200000]=0.001125;
>>>>>> printf("%f %f\n\n\n\n",REAL(Data)[8],REAL(Data)[200000]);
>>>>
>>>> And I forgot to mention, it is not a good idea to write REAL(Data)
>>>> many times like this.  REAL is a function, not a macro and you are
>>>> calling the same function over and over again unnecessarily.  It is
>>>> better to write
>>>>
>>>> double *dat = REAL(Data);
>>>>
>>>> and use the dat pointer instead of REAL(Data).
>>>>
>>>>>> Here is my confusion:
>>>>>
>>>>>> Do I always require to allocate the exact number of data  
>>>>>> elements in
>> a
>>>>>> R
>>>>>> Matrix?
>>>>>
>>>>> Yes.
>>>>>
>>>>>> In the above code segment I have clearly exceeded the number of
>>>>>> elements that have been allocated but my program doesn't crash.
>>>>>
>>>>> Remember that when programming in C you have a lot of rope with  
>>>>> which
>>>>> to hang yourself.   You have corrupted a memory location beyond  
>>>>> that
>>>>> allocated to the array but nothing bad has happened  - yet.
>>>>>
>>>>>> I don't find any specific R functions for reallocation incase  
>>>>>> my data
>>>>>> set
>>>>>> grows. How do I reallocate?
>>>>>
>>>>> You allocate a new matrix, copy the contents of the current  
>>>>> matrix to
>>>>> the new matrix, then release the old one.  It gets tricky in  
>>>>> that you
>>>>> should unprotect the old one and protect the new one but you  
>>>>> need to
>>>>> watch the order of those operations.
>>>>>
>>>>> This approach is not a very good one.  If you really need to  
>>>>> grow an
>>>>> array it is better to allocate and reallocate the memory within  
>>>>> your C
>>>>> code using calloc and realloc then, at the end of the  
>>>>> calculations,
>>>>> allocate an R matrix and copy the results over.
>>>>>
>>>>> Also, you haven't said whether you are growing the matrix by row  
>>>>> or by
>>>>> column or both.  If you are adding rows then you can't just  
>>>>> reallocate
>>>>> storage because R stores matrices in column-major order. The  
>>>>> positions
>>>>> of the elements in a matrix with n+1 rows are different from  
>>>>> those in
>>>>> a matrix with n rows.
>>>>>
>>>>>> Is it necessary to reallocate or is R handling
>>>>>> the memory management for the matrix that I have allocated?
>>>>>>
>>>>>> Regards
>>>>>>
>>>>>> Abhijit Bera
>>>>>>
>>>>>>       [[alternative HTML version deleted]]
>>>>>>
>>>>>> ______________________________________________
>>>>>> R-devel at r-project.org mailing list
>>>>>> https://stat.ethz.ch/mailman/listinfo/r-devel
>>>>>>
>>>>>
>>>
>>>
>>
>
> 	[[alternative HTML version deleted]]
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>
>



[Rd] Confusion regarding allocating Matrices.
Simon Urbanek