From ggrothendieck at myway.com  Sat Feb 26 15:28:46 2005
From: ggrothendieck at myway.com (Gabor Grothendieck)
Date: Sat Feb 26 15:33:24 2005
Subject: [Rd] Re: [R] Do environments make copies?
References: <200502242124.j1OLOlcr017655@volta.gene.com>
	<Pine.LNX.4.61.0502241606570.12878@nokomis.stat.uiowa.edu>
	<42208650.3010402@inktomi.com>
Message-ID: <loom.20050226T152805-42@post.gmane.org>


See ?gctorture

Nawaaz Ahmed <nawaaz <at> inktomi.com> writes:

: 
: Hi Folks,
: Thanks for all your replies and input. In particular, thanks Luke, for 
: explaining what is happening under the covers. In retrospect, my example 
:   using save and load to demonstrate the problem I was having was a 
: mistake - I was trying to reproduce the problem I was having in a simple 
: enough way and I thought save and load were showing the same problem 
: (i.e. an extra copy was being made). After carefully examining my gc() 
: traces,
: I've come to realize that while there are copies being made, there is 
: nothing unexpected about it - the failure to allocate memory is really 
: because R is hitting the 3GB address limit imposed by my linux box 
: during processing. So as Luke suggests, maybe 32 bits is not the right 
: platform for handling large data in R.
: 
: On the other hand, I think the problem can be somewhat alleviated 
: (though not eliminated) if we did garbage collection of temporary 
: variables immediately so that we can reduce the memory footprint and the 
: fragmentation problem that malloc() is going to be faced with 
: (gctorture() is probably too extreme . Most of the problems that I am 
: having  are in the coercion routines which do create temporary copies. 
: So in code of the form x = as.vector(x), it would be nice if the old 
: value of x was garbage collected (i.e. if there were no references to it)
: 
: nawaaz
: 
: Luke Tierney wrote:
: > On Thu, 24 Feb 2005, Berton Gunter wrote:
: > 
: >> I was hoping that one of the R gurus would reply to this, but as they 
: >> have't
: >> (thus far) I'll try. Caveat emptor!
: >>
: >> First of all, R passes function arguments by values, so as soon as you 
: >> call
: >> foo(val) you are already making (at least) one other copy of val for the
: >> call.
: > 
: > 
: > Conceptually you have a copy, but internally R trieas to use a
: > copy-on-modify strategy to avaoid copying unless necessary.  THere are
: > conservative approximations involved, so there is more copying than
: > one might like but definitely not as much as this.
: > 
: > 
: >> Second,you seem to implicitly make the assumption that assign(..., env=)
: >> uses a pointer to point to the values in the environment. I do not 
: >> know how
: >> R handles environments and assignments like this internally, but your 
: >> data
: >> seems to indicate that it copies the value and does not merely point 
: >> to it
: >> (this is where R Core folks can shed more authoritative light).
: > 
: > 
: > This assignment does just store the pointer.
: > 
: >> Finally, it makes perfect sense to me that, as a data structure, the
: >> environment itself may be small even if it effectively points to (one of
: >> several copies of) large objects, so that object.size(an.environment) 
: >> could
: >> be small although the environment may "contain" huge arguments. Again, 
: >> the
: >> details depend on the precise implementation and need clarification by
: >> someone who actually knows what's going on here, which ain't me.
: >>
: >> I think the important message is that you shouldn't treat R as C, and you
: >> shouldn't try to circumvent R's internal data structures and 
: >> conventions. R
: >> is a language designed to implements Chambers's S model of 
: >> "Programming with
: >> Data." Instead of trying to fool R to handle large data sets, maybe you
: >> should consider whether you really **need** all the data in R at one time
: >> and if sensible partitioning or sampling to analyze only a portion or
: >> portions of the data might not be a more effective strategy.
: > 
: > 
: > R can do quite a reasonable job with large data sets on a resonable
: > platform.  A 32 bit platform is not a reasonable one on which to use R
: > with 800 MB chunks of data. Automatic memory management combined with
: > the immutable vector semantics require more elbow room than that.  If
: > you really must use data of this size on a 32-bit platform you will
: > probably be muchhappier using a limited amoutn of C code and external
: > pointers.
: > 
: > As to what is happening in this example: look at the default parent
: > used by new.env and combine that with the fact that the serialization
: > code does not preserve sharing of atomic objects.  The two references
: > to the large object are shared in the original session but lead to two
: > large objects in the saved image and the load.  Using
: > 
: >     ref <- list(env = new.env(parent = .GlobalEnv))
: > 
: > in new.ref avoids the second copy both in the saved image and after
: > loading.
: > 
: > luke
: > 
: >>
: >>> -----Original Message-----
: >>> From: r-help-bounces <at> stat.math.ethz.ch
: >>> [mailto:r-help-bounces <at> stat.math.ethz.ch] On Behalf Of Nawaaz Ahmed
: >>> Sent: Thursday, February 24, 2005 10:36 AM
: >>> To: r-help <at> stat.math.ethz.ch
: >>> Subject: [R] Do environments make copies?
: >>>
: >>> I am using environments to avoid making copies (by keeping
: >>> references).
: >>> But it seems like there is a hidden copy going on somewhere - for
: >>> example in the code fragment below, I am creating a reference to "y"
: >>> (of size 500MB) and storing the reference in object "data".
: >>> But when I
: >>> save "data" and then restore it in another R session, gc()
: >>> claims it is
: >>> using twice the amount of memory. Where/How is this happening?
: >>>
: >>> Thanks for any help in working around this - my datasets are just not
: >>> fitting into my 4GB, 32 bit linux machine (even though my actual data
: >>> size is around 800MB)
: >>>
: >>> Nawaaz
: >>>
: >>> > new.ref <- function(value = NULL) {
: >>> +     ref <- list(env = new.env())
: >>> +     class(ref) <- "refObject"
: >>> +     assign("value", value, env = ref$env)
: >>> +     ref
: >>> + }
: >>> > object.size(y)
: >>> [1] 587941404
: >>> > y.ref = new.ref(y)
: >>> > object.size(y.ref)
: >>> [1] 328
: >>> > data = list()
: >>> > data$y.ref = y.ref
: >>> > object.size(data)
: >>> [1] 492
: >>> > save(data, "data.RData")
: >>>
: >>> ...
: >>>
: >>> run R again
: >>> ===========
: >>>
: >>> > load("data.RData")
: >>> > gc()
: >>>              used   (Mb) gc trigger   (Mb)
: >>> Ncells    141051    3.8     350000    9.4
: >>> Vcells 147037925 1121.9  147390241 1124.5
: >>>
: >>> ______________________________________________
: >>> R-help <at> stat.math.ethz.ch mailing list
: >>> https://stat.ethz.ch/mailman/listinfo/r-help
: >>> PLEASE do read the posting guide!
: >>> http://www.R-project.org/posting-guide.html
: >>>
: >>
: >> ______________________________________________
: >> R-help <at> stat.math.ethz.ch mailing list
: >> https://stat.ethz.ch/mailman/listinfo/r-help
: >> PLEASE do read the posting guide! 
: >> http://www.R-project.org/posting-guide.html
: >>
: >
: 
: ______________________________________________
: R-devel <at> stat.math.ethz.ch mailing list
: https://stat.ethz.ch/mailman/listinfo/r-devel
: 
:


[Rd] Re: [R] Do environments make copies?
Gabor Grothendieck