From h.wickham at gmail.com  Thu Dec 10 01:10:45 2009
From: h.wickham at gmail.com (hadley wickham)
Date: Wed, 9 Dec 2009 18:10:45 -0600
Subject: [Rd] reshape() makes R run out of memory (PR#14121)
In-Reply-To: <4B200C4C.2000506@biostat.ku.dk>
References: <20091209073511.4E6B2282EFC0@mail.pubhealth.ku.dk> 
	<4B200C4C.2000506@biostat.ku.dk>
Message-ID: <f8e6ff050912091610n2e8b56bcn8eb1f03b0f264d30@mail.gmail.com>

> Yes. The culprit would seem to be interaction(), as in
>
>> x <- y <- z <- 1:999
>> i <- interaction(x,y,z, drop=TRUE)
> Error: cannot allocate vector of size 3.7 Gb
>
> which is happening due to the occurrence of three idvar variables. This
> works basically as interaction(x,y,z)[,drop=TRUE], i.e. it first creates a
> factor with 999^3 levels, and removes the empty levels afterward.
>
> In the absense of a better interaction(), you might try making your own
> single idvar as do.call("paste",tbl[,c("ID", "DATE1", "DATE2")]) or so.

There's also ninteraction in the plyr package, which has been designed
to generate a unique integer for each combination (while maintaining
the original order of the data and any missing combinations) as
efficiently as possible.  It's much faster than interaction(..., drop
= T) and I hope it would be faster than paste since it works with
integers rather than strings.

Hadley

-- 
http://had.co.nz/



[Rd] reshape() makes R run out of memory (PR#14121)
hadley wickham