From hadley at rice.edu  Wed May  2 18:27:04 2012
From: hadley at rice.edu (Hadley Wickham)
Date: Wed, 2 May 2012 11:27:04 -0500
Subject: [Rd] Decompressing raw vectors in memory
In-Reply-To: <4FA15DDD.4040607@stats.ox.ac.uk>
References: <CABdHhvHZLaGEcmuqLN-pzd_oGfJt8O_doU9JT8wvHwn=_pMyZw@mail.gmail.com>
	<4FA150E4.3090907@stats.ox.ac.uk>
	<CABdHhvEdSD=yJLDJwM+Cht+uh0aBjpuGDTzZmPyaKKdPDBOjOQ@mail.gmail.com>
	<4FA15DDD.4040607@stats.ox.ac.uk>
Message-ID: <CABdHhvGYjNavx695yz32NBYw0Er_6oN60KnVXKQU=Vu8F724yA@mail.gmail.com>

> Well, it seems what you get there depends on the client, but I did
>
> tystie% curl -o foo "http://httpbin.org/gzip"
> tystie% file foo
> foo: gzip compressed data, last modified: Wed May ?2 17:06:24 2012, max
> compression
>
> and the final part worried me: I do not know if memDecompress() knows about
> that format. ?The help page does not claim it can do anything other than
> de-compress the results of memCompress() (although past experience has shown
> that it can in some cases). ?gzfile() supports a much wider range of
> formats.

Ah, ok.  Thanks.  Then in that case it's probably just as easy to save
it to a temp file and read that.

  con <- file(tmp) # R automatically detects compression
  open(con, "rb")
  on.exit(close(con), TRUE)

  readBin(con, raw(), file.info(tmp)$size * 10)

The only challenge is figuring out what n to give readBin. Is there a
good general strategy for this?  Guess based on the file size and then
iterate until result of readBin has length less than n?

  n <- file.info(tmp)$size * 2
  content <- readBin(con, raw(),  n)
  n_read <- length(content)
  while(n_read == n) {
    more <- readBin(con, raw(),  n)
    content <- c(content, more)
    n_read <- length(more)
  }

Which is not great style, but there shouldn't be many reads.

Hadley


-- 
Assistant Professor / Dobelman Family Junior Chair
Department of Statistics / Rice University
http://had.co.nz/



[Rd] Decompressing raw vectors in memory
Hadley Wickham