From simon.urbanek at r-project.org  Wed Aug 28 20:52:45 2013
From: simon.urbanek at r-project.org (Simon Urbanek)
Date: Wed, 28 Aug 2013 14:52:45 -0400
Subject: [Rd] Memory allocation in read.table
In-Reply-To: <CABdHhvFWjUT+9W9nGxp-JcV-w03J69rffiZi3JdWsgoNXASNgg@mail.gmail.com>
References: <CABdHhvEtVhBQS7x+S86my3ec26-iCQx+NpVdzM4nh5ca=pGfqg@mail.gmail.com>
	<6603B703-48C9-417E-9495-F68B4015F381@r-project.org>
	<CABdHhvE9LrcseP93KTG4AtkrgffDF9FgkQH-kBnP_fgJRt+i8w@mail.gmail.com>
	<0CA3536B-D70E-4B6E-A888-A6992B17DEC3@r-project.org>
	<CABdHhvFWjUT+9W9nGxp-JcV-w03J69rffiZi3JdWsgoNXASNgg@mail.gmail.com>
Message-ID: <08647436-E8B7-4579-BF51-80DADD441190@r-project.org>

On Aug 28, 2013, at 2:24 PM, Hadley Wickham wrote:

>> Yup - parsing is the most expensive part. That's why for high-throughput data you don't want to use ASCII representation. It's amazing that the disk speeds are now so high that CPUs are the bottlenecks now, not vice versa.
> 
> Do you have any recommendations for binary formats? For R, is there anything obviously better than Rdata?
> 

native formats are the fastest (and versatile), so
readBin/writeBin or mmap
I tend to avoid strings (I use dates as POSIXct which are doubles and for anything else factors - which are integers) so the above works for me just fine.
I am working on a way to do direct mmap serialization of SEXPs but it's not ready yet (basic vectors are supported but complex objects not yet).

Cheers,
Simon



[Rd] Memory allocation in read.table
Simon Urbanek