From msuzen at gmail.com  Mon Sep  4 11:35:34 2017
From: msuzen at gmail.com (Suzen, Mehmet)
Date: Mon, 4 Sep 2017 11:35:34 +0200
Subject: [Rd] Suggestion: Create On-Disk Dataframes
In-Reply-To: <CANNd7=n0Eq0jytoaxxFJeWU8uonxivfHSDbp7-fw0=_AwHFiDQ@mail.gmail.com>
References: <CANNd7=nZ1HSSv9WrQ7Zqnqtby0kUGGMSDAj8e0WFZCJ46xS3pw@mail.gmail.com>
 <CANNd7=n4SZTy1TO2B=wk3SLBh0SuC-wrcJDfBVLbdFYxeeEfgQ@mail.gmail.com>
 <CANNd7=nO2c0jzFBBLmsKYubBA87wqxMPtU3uv3JuzrUaVtj0pA@mail.gmail.com>
 <CANNd7=kYm9O0y4SJbRxpDWR_8BFYRruatCd_BP_RXWf+bdrw7A@mail.gmail.com>
 <CANNd7==4bmosbdRzuNwn3kORYRWivbi0T885s6t+TWCR4oRefQ@mail.gmail.com>
 <CANNd7=nyvPMx9myXrox0=KU-v6hB_pTg-j5gpj_+18K31uBRqg@mail.gmail.com>
 <CANNd7=n3x-y9dV2==-uiCu7MCMcWEiPES3TfoFi-m6JbLWHqTw@mail.gmail.com>
 <CANNd7=nsOBSRt0tmzpLyvf9q2a6RrvhwbHcQAJvCagZi28L6Ng@mail.gmail.com>
 <CANNd7=kg1z6rpC83QEQ8GamS4jSaWJT0Bs2aotq=_TDgyFwrBA@mail.gmail.com>
 <CANNd7=m3kMV+3ny0t5d-8m3XVj1dZKXNkrimS28UAVSHOMTiPw@mail.gmail.com>
 <CANNd7==GsZyquFb+dVx_GGnrGdpC6KgK=xp_apNp5jp0tHUBAg@mail.gmail.com>
 <CANNd7=nsbOv53+vPYrfyeP5tUgnzEFJe0pb5sW5iOxmSz-+1iw@mail.gmail.com>
 <CANNd7=n0Eq0jytoaxxFJeWU8uonxivfHSDbp7-fw0=_AwHFiDQ@mail.gmail.com>
Message-ID: <CAPtbhHxgyZxpXBQKKSTPyGSZBMa-YiyHgQibcmSG-uNT65s3Ew@mail.gmail.com>

It is not needed. There is a large community of developer using SparkR.
https://spark.apache.org/docs/latest/sparkr.html
It does exactly what you want.

On 3 September 2017 at 20:38, Juan Telleria <jtelleriar at gmail.com> wrote:
> Dear R Developers,
>
> I would like to suggest the creation of a new S4 object class for On-Disk
> data.frames which do not fit in RAM memory, which could be called
> disk.data.frame()
>
> It could be based in rsqlite for example (By translating R syntax to SQL
> syntax for example), and the syntax and way of working of the
> disk.data.frame() class could be exactly the same than with data.frame
> objects.
>
> When the session is of, is such disk.data.frames are not saved, and
> implicit DROP TABLE could be done in all the schemas created in rsqlite.
>
> Nowadays, with the SSD disk drives such new data.frame() class could have
> sense, specially when dealing with Big Data.
>
> It is true that this new class might be slower than regular data.frame,
> data.table or tibble classes, but we would be able to handle much more
> data, even if it is at cost of speed.
>
> Also with data sampling, and use of a regular odbc connection we could do
> all the work, but for people who do not know how to use RDBMS or specific
> purpose R packages for this job, this could work.
>
> Another option would be to base this new S4 class  on feather files, but
> maybe making it with rsqlite is simply easier.
>
> A GitHub project could be created for such purpose, so that all the
> community can contribute (included me :D ).
>
> Thank you,
> Juan
>
>         [[alternative HTML version deleted]]
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel



[Rd] Suggestion: Create On-Disk Dataframes
Suzen, Mehmet