Add the 800GB Pile dataset?
CONTRIBUTOR
The pile dataset would be very nice.
Benchmarks show that pile trained models achieve better results than most of actually trained models
2021-01-02T08:58:19Z
https://github.com/huggingface/datasets/issues/1675#issuecomment-753449420
https://api.github.com/repos/huggingface/datasets/issues/1675
MDEyOklzc3VlQ29tbWVudDc1MzQ0OTQyMA==
https://api.github.com/repos/huggingface/datasets/issues/comments/753449420/reactions
2021-01-02T08:58:19Z
https://api.github.com/repos/huggingface/datasets/issues/comments/753449420
https://avatars.githubusercontent.com/u/47894090?v=4
https://api.github.com/users/flozi00/events{/privacy}
https://api.github.com/users/flozi00/followers
https://api.github.com/users/flozi00/following{/other_user}
https://api.github.com/users/flozi00/gists{/gist_id}
https://github.com/flozi00
flozi00
MDQ6VXNlcjQ3ODk0MDkw
https://api.github.com/users/flozi00/orgs
https://api.github.com/users/flozi00/received_events
https://api.github.com/users/flozi00/repos
https://api.github.com/users/flozi00/starred{/owner}{/repo}
https://api.github.com/users/flozi00/subscriptions
User
https://api.github.com/users/flozi00
public
NONE
The pile can very easily be added and adapted using this [tfds implementation](https://github.com/EleutherAI/The-Pile/blob/master/the_pile/tfds_pile.py) from the repo. 

However, the question is whether you'd be ok with 800GB+ cached in your local disk, since the tfds implementation was designed to offload the storage to Google Cloud Storage.
2021-01-04T22:26:42Z
https://github.com/huggingface/datasets/issues/1675#issuecomment-754260970
https://api.github.com/repos/huggingface/datasets/issues/1675
MDEyOklzc3VlQ29tbWVudDc1NDI2MDk3MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/754260970/reactions
2021-01-04T22:26:42Z
https://api.github.com/repos/huggingface/datasets/issues/comments/754260970
https://avatars.githubusercontent.com/u/4735784?v=4
https://api.github.com/users/trisongz/events{/privacy}
https://api.github.com/users/trisongz/followers
https://api.github.com/users/trisongz/following{/other_user}
https://api.github.com/users/trisongz/gists{/gist_id}
https://github.com/trisongz
trisongz
MDQ6VXNlcjQ3MzU3ODQ=
https://api.github.com/users/trisongz/orgs
https://api.github.com/users/trisongz/received_events
https://api.github.com/users/trisongz/repos
https://api.github.com/users/trisongz/starred{/owner}{/repo}
https://api.github.com/users/trisongz/subscriptions
User
https://api.github.com/users/trisongz
public
MEMBER
With the dataset streaming feature (see #2375) it will be more convenient to play with such big datasets :)
I'm currently adding C4 (see #2511 ) but I can probably start working on this afterwards
2021-06-22T15:43:21Z
https://github.com/huggingface/datasets/issues/1675#issuecomment-866100899
https://api.github.com/repos/huggingface/datasets/issues/1675
MDEyOklzc3VlQ29tbWVudDg2NjEwMDg5OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/866100899/reactions
2021-06-22T15:43:21Z
https://api.github.com/repos/huggingface/datasets/issues/comments/866100899
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Hi folks! Just wanted to follow up on this -- would be really nice to get the Pile on HF Datasets... unclear if it would be easy to also add partitions of the Pile subject to the original 22 datasets used, but that would be nice too!
2021-07-28T21:52:28Z
https://github.com/huggingface/datasets/issues/1675#issuecomment-888645628
https://api.github.com/repos/huggingface/datasets/issues/1675
IC_kwDODunzps4096f8
https://api.github.com/repos/huggingface/datasets/issues/comments/888645628/reactions
2021-07-28T21:52:28Z
https://api.github.com/repos/huggingface/datasets/issues/comments/888645628
https://avatars.githubusercontent.com/u/2498509?v=4
https://api.github.com/users/siddk/events{/privacy}
https://api.github.com/users/siddk/followers
https://api.github.com/users/siddk/following{/other_user}
https://api.github.com/users/siddk/gists{/gist_id}
https://github.com/siddk
siddk
MDQ6VXNlcjI0OTg1MDk=
https://api.github.com/users/siddk/orgs
https://api.github.com/users/siddk/received_events
https://api.github.com/users/siddk/repos
https://api.github.com/users/siddk/starred{/owner}{/repo}
https://api.github.com/users/siddk/subscriptions
User
https://api.github.com/users/siddk
public
MEMBER
Hi folks, thanks to some awesome work by @lhoestq and @albertvillanova you can now stream the Pile as follows:

```python
# Install master branch of `datasets`
pip install git+https://github.com/huggingface/datasets.git#egg=datasets[streaming]
pip install zstandard

from datasets import load_dataset

dset = load_dataset("json", data_files="https://the-eye.eu/public/AI/pile/train/00.jsonl.zst", streaming=True, split="train")
next(iter(dset))
# {'meta': {'pile_set_name': 'Pile-CC'},
# 'text': 'It is done, and submitted. You can play “Survival of the Tastiest” on Android, and on the web ... '}
```

Next step is to add the Pile as a "canonical" dataset that can be streamed without specifying the file names explicitly :)
2021-08-16T15:51:31Z
https://github.com/huggingface/datasets/issues/1675#issuecomment-899618971
https://api.github.com/repos/huggingface/datasets/issues/1675
IC_kwDODunzps41nxib
https://api.github.com/repos/huggingface/datasets/issues/comments/899618971/reactions
2021-08-16T15:51:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/899618971
https://avatars.githubusercontent.com/u/26859204?v=4
https://api.github.com/users/lewtun/events{/privacy}
https://api.github.com/users/lewtun/followers
https://api.github.com/users/lewtun/following{/other_user}
https://api.github.com/users/lewtun/gists{/gist_id}
https://github.com/lewtun
lewtun
MDQ6VXNlcjI2ODU5MjA0
https://api.github.com/users/lewtun/orgs
https://api.github.com/users/lewtun/received_events
https://api.github.com/users/lewtun/repos
https://api.github.com/users/lewtun/starred{/owner}{/repo}
https://api.github.com/users/lewtun/subscriptions
User
https://api.github.com/users/lewtun
public
MEMBER
> Hi folks! Just wanted to follow up on this -- would be really nice to get the Pile on HF Datasets... unclear if it would be easy to also add partitions of the Pile subject to the original 22 datasets used, but that would be nice too!

Hi @siddk thanks to a tip from @richarddwang it seems we can access some of the partitions that EleutherAI created for the Pile [here](https://the-eye.eu/public/AI/pile_preliminary_components/). What's missing are links to the preprocessed versions of pre-existing datasets like DeepMind Mathematics and OpenSubtitles, but worst case we do the processing ourselves and host these components on the Hub.

My current idea is that we could provide 23 configs: one for each of the 22 datasets and an `all` config that links to the train / dev / test splits that EleutherAI released [here](https://the-eye.eu/public/AI/pile/), e.g.

```python
from datasets import load_dataset

# Load a single component
youtube_subtitles = load_dataset("the_pile", "youtube_subtitles")
# Load the train / dev / test splits of the whole corpus
dset = load_dataset("the_pile", "all")
```

Ideally we'd like everything to be compatible with the streaming API and there's ongoing work by @albertvillanova to make this happen for the various compression algorithms.

cc @lhoestq 
2021-08-19T16:54:11Z
https://github.com/huggingface/datasets/issues/1675#issuecomment-902079689
https://api.github.com/repos/huggingface/datasets/issues/1675
IC_kwDODunzps41xKTJ
https://api.github.com/repos/huggingface/datasets/issues/comments/902079689/reactions
2021-08-19T16:54:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/902079689
https://avatars.githubusercontent.com/u/26859204?v=4
https://api.github.com/users/lewtun/events{/privacy}
https://api.github.com/users/lewtun/followers
https://api.github.com/users/lewtun/following{/other_user}
https://api.github.com/users/lewtun/gists{/gist_id}
https://github.com/lewtun
lewtun
MDQ6VXNlcjI2ODU5MjA0
https://api.github.com/users/lewtun/orgs
https://api.github.com/users/lewtun/received_events
https://api.github.com/users/lewtun/repos
https://api.github.com/users/lewtun/starred{/owner}{/repo}
https://api.github.com/users/lewtun/subscriptions
User
https://api.github.com/users/lewtun
public
MEMBER
Ah I just saw that @lhoestq is already thinking about the specifying of one or more subsets in [this PR](https://github.com/huggingface/datasets/pull/2817#issuecomment-901874049) :)
2021-08-19T16:56:44Z
https://github.com/huggingface/datasets/issues/1675#issuecomment-902081391
https://api.github.com/repos/huggingface/datasets/issues/1675
IC_kwDODunzps41xKtv
https://api.github.com/repos/huggingface/datasets/issues/comments/902081391/reactions
2021-08-19T16:56:44Z
https://api.github.com/repos/huggingface/datasets/issues/comments/902081391
https://avatars.githubusercontent.com/u/26859204?v=4
https://api.github.com/users/lewtun/events{/privacy}
https://api.github.com/users/lewtun/followers
https://api.github.com/users/lewtun/following{/other_user}
https://api.github.com/users/lewtun/gists{/gist_id}
https://github.com/lewtun
lewtun
MDQ6VXNlcjI2ODU5MjA0
https://api.github.com/users/lewtun/orgs
https://api.github.com/users/lewtun/received_events
https://api.github.com/users/lewtun/repos
https://api.github.com/users/lewtun/starred{/owner}{/repo}
https://api.github.com/users/lewtun/subscriptions
User
https://api.github.com/users/lewtun
public
## Adding a Dataset
- **Name:** The Pile
- **Description:** The Pile is a 825 GiB diverse, open source language modelling data set that consists of 22 smaller, high-quality datasets combined together. See [here](https://twitter.com/nabla_theta/status/1345130408170541056?s=20) for the Twitter announcement
- **Paper:** https://pile.eleuther.ai/paper.pdf
- **Data:** https://pile.eleuther.ai/
- **Motivation:** Enables hardcore  (GPT-3 scale!) language modelling

## Remarks
Given the extreme size of this dataset, I'm not sure how feasible this will be to include in `datasets` 🤯  . I'm also unsure how many `datasets` users are pretraining LMs, so the usage of this dataset may not warrant the effort to integrate it.
