Enwik8 should support the standard split
NONE
#self-assign
2023-04-11T08:46:16Z
https://github.com/huggingface/datasets/issues/5732#issuecomment-1502926441
https://api.github.com/repos/huggingface/datasets/issues/5732
IC_kwDODunzps5ZlNZp
https://api.github.com/repos/huggingface/datasets/issues/comments/1502926441/reactions
2023-04-11T08:46:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1502926441
https://avatars.githubusercontent.com/u/10287371?v=4
https://api.github.com/users/lucaslingle/events{/privacy}
https://api.github.com/users/lucaslingle/followers
https://api.github.com/users/lucaslingle/following{/other_user}
https://api.github.com/users/lucaslingle/gists{/gist_id}
https://github.com/lucaslingle
lucaslingle
MDQ6VXNlcjEwMjg3Mzcx
https://api.github.com/users/lucaslingle/orgs
https://api.github.com/users/lucaslingle/received_events
https://api.github.com/users/lucaslingle/repos
https://api.github.com/users/lucaslingle/starred{/owner}{/repo}
https://api.github.com/users/lucaslingle/subscriptions
User
https://api.github.com/users/lucaslingle
public
NONE
The Enwik8 pipeline is not present in this codebase, and is hosted elsewhere. I have opened a PR [there](https://huggingface.co/datasets/enwik8/discussions/4) instead. 
2023-04-11T09:28:16Z
https://github.com/huggingface/datasets/issues/5732#issuecomment-1502987625
https://api.github.com/repos/huggingface/datasets/issues/5732
IC_kwDODunzps5ZlcVp
https://api.github.com/repos/huggingface/datasets/issues/comments/1502987625/reactions
2023-04-11T09:28:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1502987625
https://avatars.githubusercontent.com/u/10287371?v=4
https://api.github.com/users/lucaslingle/events{/privacy}
https://api.github.com/users/lucaslingle/followers
https://api.github.com/users/lucaslingle/following{/other_user}
https://api.github.com/users/lucaslingle/gists{/gist_id}
https://github.com/lucaslingle
lucaslingle
MDQ6VXNlcjEwMjg3Mzcx
https://api.github.com/users/lucaslingle/orgs
https://api.github.com/users/lucaslingle/received_events
https://api.github.com/users/lucaslingle/repos
https://api.github.com/users/lucaslingle/starred{/owner}{/repo}
https://api.github.com/users/lucaslingle/subscriptions
User
https://api.github.com/users/lucaslingle
public
### Feature request

The HuggingFace Datasets library currently supports two BuilderConfigs for Enwik8. One config yields individual lines as examples, while the other config yields the entire dataset as a single example. Both support only a monolithic split: it is all grouped as "train". 

The HuggingFace Datasets library should include a BuilderConfig for Enwik8 with train, validation, and test sets derived from the first 90 million bytes, next 5 million bytes, and last 5 million bytes, respectively. This Enwik8 split is standard practice in LM papers, as elaborated and motivated below. 

### Motivation

Enwik8 is commonly split into 90M, 5M, 5M consecutive bytes. This is done in the Transformer-XL [codebase](https://github.com/kimiyoung/transformer-xl/blob/44781ed21dbaec88b280f74d9ae2877f52b492a5/getdata.sh#L34), and is additionally mentioned in the Sparse Transformers [paper](https://arxiv.org/abs/1904.10509) and the Compressive Transformers [paper](https://arxiv.org/abs/1911.05507). This split is pretty much universal among language modeling papers. 

One may obtain the splits by manual wrangling, using the data yielded by the ```enwik8-raw``` BuilderConfig. However, this undermines the seamless functionality of the library: one must slice the single raw example, extract it into three tensors, and wrap each in a separate dataset. 

This becomes even more of a nuisance if using the current Enwik8 HuggingFace dataset as a TfdsDataSource with [SeqIO](https://github.com/google/seqio), where a pipeline of preprocessors is typically included in a SeqIO Task definition, to be applied immediately after loading the data with TFDS. 

### Your contribution

Supporting this functionality in HuggingFace Datasets will only require an additional BuilderConfig for Enwik8 and a few additional lines of code. I will submit a PR. 