add openwebtext
CONTRIBUTOR
BTW, is there a one-line command to make our building scripts pass flake8 test? (included code quality test), I got like trailing space or mixed space and tab warning and error, and fixed them manually.
2020-09-23T10:44:33Z
https://github.com/huggingface/datasets/pull/660#issuecomment-697284077
https://api.github.com/repos/huggingface/datasets/issues/660
MDEyOklzc3VlQ29tbWVudDY5NzI4NDA3Nw==
https://api.github.com/repos/huggingface/datasets/issues/comments/697284077/reactions
2020-09-23T10:44:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/697284077
https://avatars.githubusercontent.com/u/17963619?v=4
https://api.github.com/users/richarddwang/events{/privacy}
https://api.github.com/users/richarddwang/followers
https://api.github.com/users/richarddwang/following{/other_user}
https://api.github.com/users/richarddwang/gists{/gist_id}
https://github.com/richarddwang
richarddwang
MDQ6VXNlcjE3OTYzNjE5
https://api.github.com/users/richarddwang/orgs
https://api.github.com/users/richarddwang/received_events
https://api.github.com/users/richarddwang/repos
https://api.github.com/users/richarddwang/starred{/owner}{/repo}
https://api.github.com/users/richarddwang/subscriptions
User
https://api.github.com/users/richarddwang
public
MEMBER
> BTW, is there a one-line command to make our building scripts pass flake8 test? (included code quality test), I got like trailing space or mixed space and tab warning and error, and fixed them manually.

I don't think so.
We have a command for black and isort but not flake8 as far as I know.
2020-09-28T08:52:37Z
https://github.com/huggingface/datasets/pull/660#issuecomment-699873772
https://api.github.com/repos/huggingface/datasets/issues/660
MDEyOklzc3VlQ29tbWVudDY5OTg3Mzc3Mg==
https://api.github.com/repos/huggingface/datasets/issues/comments/699873772/reactions
2020-09-28T08:52:37Z
https://api.github.com/repos/huggingface/datasets/issues/comments/699873772
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Thanks for your awesome work too.
BTW a little reminder, this solves #132 
2020-09-29T04:51:23Z
https://github.com/huggingface/datasets/pull/660#issuecomment-700424779
https://api.github.com/repos/huggingface/datasets/issues/660
MDEyOklzc3VlQ29tbWVudDcwMDQyNDc3OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/700424779/reactions
2020-09-29T04:51:50Z
https://api.github.com/repos/huggingface/datasets/issues/comments/700424779
https://avatars.githubusercontent.com/u/17963619?v=4
https://api.github.com/users/richarddwang/events{/privacy}
https://api.github.com/users/richarddwang/followers
https://api.github.com/users/richarddwang/following{/other_user}
https://api.github.com/users/richarddwang/gists{/gist_id}
https://github.com/richarddwang
richarddwang
MDQ6VXNlcjE3OTYzNjE5
https://api.github.com/users/richarddwang/orgs
https://api.github.com/users/richarddwang/received_events
https://api.github.com/users/richarddwang/repos
https://api.github.com/users/richarddwang/starred{/owner}{/repo}
https://api.github.com/users/richarddwang/subscriptions
User
https://api.github.com/users/richarddwang
public
This adds [The OpenWebText Corpus](https://skylion007.github.io/OpenWebTextCorpus/), which is a clean and large text corpus for nlp pretraining. It is an open source effort to reproduce OpenAI’s WebText dataset used by GPT-2, and it is also needed to reproduce ELECTRA.

It solves #132 .

### Besides dataset building script, I made some changes to the library.

1. Extract large amount of compressed files with multi processing
I add a `num_proc` argument to `DownloadManager.extract` and pass this `num_proc` to `map_nested`. So I can decompress 20 thousands compressed files faster. `num_proc` I add is default to `None`, so it shouldn't break any other thing.

2. In `cached_path`, I change the order to deal with different kind of compressed files (zip, tar, gzip)
Because there is no way to 100% detect a file is a zip file (see [this](https://stackoverflow.com/questions/18194688/how-can-i-determine-if-a-file-is-a-zip-file)), I found it wrongly detect `'./datasets/downloads/extracted/58764bd6898fa339b25d92e7fbbc3d8dbf64fb504edff1a30a1d7d99d1561027/openwebtext/urlsf_subset13-630_data.xz'` as a zip and try decompress it with zip, sure it will get error. So I made it detect wheter the file is tar or gzip first and detect zip in the last.

3. `MockDownloadManager.extract`
Cuz I pass `num_proc` to `DownloadManager.extract`, I also have to make `MockDownloadManager.extract` to accept extra keywork arguments. So I make it `extract(path, *args, **kwargs)`, but just return the path as original implementation.

**Note**: If there is better way for points mentioned above, thought I would like to help, unless we can solve point4 (make dataset building fast), I may not be able to afford rebuild the dataset again because of change of the dataset script (Building the dataset cost me 4 days). 

### There is something I think we can improve

4. Long time to decompress compressed files
Even I decompress those 20 thousands compressed files with 12 process on my 16 core 3.x Ghz server. It still took about 3 ~ 4days to complete dataset building. Most of time spent on decompress those files.

### Info about the source data
The source data is an  tar.xz file with following structure, files/directory beyond compressed file is what can we get after decompress it.
```
openwebtext.tar.xz
  |__ openwebtext
         |__subset000.xz
         |     |__ ....txt
         |     |__ ....txt
         |     ...
         |__ subset001.xz
         |
         ....
```
And this the structure of dummy data, same as the original one.
```
dummy_data.zip
  |__ dummy_data
         |__ openwebtext
              |__fake_subset-1_data-dirxz # actually it is a directory
              |     |__ ....txt
              |     |__ ....txt
              |__ fake_subset-2_data-dirxz
                    |__ ....txt
                    |__ ....txt
```