[Datasets ToDo-List] add datasets
MEMBER
Note:
```
nlp-cli test datasets/nlp/<your-dataset-folder> --save_checksums --all_configs
```
directly saves the checksums in the right place, and runs for all the dataset configurations.
2020-05-04T08:47:09Z
https://github.com/huggingface/datasets/pull/37#issuecomment-623337880
https://api.github.com/repos/huggingface/datasets/issues/37
MDEyOklzc3VlQ29tbWVudDYyMzMzNzg4MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/623337880/reactions
2020-05-04T08:47:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/623337880
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
@patrickvonplaten can you provide the add the link to the PR for the dummy data? 
2020-05-04T09:16:49Z
https://github.com/huggingface/datasets/pull/37#issuecomment-623352235
https://api.github.com/repos/huggingface/datasets/issues/37
MDEyOklzc3VlQ29tbWVudDYyMzM1MjIzNQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/623352235/reactions
2020-05-04T09:16:49Z
https://api.github.com/repos/huggingface/datasets/issues/comments/623352235
https://avatars.githubusercontent.com/u/38249783?v=4
https://api.github.com/users/mariamabarham/events{/privacy}
https://api.github.com/users/mariamabarham/followers
https://api.github.com/users/mariamabarham/following{/other_user}
https://api.github.com/users/mariamabarham/gists{/gist_id}
https://github.com/mariamabarham
mariamabarham
MDQ6VXNlcjM4MjQ5Nzgz
https://api.github.com/users/mariamabarham/orgs
https://api.github.com/users/mariamabarham/received_events
https://api.github.com/users/mariamabarham/repos
https://api.github.com/users/mariamabarham/starred{/owner}{/repo}
https://api.github.com/users/mariamabarham/subscriptions
User
https://api.github.com/users/mariamabarham
public
CONTRIBUTOR
https://github.com/huggingface/nlp/pull/15 - But it's probably best to checkout into this branch and look how the dummy data strtucture is for `squad` for example.
2020-05-04T09:20:04Z
https://github.com/huggingface/datasets/pull/37#issuecomment-623353725
https://api.github.com/repos/huggingface/datasets/issues/37
MDEyOklzc3VlQ29tbWVudDYyMzM1MzcyNQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/623353725/reactions
2020-05-04T09:20:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/623353725
https://avatars.githubusercontent.com/u/23423619?v=4
https://api.github.com/users/patrickvonplaten/events{/privacy}
https://api.github.com/users/patrickvonplaten/followers
https://api.github.com/users/patrickvonplaten/following{/other_user}
https://api.github.com/users/patrickvonplaten/gists{/gist_id}
https://github.com/patrickvonplaten
patrickvonplaten
MDQ6VXNlcjIzNDIzNjE5
https://api.github.com/users/patrickvonplaten/orgs
https://api.github.com/users/patrickvonplaten/received_events
https://api.github.com/users/patrickvonplaten/repos
https://api.github.com/users/patrickvonplaten/starred{/owner}{/repo}
https://api.github.com/users/patrickvonplaten/subscriptions
User
https://api.github.com/users/patrickvonplaten
public
MEMBER
are lock files supposed to stay ?
2020-05-04T13:26:59Z
https://github.com/huggingface/datasets/pull/37#issuecomment-623463019
https://api.github.com/repos/huggingface/datasets/issues/37
MDEyOklzc3VlQ29tbWVudDYyMzQ2MzAxOQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/623463019/reactions
2020-05-04T13:26:59Z
https://api.github.com/repos/huggingface/datasets/issues/comments/623463019
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
> are lock files supposed to stay ?

Not sure! I think the checksum command creates them, so I just uploaded them as well.
2020-05-04T13:31:06Z
https://github.com/huggingface/datasets/pull/37#issuecomment-623465237
https://api.github.com/repos/huggingface/datasets/issues/37
MDEyOklzc3VlQ29tbWVudDYyMzQ2NTIzNw==
https://api.github.com/repos/huggingface/datasets/issues/comments/623465237/reactions
2020-05-04T13:31:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/623465237
https://avatars.githubusercontent.com/u/23423619?v=4
https://api.github.com/users/patrickvonplaten/events{/privacy}
https://api.github.com/users/patrickvonplaten/followers
https://api.github.com/users/patrickvonplaten/following{/other_user}
https://api.github.com/users/patrickvonplaten/gists{/gist_id}
https://github.com/patrickvonplaten
patrickvonplaten
MDQ6VXNlcjIzNDIzNjE5
https://api.github.com/users/patrickvonplaten/orgs
https://api.github.com/users/patrickvonplaten/received_events
https://api.github.com/users/patrickvonplaten/repos
https://api.github.com/users/patrickvonplaten/starred{/owner}{/repo}
https://api.github.com/users/patrickvonplaten/subscriptions
User
https://api.github.com/users/patrickvonplaten
public
MEMBER
We can trash the `lock` file, they are dummy file that are only used to avoid concurrent access when the library is run.
You can read the filelock readme and code, it's a very simple single-file library: https://github.com/benediktschmitt/py-filelock
2020-05-04T19:35:15Z
https://github.com/huggingface/datasets/pull/37#issuecomment-623662192
https://api.github.com/repos/huggingface/datasets/issues/37
MDEyOklzc3VlQ29tbWVudDYyMzY2MjE5Mg==
https://api.github.com/repos/huggingface/datasets/issues/comments/623662192/reactions
2020-05-04T19:35:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/623662192
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
CONTRIBUTOR
The testing design was slightly changed as explained in https://github.com/huggingface/nlp/pull/51 . 
If creating the dummy folder is too confusing it helps to upload everything else to AWS, then run the test and check the INFO when testing on how to create the dummy folder structure.
2020-05-06T13:29:05Z
https://github.com/huggingface/datasets/pull/37#issuecomment-624649759
https://api.github.com/repos/huggingface/datasets/issues/37
MDEyOklzc3VlQ29tbWVudDYyNDY0OTc1OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/624649759/reactions
2020-05-06T13:29:05Z
https://api.github.com/repos/huggingface/datasets/issues/comments/624649759
https://avatars.githubusercontent.com/u/23423619?v=4
https://api.github.com/users/patrickvonplaten/events{/privacy}
https://api.github.com/users/patrickvonplaten/followers
https://api.github.com/users/patrickvonplaten/following{/other_user}
https://api.github.com/users/patrickvonplaten/gists{/gist_id}
https://github.com/patrickvonplaten
patrickvonplaten
MDQ6VXNlcjIzNDIzNjE5
https://api.github.com/users/patrickvonplaten/orgs
https://api.github.com/users/patrickvonplaten/received_events
https://api.github.com/users/patrickvonplaten/repos
https://api.github.com/users/patrickvonplaten/starred{/owner}{/repo}
https://api.github.com/users/patrickvonplaten/subscriptions
User
https://api.github.com/users/patrickvonplaten
public
CONTRIBUTOR
Closing because we can now work on master
2020-05-08T13:48:23Z
https://github.com/huggingface/datasets/pull/37#issuecomment-625824710
https://api.github.com/repos/huggingface/datasets/issues/37
MDEyOklzc3VlQ29tbWVudDYyNTgyNDcxMA==
https://api.github.com/repos/huggingface/datasets/issues/comments/625824710/reactions
2020-05-08T13:48:23Z
https://api.github.com/repos/huggingface/datasets/issues/comments/625824710
https://avatars.githubusercontent.com/u/23423619?v=4
https://api.github.com/users/patrickvonplaten/events{/privacy}
https://api.github.com/users/patrickvonplaten/followers
https://api.github.com/users/patrickvonplaten/following{/other_user}
https://api.github.com/users/patrickvonplaten/gists{/gist_id}
https://github.com/patrickvonplaten
patrickvonplaten
MDQ6VXNlcjIzNDIzNjE5
https://api.github.com/users/patrickvonplaten/orgs
https://api.github.com/users/patrickvonplaten/received_events
https://api.github.com/users/patrickvonplaten/repos
https://api.github.com/users/patrickvonplaten/starred{/owner}{/repo}
https://api.github.com/users/patrickvonplaten/subscriptions
User
https://api.github.com/users/patrickvonplaten
public
## Description

This PR acts as a dashboard to see which datasets are added to the library and work. 

Cicle-ci should always be green so that we can be sure that newly added datasets are functional. 
This PR should not be merged.


## Progress

**For the following datasets the test commands**:
```
RUN_SLOW=1 pytest tests/test_dataset_common.py::DatasetTest::test_load_real_dataset_<your-dataset-name>
```
and 
```
RUN_SLOW=1 pytest tests/test_dataset_common.py::DatasetTest::test_load_dataset_all_configs_<your-dataset-name>
```

**passes**.

- [x] Squad
- [x] Sentiment140
- [x] XNLI
- [x] Crime_and_Punish
- [x] movie_rationales
- [x] ai2_arc
- [x] anli
- [x] event2Mind
- [x] Fquad
- [x] blimp
- [x] empathetic_dialogues
- [x] cosmos_qa
- [x] xquad
- [x] blog_authorship_corpus
- [x] SNLI
- [x] break_data
- [x] SQuAD v2
- [x] cfq
- [x] eraser_multi_rc
- [x] Glue
- [x] Tydiqa
- [x] wiki_qa
- [x] wikitext
- [x] winogrande
- [x] wiqa
- [x] esnli
- [x] civil_comments
- [x] commonsense_qa
- [x] com_qa
- [x] coqa
- [x] wiki_split
- [x] cos_e
- [x] xcopa
- [x] quarel
- [x] quartz
- [x] squad_it
- [x] quoref 
- [x] squad_pt
- [x] cornell_movie_dialog
- [x] SciQ
- [x] Scifact
- [x] hellaswag
- [x] ted_multi (in translate)
- [x] Aeslc (summarization)
- [x] drop
- [x] gap
- [x] hansard
- [x] opinosis
- [x] MLQA
- [x] math_dataset

## How-To-Add a dataset

**Before adding a dataset make sure that your branch is up to date**:
1. `git checkout add_datasets`
2. `git pull`

**Add a dataset via the `convert_dataset.sh` bash script:**  

Running `bash convert_dataset.sh <file/to/tfds/datascript.py>` (*e.g.* `bash convert_dataset.sh ../tensorflow-datasets/tensorflow_datasets/text/movie_rationales.py`) will automatically run all the steps mentioned in **Add a dataset manually** below. 

Make sure that you run `convert_dataset.sh` from the root folder of `nlp`.

The conversion script should work almost always for step 1): "convert dataset script from tfds to nlp format" and 2) "create checksum file" and step 3) "make style".

It can also sometimes automatically run step 4) "create the correct dummy data from tfds", but this will only work if a) there is either no config name or only one config name and b) the `tfds testing/test_data/fake_example` is in the correct form.

Nevertheless, the script should always be run in the beginning until an error occurs to be more efficient. 

If the conversion script does not work or fails at some step, then you can run the steps manually as follows:

**Add a dataset manually** 

Make sure you run all of the following commands from the root of your `nlp` git clone.
Also make sure that you changed to this branch:
```
git checkout add_datasets
```

1) the tfds datascript file should be converted to `nlp` style:

```
python nlp-cli convert --tfds_path <path/to/tensorflow_datasets/text/your_dataset_name>.py --nlp_directory datasets/nlp
```

This will convert the tdfs script and create a folder with the correct name.

2) the checksum file should be added. Use the command:
```
python nlp-cli test datasets/nlp/<your-dataset-folder> --save_checksums --all_configs
```

A checksums.txt file should be created in your folder and the structure should look as follows:

squad/
├── squad.py/
└── urls_checksums/
...........└── checksums.txt

Delete the created `*.lock` file afterward - it should not be uploaded to AWS.

3) run black and isort on your newly added datascript files so that they look nice:

```
make style
```

4) the dummy data should be added. For this it might be useful to take a look into the structure of other examples as shown in the PR here and at `<path/to/tensorflow_datasets/testing/test_data/test_data/fake_examples>` whether the same  data can be used.

5)  the data can be uploaded to AWS using the command
```
aws s3 cp datasets/nlp/<your-dataset-folder> s3://datasets.huggingface.co/nlp/<your-dataset-folder> --recursive
```

6) check whether all works as expected using: 
```
RUN_SLOW=1 pytest tests/test_dataset_common.py::DatasetTest::test_load_real_dataset_<your-dataset-name>
```
and 
```
RUN_SLOW=1 pytest tests/test_dataset_common.py::DatasetTest::test_load_dataset_all_configs_<your-dataset-name>
```

7) push to this PR and rerun the circle ci workflow to check whether circle ci stays green.

8) Edit this commend and tick off your newly added dataset :-) 

## TODO-list

Maybe we can add a TODO-list here for everybody that feels like adding new datasets so that we will not add the same datasets.

Here a link to available datasets: https://docs.google.com/spreadsheets/d/1zOtEqOrnVQwdgkC4nJrTY6d-Av02u0XFzeKAtBM2fUI/edit#gid=0

Patrick:

- [ ] boolq - *weird download link*
- [ ] c4 - *beam dataset*