Auto generate dummy data
MEMBER
I took your comments into account.
Also now after compressing the dummy_data.zip file it runs a dummy data test (=make sure each split has at least 1 example using the dummy data)
2020-11-25T16:23:17Z
https://github.com/huggingface/datasets/pull/884#issuecomment-733809543
https://api.github.com/repos/huggingface/datasets/issues/884
MDEyOklzc3VlQ29tbWVudDczMzgwOTU0Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/733809543/reactions
2020-11-25T16:23:17Z
https://api.github.com/repos/huggingface/datasets/issues/comments/733809543
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
I just tested the tool with some datasets and found out that it's not working for datasets that download files using `download_and_extract(file_url)` (where file_url is a `str`). That's because in that case the dummy_data.zip is not a folder but a single zipped file.

I think we have to fix that or we can have unexpected behavior when a scripts calls `download_and_extract(file_url)` several times, since it would always point to the same dummy data file.

So I decided to change that to have a folder containing the dummy files instead but it breaks around 90 tests so I need to update 90 dummy data files to follow this scheme. I'll probably fix them tomorrow morning.

What do you guys think ? Also cc @patrickvonplaten to make sure I understand things correctly
2020-11-25T17:57:37Z
https://github.com/huggingface/datasets/pull/884#issuecomment-733861406
https://api.github.com/repos/huggingface/datasets/issues/884
MDEyOklzc3VlQ29tbWVudDczMzg2MTQwNg==
https://api.github.com/repos/huggingface/datasets/issues/comments/733861406/reactions
2020-11-25T17:57:37Z
https://api.github.com/repos/huggingface/datasets/issues/comments/733861406
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Ok I changed to use the dummy_data.zip content to be a folder even for single url calls to `dl_manager.download_and_extract`. Therefore the automatic dummy data generation tool works for most datasets now.

To avoid having to change all the old dummy_data.zip files I added backward compatiblity. 

The only test failing is `tests/test_dataset_common.py::RemoteDatasetTest::test_load_dataset_xcopa`
It is expected to fail since I had modify its dummy data structure that was wrong. It was causing issue with backward compatibility. It will be fixed as soon as this PR is merged
2020-11-26T11:29:31Z
https://github.com/huggingface/datasets/pull/884#issuecomment-734244569
https://api.github.com/repos/huggingface/datasets/issues/884
MDEyOklzc3VlQ29tbWVudDczNDI0NDU2OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/734244569/reactions
2020-11-26T11:40:35Z
https://api.github.com/repos/huggingface/datasets/issues/comments/734244569
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
When adding a new dataset to the library, dummy data creation can take some time.
To make things easier I added a command line tool that automatically generates dummy data when possible.

The tool only supports certain data files types: txt, csv, tsv, jsonl, json and xml.

Here are some examples:
```
python datasets-cli dummy_data ./datasets/snli --auto_generate
python datasets-cli dummy_data ./datasets/squad --auto_generate --json_field data
python datasets-cli dummy_data ./datasets/iwslt2017 --auto_generate --xml_tag seg --match_text_files "train*" --n_lines 15
# --xml_tag seg => each sample corresponds to a "seg" tag in the xml tree
# --match_text_files "train*" =>  also match text files that don't have a proper text file extension (no suffix like ".txt" for example)
# --n_lines 15 => some text files have headers so we have to use at least 15 lines
```

and here is the command usage:

```
usage: datasets-cli <command> [<args>] dummy_data [-h] [--auto_generate]
                                  [--n_lines N_LINES]
                                  [--json_field JSON_FIELD]
                                  [--xml_tag XML_TAG]
                                  [--match_text_files MATCH_TEXT_FILES]
                                  [--keep_uncompressed]
                                  [--cache_dir CACHE_DIR]
                                  path_to_dataset

positional arguments:
path_to_dataset       Path to the dataset (example: ./datasets/squad)

optional arguments:
-h, --help            show this help message and exit
--auto_generate       Try to automatically generate dummy data
--n_lines N_LINES     Number of lines or samples to keep when auto-
                                                generating dummy data
--json_field JSON_FIELD
                                                Optional, json field to read the data from when auto-
                                                generating dummy data. In the json data files, this
                                                field must point to a list of samples as json objects
                                                (ex: the 'data' field for squad-like files)
--xml_tag XML_TAG     Optional, xml tag name of the samples inside the xml
                                                files when auto-generating dummy data.
--match_text_files MATCH_TEXT_FILES
                                                Optional, a comma separated list of file patterns that
                                                looks for line-by-line text files other than *.txt or
                                                *.csv. Example: --match_text_files *.label
--keep_uncompressed   Don't compress the dummy data folders when auto-
                                                generating dummy data. Useful for debugging for to do
                                                manual adjustements before compressing.
--cache_dir CACHE_DIR
                                                Cache directory to download and cache files when auto-
                                                generating dummy data
```

The command generates all the necessary `dummy_data.zip` files (one per config).

How it works:
- it runs the split_generators() method of the dataset script to download the original data files
- when downloading it records a mapping between the downloaded files and the corresponding expected dummy data files paths
- then for each data file it creates the dummy data file keeping only the first samples (the strategy depends on the type of file)
- finally it compresses the dummy data folders into dummy_zip files ready for dataset tests

Let me know if that makes sense or if you have ideas to improve this tool !

I also added a unit test.