Add checksums
CONTRIBUTOR
Looks good to me :-) 

Just would prefer to get rid of the `_DYNAMICALLY_IMPORTED_MODULE` attribute and replace it by a `get_imported_module()` function. Maybe there is something I'm not seeing here though - what do you think? 
2020-04-30T07:12:32Z
https://github.com/huggingface/datasets/pull/24#issuecomment-621656344
https://api.github.com/repos/huggingface/datasets/issues/24
MDEyOklzc3VlQ29tbWVudDYyMTY1NjM0NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/621656344/reactions
2020-04-30T07:12:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/621656344
https://avatars.githubusercontent.com/u/23423619?v=4
https://api.github.com/users/patrickvonplaten/events{/privacy}
https://api.github.com/users/patrickvonplaten/followers
https://api.github.com/users/patrickvonplaten/following{/other_user}
https://api.github.com/users/patrickvonplaten/gists{/gist_id}
https://github.com/patrickvonplaten
patrickvonplaten
MDQ6VXNlcjIzNDIzNjE5
https://api.github.com/users/patrickvonplaten/orgs
https://api.github.com/users/patrickvonplaten/received_events
https://api.github.com/users/patrickvonplaten/repos
https://api.github.com/users/patrickvonplaten/starred{/owner}{/repo}
https://api.github.com/users/patrickvonplaten/subscriptions
User
https://api.github.com/users/patrickvonplaten
public
MEMBER
> * I'm not sure I understand the general organization of checksums. I see we have a checksum folder with potentially several checksum files but I also see that checksum files can potentially contain several checksums. Could you explain a bit more how this is organized?

It should look like this:
squad/
├── squad.py/
└── urls_checksums/
...........└── checksums.txt

In checksums.txt, the format is one line per (url, size, checksum)

I don't have a strong opinion between `urls_checksums/checksums.txt` or directly `checksums.txt` (not inside the `urls_checksums` folder), let me know what you think.


> * Also regarding your comment on checksum files for "canonical" datasets. I understand we can just create these with `nlp-cli test` and then upload them manually to our S3, right?

Yes you're right
2020-04-30T08:53:26Z
https://github.com/huggingface/datasets/pull/24#issuecomment-621704068
https://api.github.com/repos/huggingface/datasets/issues/24
MDEyOklzc3VlQ29tbWVudDYyMTcwNDA2OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/621704068/reactions
2020-04-30T11:48:50Z
https://api.github.com/repos/huggingface/datasets/issues/comments/621704068
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Update of the commands:

- nlp-cli test \<dataset\> : Run download_and_prepare and verify checksums
  * --name \<name\> : run only for the name
  * --all_configs : run for all configs
  * --save_checksums : instead of verifying checksums, compute and save them
  * --ignore_checksums : don't do checksums verification

- nlp-cli upload \<dataset_folder\> : Upload a dataset
  * --upload_checksums : compute and upload checksums for uploaded files

TODO:
- don't overwrite checksums files on S3, to let the user upload a dataset in several steps if needed

Question:
- One idea from @patrickvonplaten : shall we upload checksums everytime we upload files ? (and therefore remove the upload_checksums parameter)
2020-04-30T12:39:43Z
https://github.com/huggingface/datasets/pull/24#issuecomment-621807985
https://api.github.com/repos/huggingface/datasets/issues/24
MDEyOklzc3VlQ29tbWVudDYyMTgwNzk4NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/621807985/reactions
2020-04-30T12:39:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/621807985
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Ok, ready to merge, then @lhoestq ?
2020-04-30T19:19:15Z
https://github.com/huggingface/datasets/pull/24#issuecomment-622052653
https://api.github.com/repos/huggingface/datasets/issues/24
MDEyOklzc3VlQ29tbWVudDYyMjA1MjY1Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/622052653/reactions
2020-04-30T19:19:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/622052653
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
MEMBER
Yep :)
2020-04-30T19:49:31Z
https://github.com/huggingface/datasets/pull/24#issuecomment-622069889
https://api.github.com/repos/huggingface/datasets/issues/24
MDEyOklzc3VlQ29tbWVudDYyMjA2OTg4OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/622069889/reactions
2020-04-30T19:49:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/622069889
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
### Checksums files

They are stored next to the dataset script in urls_checksums/checksums.txt.
They are used to check the integrity of the datasets downloaded files.
I kept the same format as tensorflow-datasets.
There is one checksums file for all configs.

### Load a dataset

When you do `load("squad")`, it will also download the checksums file and put it next to the script in nlp/datasets/hash/urls_checksums/checksums.txt.
It also verifies that the downloaded files checksums match the expected ones.

You can ignore checksum tests with `load("squad", ignore_checksums=True)` (under the hood it just adds `ignore_checksums=True` in the `DownloadConfig`)

### Test a dataset

There is a new command `nlp-cli test squad` that runs `download_and_prepare` to see if it runs ok, and that verifies that all the checksums match. Allowed arguments are `--name`, `--all_configs`, `--ignore_checksums` and `--register_checksums`.

### Register checksums

1. If the dataset has external dataset files

The command `nlp-cli test squad --register_checksums --all_configs` runs `download_and_prepare` on all configs to see if it runs ok, and it creates the checksums file.
You can also register one config at a time using `--name` instead ; the checksums file will be completed and not overwritten.

If the script is a local script, the checksum file is moved to urls_checksums/checksums.txt next to the local script, to enable the user to upload both the script and the checksums file afterwards with `nlp-cli upload squad`.

2. If the dataset files are all inside the directory of the dataset script

The user can directly do `nlp-cli upload squad --register_checksums`, as there is no need to download anything.
In this case however, all the dataset must be uploaded at once.

--

PS : it doesn't allow to register checksums for canonical datasets, the file has to be added manually on S3 for now (I guess ?)

Also I feel like we must be sure that this processes would not constrain too much any user from uploading its dataset.
Let me know what you think :)