Deterministic dataset loading
MEMBER
Yes good point !
2020-06-05T09:06:32Z
https://github.com/huggingface/datasets/issues/240#issuecomment-639356179
https://api.github.com/repos/huggingface/datasets/issues/240
MDEyOklzc3VlQ29tbWVudDYzOTM1NjE3OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/639356179/reactions
2020-06-05T09:06:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/639356179
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
I think using `sorted(glob.glob())` would actually solve this problem. Can you think of other reasons why dataset loading might not be deterministic? @mariamabarham @yjernite @lhoestq @thomwolf . 

I can do a sweep through the dataset scripts and fix the glob.glob() if you guys are ok with it
2020-06-05T09:06:40Z
https://github.com/huggingface/datasets/issues/240#issuecomment-639356249
https://api.github.com/repos/huggingface/datasets/issues/240
MDEyOklzc3VlQ29tbWVudDYzOTM1NjI0OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/639356249/reactions
2020-06-05T09:06:40Z
https://api.github.com/repos/huggingface/datasets/issues/comments/639356249
https://avatars.githubusercontent.com/u/23423619?v=4
https://api.github.com/users/patrickvonplaten/events{/privacy}
https://api.github.com/users/patrickvonplaten/followers
https://api.github.com/users/patrickvonplaten/following{/other_user}
https://api.github.com/users/patrickvonplaten/gists{/gist_id}
https://github.com/patrickvonplaten
patrickvonplaten
MDQ6VXNlcjIzNDIzNjE5
https://api.github.com/users/patrickvonplaten/orgs
https://api.github.com/users/patrickvonplaten/received_events
https://api.github.com/users/patrickvonplaten/repos
https://api.github.com/users/patrickvonplaten/starred{/owner}{/repo}
https://api.github.com/users/patrickvonplaten/subscriptions
User
https://api.github.com/users/patrickvonplaten
public
MEMBER
I'm pretty sure it would solve the problem too.

The only other dataset that is not deterministic right now is `blog_authorship_corpus` (see #215) but this is a problem related to string encodings.
2020-06-05T09:13:45Z
https://github.com/huggingface/datasets/issues/240#issuecomment-639359584
https://api.github.com/repos/huggingface/datasets/issues/240
MDEyOklzc3VlQ29tbWVudDYzOTM1OTU4NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/639359584/reactions
2020-06-05T09:13:45Z
https://api.github.com/repos/huggingface/datasets/issues/comments/639359584
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
I think we should do the same also for `os.list_dir`
2020-06-05T09:14:58Z
https://github.com/huggingface/datasets/issues/240#issuecomment-639360176
https://api.github.com/repos/huggingface/datasets/issues/240
MDEyOklzc3VlQ29tbWVudDYzOTM2MDE3Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/639360176/reactions
2020-06-05T09:14:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/639360176
https://avatars.githubusercontent.com/u/38249783?v=4
https://api.github.com/users/mariamabarham/events{/privacy}
https://api.github.com/users/mariamabarham/followers
https://api.github.com/users/mariamabarham/following{/other_user}
https://api.github.com/users/mariamabarham/gists{/gist_id}
https://github.com/mariamabarham
mariamabarham
MDQ6VXNlcjM4MjQ5Nzgz
https://api.github.com/users/mariamabarham/orgs
https://api.github.com/users/mariamabarham/received_events
https://api.github.com/users/mariamabarham/repos
https://api.github.com/users/mariamabarham/starred{/owner}{/repo}
https://api.github.com/users/mariamabarham/subscriptions
User
https://api.github.com/users/mariamabarham
public
When calling:
```python 
import nlp
dataset = nlp.load_dataset("trivia_qa", split="validation[:1%]")
```

the resulting dataset is not deterministic over different google colabs. 
After talking to @thomwolf, I suspect the reason to be the use of `glob.glob` in line:

https://github.com/huggingface/nlp/blob/2e0a8639a79b1abc848cff5c669094d40bba0f63/datasets/trivia_qa/trivia_qa.py#L180

which seems to return an ordering of files that depends on the filesystem:
https://stackoverflow.com/questions/6773584/how-is-pythons-glob-glob-ordered

I think we should go through all the dataset scripts and make sure to have deterministic behavior.

A simple solution for `glob.glob()` would be to just replace it with `sorted(glob.glob())` to have everything sorted by name. 

What do you think @lhoestq?