Fix csv/json/txt cache dir
MEMBER
I think this is the way to go but I’m afraid this might be a little slow. I was thinking that we could use a high quality very fast non crypto hash like xxhash for these stuff (hashing data files)
2020-07-28T16:44:56Z
https://github.com/huggingface/datasets/pull/451#issuecomment-665150150
https://api.github.com/repos/huggingface/datasets/issues/451
MDEyOklzc3VlQ29tbWVudDY2NTE1MDE1MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/665150150/reactions
2020-07-28T16:44:56Z
https://api.github.com/repos/huggingface/datasets/issues/comments/665150150
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
MEMBER
Yep good idea, I'll take a look
2020-07-28T16:46:13Z
https://github.com/huggingface/datasets/pull/451#issuecomment-665150814
https://api.github.com/repos/huggingface/datasets/issues/451
MDEyOklzc3VlQ29tbWVudDY2NTE1MDgxNA==
https://api.github.com/repos/huggingface/datasets/issues/comments/665150814/reactions
2020-07-28T16:46:13Z
https://api.github.com/repos/huggingface/datasets/issues/comments/665150814
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
I tested the hashing speed [here](https://colab.research.google.com/drive/1hlhP84kLIHmOzMRQN1h8x10hKWpXXyud?usp=sharing).
I was able to get 8x speed with `xxhashlib` (42ms vs 345ms for 100MiB of data).
What do you think @thomwolf ?
2020-07-29T08:31:59Z
https://github.com/huggingface/datasets/pull/451#issuecomment-665520137
https://api.github.com/repos/huggingface/datasets/issues/451
MDEyOklzc3VlQ29tbWVudDY2NTUyMDEzNw==
https://api.github.com/repos/huggingface/datasets/issues/comments/665520137/reactions
2020-07-29T08:31:59Z
https://api.github.com/repos/huggingface/datasets/issues/comments/665520137
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
I added xxhash and some tests
2020-07-29T11:16:23Z
https://github.com/huggingface/datasets/pull/451#issuecomment-665604002
https://api.github.com/repos/huggingface/datasets/issues/451
MDEyOklzc3VlQ29tbWVudDY2NTYwNDAwMg==
https://api.github.com/repos/huggingface/datasets/issues/comments/665604002/reactions
2020-07-29T11:16:35Z
https://api.github.com/repos/huggingface/datasets/issues/comments/665604002
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
The cache dir for csv/json/txt datasets was always the same. This is an issue because it should be different depending on the data files provided by the user.

To fix that, I added a line that use the hash of the data files provided by the user to define the cache dir.

This should fix #444 