[Breaking] Switch text loading to multi-threaded PyArrow loading
MEMBER
Awesome !
Also I was wondering if we should try to make the hashing of the `data_files` faster (it is used to build the cache directory of datasets like `text` or `json`). Right now it reads each file and hashes all of its data. We could simply hash the path and some metadata including the `time last modified` tag no ? Apparently we can get this tag with `os.path.getmtime(path)`
2020-08-31T18:15:05Z
https://github.com/huggingface/datasets/pull/548#issuecomment-683944191
https://api.github.com/repos/huggingface/datasets/issues/548
MDEyOklzc3VlQ29tbWVudDY4Mzk0NDE5MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/683944191/reactions
2020-08-31T18:16:20Z
https://api.github.com/repos/huggingface/datasets/issues/comments/683944191
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
I just rebased from master to include the hashing changes from #573 
2020-09-04T14:58:11Z
https://github.com/huggingface/datasets/pull/548#issuecomment-687201368
https://api.github.com/repos/huggingface/datasets/issues/548
MDEyOklzc3VlQ29tbWVudDY4NzIwMTM2OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/687201368/reactions
2020-09-04T14:58:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/687201368
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
I think this is ready to merge, no?
2020-09-04T18:00:18Z
https://github.com/huggingface/datasets/pull/548#issuecomment-687298398
https://api.github.com/repos/huggingface/datasets/issues/548
MDEyOklzc3VlQ29tbWVudDY4NzI5ODM5OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/687298398/reactions
2020-09-04T18:00:18Z
https://api.github.com/repos/huggingface/datasets/issues/comments/687298398
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
MEMBER
Indeed it's ready to merge :)
2020-09-08T07:44:37Z
https://github.com/huggingface/datasets/pull/548#issuecomment-688685160
https://api.github.com/repos/huggingface/datasets/issues/548
MDEyOklzc3VlQ29tbWVudDY4ODY4NTE2MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/688685160/reactions
2020-09-08T07:44:37Z
https://api.github.com/repos/huggingface/datasets/issues/comments/688685160
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Ok added the breaking change info and we can merge indeed.

2020-09-08T10:19:52Z
https://github.com/huggingface/datasets/pull/548#issuecomment-688772848
https://api.github.com/repos/huggingface/datasets/issues/548
MDEyOklzc3VlQ29tbWVudDY4ODc3Mjg0OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/688772848/reactions
2020-09-08T10:19:52Z
https://api.github.com/repos/huggingface/datasets/issues/comments/688772848
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
Test if we can get better performances for large-scale text datasets by using multi-threaded text file loading based on Apache Arrow multi-threaded CSV loader.

If it works ok, it would fix #546.

**Breaking change**:
The text lines now do not include final line-breaks anymore.