Use python read for text dataset
MEMBER
One thing though, could we try to read the files in parallel?
2020-10-05T09:59:28Z
https://github.com/huggingface/datasets/pull/715#issuecomment-703530532
https://api.github.com/repos/huggingface/datasets/issues/715
MDEyOklzc3VlQ29tbWVudDcwMzUzMDUzMg==
https://api.github.com/repos/huggingface/datasets/issues/comments/703530532/reactions
2020-10-05T09:59:28Z
https://api.github.com/repos/huggingface/datasets/issues/comments/703530532
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
MEMBER
We could but I'm not sure this would help a lot since the bottleneck is the drive IO if the files are big enough.
It could make sense for very small files.
2020-10-05T10:07:22Z
https://github.com/huggingface/datasets/pull/715#issuecomment-703534612
https://api.github.com/repos/huggingface/datasets/issues/715
MDEyOklzc3VlQ29tbWVudDcwMzUzNDYxMg==
https://api.github.com/repos/huggingface/datasets/issues/comments/703534612/reactions
2020-10-05T10:07:22Z
https://api.github.com/repos/huggingface/datasets/issues/comments/703534612
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Looks like windows is not a big fan of this approach
I'm working on a fix
2020-10-05T10:08:27Z
https://github.com/huggingface/datasets/pull/715#issuecomment-703535119
https://api.github.com/repos/huggingface/datasets/issues/715
MDEyOklzc3VlQ29tbWVudDcwMzUzNTExOQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/703535119/reactions
2020-10-05T10:08:27Z
https://api.github.com/repos/huggingface/datasets/issues/comments/703535119
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
I remember issue https://github.com/huggingface/datasets/issues/546 where this was kinda requested (but maybe IO would bottleneck). What do you think?
2020-10-05T10:20:08Z
https://github.com/huggingface/datasets/pull/715#issuecomment-703540766
https://api.github.com/repos/huggingface/datasets/issues/715
MDEyOklzc3VlQ29tbWVudDcwMzU0MDc2Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/703540766/reactions
2020-10-05T10:20:08Z
https://api.github.com/repos/huggingface/datasets/issues/comments/703540766
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
MEMBER
I think it's worth testing multiprocessing. It could also be something we add to our speed benchmarks
2020-10-05T10:28:44Z
https://github.com/huggingface/datasets/pull/715#issuecomment-703544572
https://api.github.com/repos/huggingface/datasets/issues/715
MDEyOklzc3VlQ29tbWVudDcwMzU0NDU3Mg==
https://api.github.com/repos/huggingface/datasets/issues/comments/703544572/reactions
2020-10-05T10:28:44Z
https://api.github.com/repos/huggingface/datasets/issues/comments/703544572
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
> I remember issue #546 where this was kinda requested (but maybe IO would bottleneck). What do you think?

It still would be interesting I think, especially in scenarios where IO is less of an issue (SSDs particularly) and where there are many smaller files. Wrapping this function in a `pool.map` is perhaps an easy thing to try. 
2020-10-05T11:23:57Z
https://github.com/huggingface/datasets/pull/715#issuecomment-703569649
https://api.github.com/repos/huggingface/datasets/issues/715
MDEyOklzc3VlQ29tbWVudDcwMzU2OTY0OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/703569649/reactions
2020-10-05T11:23:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/703569649
https://avatars.githubusercontent.com/u/2779410?v=4
https://api.github.com/users/BramVanroy/events{/privacy}
https://api.github.com/users/BramVanroy/followers
https://api.github.com/users/BramVanroy/following{/other_user}
https://api.github.com/users/BramVanroy/gists{/gist_id}
https://github.com/BramVanroy
BramVanroy
MDQ6VXNlcjI3Nzk0MTA=
https://api.github.com/users/BramVanroy/orgs
https://api.github.com/users/BramVanroy/received_events
https://api.github.com/users/BramVanroy/repos
https://api.github.com/users/BramVanroy/starred{/owner}{/repo}
https://api.github.com/users/BramVanroy/subscriptions
User
https://api.github.com/users/BramVanroy
public
MEMBER
Merging this one for now for the patch release
2020-10-05T13:12:48Z
https://github.com/huggingface/datasets/pull/715#issuecomment-703622728
https://api.github.com/repos/huggingface/datasets/issues/715
MDEyOklzc3VlQ29tbWVudDcwMzYyMjcyOA==
https://api.github.com/repos/huggingface/datasets/issues/comments/703622728/reactions
2020-10-05T13:12:48Z
https://api.github.com/repos/huggingface/datasets/issues/comments/703622728
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
As mentioned in #622 the pandas reader used for text dataset doesn't work properly when there are \r characters in the text file.

Instead I switched to pure python using `open` and `read`.
From my benchmark on a 100MB text file, it's the same speed as the previous pandas reader.