Fix text delimiter
MEMBER
Which OS are you using ?@abhi1nandy2
2020-09-22T14:46:17Z
https://github.com/huggingface/datasets/pull/631#issuecomment-696769819
https://api.github.com/repos/huggingface/datasets/issues/631
MDEyOklzc3VlQ29tbWVudDY5Njc2OTgxOQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/696769819/reactions
2020-09-22T14:47:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/696769819
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
> Which OS are you using ?

PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
VERSION_CODENAME=stretch
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
2020-09-22T14:47:21Z
https://github.com/huggingface/datasets/pull/631#issuecomment-696770452
https://api.github.com/repos/huggingface/datasets/issues/631
MDEyOklzc3VlQ29tbWVudDY5Njc3MDQ1Mg==
https://api.github.com/repos/huggingface/datasets/issues/comments/696770452/reactions
2020-09-22T14:47:21Z
https://api.github.com/repos/huggingface/datasets/issues/comments/696770452
https://avatars.githubusercontent.com/u/19462637?v=4
https://api.github.com/users/abhi1nandy2/events{/privacy}
https://api.github.com/users/abhi1nandy2/followers
https://api.github.com/users/abhi1nandy2/following{/other_user}
https://api.github.com/users/abhi1nandy2/gists{/gist_id}
https://github.com/abhi1nandy2
abhi1nandy2
MDQ6VXNlcjE5NDYyNjM3
https://api.github.com/users/abhi1nandy2/orgs
https://api.github.com/users/abhi1nandy2/received_events
https://api.github.com/users/abhi1nandy2/repos
https://api.github.com/users/abhi1nandy2/starred{/owner}{/repo}
https://api.github.com/users/abhi1nandy2/subscriptions
User
https://api.github.com/users/abhi1nandy2
public
MEMBER
Do you mind sharing the data you used (or part of it), so I can try to reproduce ?
Or at least some info about the text file you're using ? (size, n of lines, encoding)
2020-09-22T14:49:03Z
https://github.com/huggingface/datasets/pull/631#issuecomment-696771509
https://api.github.com/repos/huggingface/datasets/issues/631
MDEyOklzc3VlQ29tbWVudDY5Njc3MTUwOQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/696771509/reactions
2020-09-22T14:49:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/696771509
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Lot of data, difficult to share. There are 46 shards, each having about 256000 lines. using `file` command gives this - `ASCII text, with very long lines`.
2020-09-22T14:59:18Z
https://github.com/huggingface/datasets/pull/631#issuecomment-696778109
https://api.github.com/repos/huggingface/datasets/issues/631
MDEyOklzc3VlQ29tbWVudDY5Njc3ODEwOQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/696778109/reactions
2020-09-22T14:59:18Z
https://api.github.com/repos/huggingface/datasets/issues/comments/696778109
https://avatars.githubusercontent.com/u/19462637?v=4
https://api.github.com/users/abhi1nandy2/events{/privacy}
https://api.github.com/users/abhi1nandy2/followers
https://api.github.com/users/abhi1nandy2/following{/other_user}
https://api.github.com/users/abhi1nandy2/gists{/gist_id}
https://github.com/abhi1nandy2
abhi1nandy2
MDQ6VXNlcjE5NDYyNjM3
https://api.github.com/users/abhi1nandy2/orgs
https://api.github.com/users/abhi1nandy2/received_events
https://api.github.com/users/abhi1nandy2/repos
https://api.github.com/users/abhi1nandy2/starred{/owner}{/repo}
https://api.github.com/users/abhi1nandy2/subscriptions
User
https://api.github.com/users/abhi1nandy2
public
MEMBER
Ok I see, no problem :) 
I'll see what I can do

Could you just test with one single dummy text file with a few lines to see if you're having the issue ?
Also which version of `datasets` do you have ?
2020-09-22T15:03:05Z
https://github.com/huggingface/datasets/pull/631#issuecomment-696780496
https://api.github.com/repos/huggingface/datasets/issues/631
MDEyOklzc3VlQ29tbWVudDY5Njc4MDQ5Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/696780496/reactions
2020-09-22T15:03:05Z
https://api.github.com/repos/huggingface/datasets/issues/comments/696780496
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
I changed the delimiter in the `text` dataset script.
It should fix the `pyarrow.lib.ArrowInvalid: CSV parse error` from #622 

I changed the delimiter to an unused ascii character that is not present in text files : `\b`