load_dataset with 'csv' is not working. while the same file is loading with 'text' mode or with pandas
MEMBER
Which version of pyarrow do you have ? Could you try to update pyarrow and try again ?
2020-11-11T12:54:32Z
https://github.com/huggingface/datasets/issues/836#issuecomment-725407423
https://api.github.com/repos/huggingface/datasets/issues/836
MDEyOklzc3VlQ29tbWVudDcyNTQwNzQyMw==
https://api.github.com/repos/huggingface/datasets/issues/comments/725407423/reactions
2020-11-11T12:54:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/725407423
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Thanks for the fast response. I have the latest version '2.0.0' (I tried to update)
I am working with Python 3.8.5
2020-11-11T12:59:14Z
https://github.com/huggingface/datasets/issues/836#issuecomment-725409507
https://api.github.com/repos/huggingface/datasets/issues/836
MDEyOklzc3VlQ29tbWVudDcyNTQwOTUwNw==
https://api.github.com/repos/huggingface/datasets/issues/comments/725409507/reactions
2020-11-11T12:59:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/725409507
https://avatars.githubusercontent.com/u/8919490?v=4
https://api.github.com/users/randubin/events{/privacy}
https://api.github.com/users/randubin/followers
https://api.github.com/users/randubin/following{/other_user}
https://api.github.com/users/randubin/gists{/gist_id}
https://github.com/randubin
randubin
MDQ6VXNlcjg5MTk0OTA=
https://api.github.com/users/randubin/orgs
https://api.github.com/users/randubin/received_events
https://api.github.com/users/randubin/repos
https://api.github.com/users/randubin/starred{/owner}{/repo}
https://api.github.com/users/randubin/subscriptions
User
https://api.github.com/users/randubin
public
NONE
I think that the issue is similar to this one:https://issues.apache.org/jira/browse/ARROW-9612
The problem is in arrow when the column data contains long strings.
Any ideas on how to bypass this?
2020-11-11T17:23:38Z
https://github.com/huggingface/datasets/issues/836#issuecomment-725552966
https://api.github.com/repos/huggingface/datasets/issues/836
MDEyOklzc3VlQ29tbWVudDcyNTU1Mjk2Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/725552966/reactions
2020-11-11T17:23:38Z
https://api.github.com/repos/huggingface/datasets/issues/comments/725552966
https://avatars.githubusercontent.com/u/8919490?v=4
https://api.github.com/users/randubin/events{/privacy}
https://api.github.com/users/randubin/followers
https://api.github.com/users/randubin/following{/other_user}
https://api.github.com/users/randubin/gists{/gist_id}
https://github.com/randubin
randubin
MDQ6VXNlcjg5MTk0OTA=
https://api.github.com/users/randubin/orgs
https://api.github.com/users/randubin/received_events
https://api.github.com/users/randubin/repos
https://api.github.com/users/randubin/starred{/owner}{/repo}
https://api.github.com/users/randubin/subscriptions
User
https://api.github.com/users/randubin
public
MEMBER
We should expose the [`block_size` argument](https://arrow.apache.org/docs/python/generated/pyarrow.csv.ReadOptions.html#pyarrow.csv.ReadOptions) of Apache Arrow csv `ReadOptions` in the [script](https://github.com/huggingface/datasets/blob/master/datasets/csv/csv.py).


In the meantime you can specify yourself the `ReadOptions` config like this:
```python
import pyarrow.csv as pac   # PyArrow is installed with `datasets`

read_options = pac.ReadOptions(block_size=1e9)  # try to find the right value for your use-case
dataset = load_dataset('csv', data_files=files, read_options=read_options)
```

2020-11-11T17:37:34Z
https://github.com/huggingface/datasets/issues/836#issuecomment-725560447
https://api.github.com/repos/huggingface/datasets/issues/836
MDEyOklzc3VlQ29tbWVudDcyNTU2MDQ0Nw==
https://api.github.com/repos/huggingface/datasets/issues/comments/725560447/reactions
2020-11-11T17:37:34Z
https://api.github.com/repos/huggingface/datasets/issues/comments/725560447
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
NONE
This did help to load the data. But the problem now is that I get:
ArrowInvalid: CSV parse error: Expected 5 columns, got 187

It seems that this change the parsing so I changed the table to tab-separated and tried to load it directly from pyarrow
But I got a similar error, again it loaded fine in pandas so I am not sure what to do.




2020-11-11T19:43:12Z
https://github.com/huggingface/datasets/issues/836#issuecomment-725623454
https://api.github.com/repos/huggingface/datasets/issues/836
MDEyOklzc3VlQ29tbWVudDcyNTYyMzQ1NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/725623454/reactions
2020-11-11T19:43:12Z
https://api.github.com/repos/huggingface/datasets/issues/comments/725623454
https://avatars.githubusercontent.com/u/8919490?v=4
https://api.github.com/users/randubin/events{/privacy}
https://api.github.com/users/randubin/followers
https://api.github.com/users/randubin/following{/other_user}
https://api.github.com/users/randubin/gists{/gist_id}
https://github.com/randubin
randubin
MDQ6VXNlcjg5MTk0OTA=
https://api.github.com/users/randubin/orgs
https://api.github.com/users/randubin/received_events
https://api.github.com/users/randubin/repos
https://api.github.com/users/randubin/starred{/owner}{/repo}
https://api.github.com/users/randubin/subscriptions
User
https://api.github.com/users/randubin
public
NONE
Got almost the same error loading a ~5GB TSV file, first got the same error as OP, then tried giving it my own ReadOptions and also got the same CSV parse error.
2020-11-13T06:55:52Z
https://github.com/huggingface/datasets/issues/836#issuecomment-726561488
https://api.github.com/repos/huggingface/datasets/issues/836
MDEyOklzc3VlQ29tbWVudDcyNjU2MTQ4OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/726561488/reactions
2020-11-13T06:55:52Z
https://api.github.com/repos/huggingface/datasets/issues/comments/726561488
https://avatars.githubusercontent.com/u/18243483?v=4
https://api.github.com/users/Justin1904/events{/privacy}
https://api.github.com/users/Justin1904/followers
https://api.github.com/users/Justin1904/following{/other_user}
https://api.github.com/users/Justin1904/gists{/gist_id}
https://github.com/Justin1904
Justin1904
MDQ6VXNlcjE4MjQzNDgz
https://api.github.com/users/Justin1904/orgs
https://api.github.com/users/Justin1904/received_events
https://api.github.com/users/Justin1904/repos
https://api.github.com/users/Justin1904/starred{/owner}{/repo}
https://api.github.com/users/Justin1904/subscriptions
User
https://api.github.com/users/Justin1904
public
NONE
> We should expose the [`block_size` argument](https://arrow.apache.org/docs/python/generated/pyarrow.csv.ReadOptions.html#pyarrow.csv.ReadOptions) of Apache Arrow csv `ReadOptions` in the [script](https://github.com/huggingface/datasets/blob/master/datasets/csv/csv.py).
> 
> In the meantime you can specify yourself the `ReadOptions` config like this:
> 
> ```python
> import pyarrow.csv as pac   # PyArrow is installed with `datasets`
> 
> read_options = pac.ReadOptions(block_size=1e9)  # try to find the right value for your use-case
> dataset = load_dataset('csv', data_files=files, read_options=read_options)
> ```

This did not work for me, I got
`TypeError: __init__() got an unexpected keyword argument 'read_options'`
2021-11-19T17:46:08Z
https://github.com/huggingface/datasets/issues/836#issuecomment-974278502
https://api.github.com/repos/huggingface/datasets/issues/836
IC_kwDODunzps46Ek9m
https://api.github.com/repos/huggingface/datasets/issues/comments/974278502/reactions
2021-11-19T17:46:08Z
https://api.github.com/repos/huggingface/datasets/issues/comments/974278502
https://avatars.githubusercontent.com/u/86256217?v=4
https://api.github.com/users/wzkariampuzha/events{/privacy}
https://api.github.com/users/wzkariampuzha/followers
https://api.github.com/users/wzkariampuzha/following{/other_user}
https://api.github.com/users/wzkariampuzha/gists{/gist_id}
https://github.com/wzkariampuzha
wzkariampuzha
MDQ6VXNlcjg2MjU2MjE3
https://api.github.com/users/wzkariampuzha/orgs
https://api.github.com/users/wzkariampuzha/received_events
https://api.github.com/users/wzkariampuzha/repos
https://api.github.com/users/wzkariampuzha/starred{/owner}{/repo}
https://api.github.com/users/wzkariampuzha/subscriptions
User
https://api.github.com/users/wzkariampuzha
public
MEMBER
Hi ! Yes because of issues with PyArrow's CSV reader we switched to using the Pandas CSV reader. In particular the `read_options` argument is not supported anymore, but you can pass any parameter of Pandas' `read_csv` function (see the list here in [Pandas documentation](https://pandas.pydata.org/docs/reference/api/pandas.read_csv.html))
2021-11-24T16:59:19Z
https://github.com/huggingface/datasets/issues/836#issuecomment-978060640
https://api.github.com/repos/huggingface/datasets/issues/836
IC_kwDODunzps46TAVg
https://api.github.com/repos/huggingface/datasets/issues/comments/978060640/reactions
2021-11-24T16:59:19Z
https://api.github.com/repos/huggingface/datasets/issues/comments/978060640
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
Hi All
I am trying to load a custom dataset  and I am trying to load a single file to make sure the file is loading correctly:
dataset = load_dataset('csv', data_files=files)
When I run it I get:

Downloading and preparing dataset csv/default-35575a1051604c88 (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) tocache/huggingface/datasets/csv/default-35575a1051604c88/0.0.0/49187751790fa4d820300fd4d0707896e5b941f1a9c644652645b866716a4ac4...

I am getting this error:
6a4ac4/csv.py in _generate_tables(self, files)
     78     def _generate_tables(self, files):
     79         for i, file in enumerate(files):
---> 80             pa_table = pac.read_csv(
     81                 file,
     82                 read_options=self.config.pa_read_options,

~/anaconda2/envs/nlp/lib/python3.8/site-packages/pyarrow/_csv.pyx in pyarrow._csv.read_csv()

~/anaconda2/envs/nlp/lib/python3.8/site-packages/pyarrow/error.pxi in pyarrow.lib.pyarrow_internal_check_status()

~/anaconda2/envs/nlp/lib/python3.8/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

**ArrowInvalid: straddling object straddles two block boundaries (try to increase block size?)**



The size of the file is 3.5 GB. When I try smaller files I do not have an issue. When I load it with 'text' parser I can see all data but it is not what I need.
There is no issue reading the file with pandas. any idea what could be the issue?
When I am running a different CSV I do not get  this line:
 (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size)

Any ideas?
