Question/problem with dataset labels
NONE
It seems that I get parsing errors for various fields in my data. For example now I get this:
```
  File "../../../models/tr-4.3.2/run_puppets.py", line 523, in <module>
    main()
  File "../../../models/tr-4.3.2/run_puppets.py", line 249, in main
    datasets = load_dataset("csv", data_files=data_files)
  File "/dccstor/redrug_ier/envs/last-tr/lib/python3.8/site-packages/datasets/load.py", line 740, in load_dataset
    builder_instance.download_and_prepare(
  File "/dccstor/redrug_ier/envs/last-tr/lib/python3.8/site-packages/datasets/builder.py", line 572, in download_and_prepare
    self._download_and_prepare(
  File "/dccstor/redrug_ier/envs/last-tr/lib/python3.8/site-packages/datasets/builder.py", line 650, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/dccstor/redrug_ier/envs/last-tr/lib/python3.8/site-packages/datasets/builder.py", line 1028, in _prepare_split
    writer.write_table(table)
  File "/dccstor/redrug_ier/envs/last-tr/lib/python3.8/site-packages/datasets/arrow_writer.py", line 292, in write_table
    pa_table = pa_table.cast(self._schema)
  File "pyarrow/table.pxi", line 1311, in pyarrow.lib.Table.cast
  File "pyarrow/table.pxi", line 265, in pyarrow.lib.ChunkedArray.cast
  File "/dccstor/redrug_ier/envs/last-tr/lib/python3.8/site-packages/pyarrow/compute.py", line 87, in cast
    return call_function("cast", [arr], options)
  File "pyarrow/_compute.pyx", line 298, in pyarrow._compute.call_function
  File "pyarrow/_compute.pyx", line 192, in pyarrow._compute.Function.call
  File "pyarrow/error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 84, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Failed to parse string: https://www.netgalley.com/catalog/book/121872
```
2021-03-04T18:44:57Z
https://github.com/huggingface/datasets/issues/1989#issuecomment-790842771
https://api.github.com/repos/huggingface/datasets/issues/1989
MDEyOklzc3VlQ29tbWVudDc5MDg0Mjc3MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/790842771/reactions
2021-03-04T18:44:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/790842771
https://avatars.githubusercontent.com/u/17202292?v=4
https://api.github.com/users/ioana-blue/events{/privacy}
https://api.github.com/users/ioana-blue/followers
https://api.github.com/users/ioana-blue/following{/other_user}
https://api.github.com/users/ioana-blue/gists{/gist_id}
https://github.com/ioana-blue
ioana-blue
MDQ6VXNlcjE3MjAyMjky
https://api.github.com/users/ioana-blue/orgs
https://api.github.com/users/ioana-blue/received_events
https://api.github.com/users/ioana-blue/repos
https://api.github.com/users/ioana-blue/starred{/owner}{/repo}
https://api.github.com/users/ioana-blue/subscriptions
User
https://api.github.com/users/ioana-blue
public
NONE
Not sure if this helps, this is how I load my files (as in the sample scripts on transformers):

```
    if data_args.train_file.endswith(".csv"):
        # Loading a dataset from local csv files
        datasets = load_dataset("csv", data_files=data_files)
```
2021-03-04T19:13:02Z
https://github.com/huggingface/datasets/issues/1989#issuecomment-790860788
https://api.github.com/repos/huggingface/datasets/issues/1989
MDEyOklzc3VlQ29tbWVudDc5MDg2MDc4OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/790860788/reactions
2021-03-04T19:13:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/790860788
https://avatars.githubusercontent.com/u/17202292?v=4
https://api.github.com/users/ioana-blue/events{/privacy}
https://api.github.com/users/ioana-blue/followers
https://api.github.com/users/ioana-blue/following{/other_user}
https://api.github.com/users/ioana-blue/gists{/gist_id}
https://github.com/ioana-blue
ioana-blue
MDQ6VXNlcjE3MjAyMjky
https://api.github.com/users/ioana-blue/orgs
https://api.github.com/users/ioana-blue/received_events
https://api.github.com/users/ioana-blue/repos
https://api.github.com/users/ioana-blue/starred{/owner}{/repo}
https://api.github.com/users/ioana-blue/subscriptions
User
https://api.github.com/users/ioana-blue
public
NONE
Since this worked out of the box in a few examples before, I wonder if it's some quoting issue or something else. 
2021-03-04T19:15:36Z
https://github.com/huggingface/datasets/issues/1989#issuecomment-790862303
https://api.github.com/repos/huggingface/datasets/issues/1989
MDEyOklzc3VlQ29tbWVudDc5MDg2MjMwMw==
https://api.github.com/repos/huggingface/datasets/issues/comments/790862303/reactions
2021-03-04T19:15:36Z
https://api.github.com/repos/huggingface/datasets/issues/comments/790862303
https://avatars.githubusercontent.com/u/17202292?v=4
https://api.github.com/users/ioana-blue/events{/privacy}
https://api.github.com/users/ioana-blue/followers
https://api.github.com/users/ioana-blue/following{/other_user}
https://api.github.com/users/ioana-blue/gists{/gist_id}
https://github.com/ioana-blue
ioana-blue
MDQ6VXNlcjE3MjAyMjky
https://api.github.com/users/ioana-blue/orgs
https://api.github.com/users/ioana-blue/received_events
https://api.github.com/users/ioana-blue/repos
https://api.github.com/users/ioana-blue/starred{/owner}{/repo}
https://api.github.com/users/ioana-blue/subscriptions
User
https://api.github.com/users/ioana-blue
public
CONTRIBUTOR
Hi @ioana-blue,
Can you share a sample from your .csv? A dummy where you get this error will also help.

I tried this csv:
```csv
feature,label
1.2,not nurse
1.3,nurse
1.5,surgeon
```
and the following snippet:
```python
from datasets import load_dataset

d = load_dataset("csv",data_files=['test.csv'])

print(d)
print(d['train']['label'])
```
and this works perfectly fine for me:
```sh
DatasetDict({
    train: Dataset({
        features: ['feature', 'label'],
        num_rows: 3
    })
})
['not nurse', 'nurse', 'surgeon']
```
I'm sure your csv is more complicated than this one. But it is hard to tell where the issue might be without looking at a sample.
2021-03-05T03:23:16Z
https://github.com/huggingface/datasets/issues/1989#issuecomment-791123097
https://api.github.com/repos/huggingface/datasets/issues/1989
MDEyOklzc3VlQ29tbWVudDc5MTEyMzA5Nw==
https://api.github.com/repos/huggingface/datasets/issues/comments/791123097/reactions
2021-03-05T03:24:22Z
https://api.github.com/repos/huggingface/datasets/issues/comments/791123097
https://avatars.githubusercontent.com/u/29076344?v=4
https://api.github.com/users/gchhablani/events{/privacy}
https://api.github.com/users/gchhablani/followers
https://api.github.com/users/gchhablani/following{/other_user}
https://api.github.com/users/gchhablani/gists{/gist_id}
https://github.com/gchhablani
gchhablani
MDQ6VXNlcjI5MDc2MzQ0
https://api.github.com/users/gchhablani/orgs
https://api.github.com/users/gchhablani/received_events
https://api.github.com/users/gchhablani/repos
https://api.github.com/users/gchhablani/starred{/owner}{/repo}
https://api.github.com/users/gchhablani/subscriptions
User
https://api.github.com/users/gchhablani
public
NONE
I've had versions where it worked fain. For this dataset, I had all kind of parsing issues that I couldn't understand. What I ended up doing is strip all the columns that I didn't need and also make the label 0/1. 

I think one line that may have caused a problem was the csv version of this:

```crawl-data/CC-MAIN-2017-47/segments/1510934806225.78/wet/CC-MAIN-20171120203833-20171120223833-00571.warc.wet.gz        Rose Blakey is an aspiring journalist. She is desperate to escape the from the small Australian town in which she lives. Rejection after rejection mean she is stuck in what she sees as a dead-end waitressing job. ^M ('Rose', '', 'Blakey')  journalist      F       38      journalist      https://www.netgalley.com/catalog/book/121872   _ is desperate to escape the from the small Australian town in which _ lives. Rejection after rejection mean _ is stuck in what _ sees as a dead-end waitressing job.    She is desperate to escape the from the small Australian town in which she lives. Rejection after rejection mean she is stuck in what she sees as a dead-end waitressing job.```

The error I got in this case is this one: https://github.com/huggingface/datasets/issues/1989#issuecomment-790842771

Note, this line was part of a much larger file and until this line I guess it was working fine. 
2021-03-05T13:35:00Z
https://github.com/huggingface/datasets/issues/1989#issuecomment-791422646
https://api.github.com/repos/huggingface/datasets/issues/1989
MDEyOklzc3VlQ29tbWVudDc5MTQyMjY0Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/791422646/reactions
2021-03-05T13:35:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/791422646
https://avatars.githubusercontent.com/u/17202292?v=4
https://api.github.com/users/ioana-blue/events{/privacy}
https://api.github.com/users/ioana-blue/followers
https://api.github.com/users/ioana-blue/following{/other_user}
https://api.github.com/users/ioana-blue/gists{/gist_id}
https://github.com/ioana-blue
ioana-blue
MDQ6VXNlcjE3MjAyMjky
https://api.github.com/users/ioana-blue/orgs
https://api.github.com/users/ioana-blue/received_events
https://api.github.com/users/ioana-blue/repos
https://api.github.com/users/ioana-blue/starred{/owner}{/repo}
https://api.github.com/users/ioana-blue/subscriptions
User
https://api.github.com/users/ioana-blue
public
CONTRIBUTOR
Hi @ioana-blue,

What is the separator you're using for the csv? I see there are only two commas in the given line, but they don't seem like appropriate points. Also, is this a string part of one line, or an entire line? There should also be a label, right?
2021-03-05T14:37:12Z
https://github.com/huggingface/datasets/issues/1989#issuecomment-791458626
https://api.github.com/repos/huggingface/datasets/issues/1989
MDEyOklzc3VlQ29tbWVudDc5MTQ1ODYyNg==
https://api.github.com/repos/huggingface/datasets/issues/comments/791458626/reactions
2021-03-05T14:49:42Z
https://api.github.com/repos/huggingface/datasets/issues/comments/791458626
https://avatars.githubusercontent.com/u/29076344?v=4
https://api.github.com/users/gchhablani/events{/privacy}
https://api.github.com/users/gchhablani/followers
https://api.github.com/users/gchhablani/following{/other_user}
https://api.github.com/users/gchhablani/gists{/gist_id}
https://github.com/gchhablani
gchhablani
MDQ6VXNlcjI5MDc2MzQ0
https://api.github.com/users/gchhablani/orgs
https://api.github.com/users/gchhablani/received_events
https://api.github.com/users/gchhablani/repos
https://api.github.com/users/gchhablani/starred{/owner}{/repo}
https://api.github.com/users/gchhablani/subscriptions
User
https://api.github.com/users/gchhablani
public
NONE
Sorry for the confusion, the sample above was from a tsv that was used to derive the csv. Let me construct the csv again (I had remove it). 

This is the line in the csv - this is the whole line:
```crawl-data/CC-MAIN-2017-47/segments/1510934806225.78/wet/CC-MAIN-20171120203833-20171120223833-00571.warc.wet.gz,Rose Blakey is an aspiring journalist. She is desperate to escape the from the small Australian town in which she lives. Rejection after rejection mean she is stuck in what she sees as a dead,"('Rose', '', 'Blakey')",journalist,F,38,journalist,https://www.netgalley.com/catalog/book/121872,_ is desperate to escape the from the small Australian town in which _ lives. Rejection after rejection mean _ is stuck in what _ sees as a dead-end waitressing job., She is desperate to escape the from the small Australian town in which she lives. Rejection after rejection mean she is stuck in what she sees as a dead-end waitressing job.```
2021-03-05T15:33:56Z
https://github.com/huggingface/datasets/issues/1989#issuecomment-791494268
https://api.github.com/repos/huggingface/datasets/issues/1989
MDEyOklzc3VlQ29tbWVudDc5MTQ5NDI2OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/791494268/reactions
2021-03-05T15:33:56Z
https://api.github.com/repos/huggingface/datasets/issues/comments/791494268
https://avatars.githubusercontent.com/u/17202292?v=4
https://api.github.com/users/ioana-blue/events{/privacy}
https://api.github.com/users/ioana-blue/followers
https://api.github.com/users/ioana-blue/following{/other_user}
https://api.github.com/users/ioana-blue/gists{/gist_id}
https://github.com/ioana-blue
ioana-blue
MDQ6VXNlcjE3MjAyMjky
https://api.github.com/users/ioana-blue/orgs
https://api.github.com/users/ioana-blue/received_events
https://api.github.com/users/ioana-blue/repos
https://api.github.com/users/ioana-blue/starred{/owner}{/repo}
https://api.github.com/users/ioana-blue/subscriptions
User
https://api.github.com/users/ioana-blue
public
CONTRIBUTOR
Hi,
Just in case you want to use tsv directly, you can use the separator argument while loading the dataset.
```python
d = load_dataset("csv",data_files=['test.csv'],sep="\t")
```

Additionally, I don't face the issues with the following csv (same as the one you provided):

```sh
link1,text1,info1,info2,info3,info4,info5,link2,text2,text3
crawl-data/CC-MAIN-2017-47/segments/1510934806225.78/wet/CC-MAIN-20171120203833-20171120223833-00571.warc.wet.gz,Rose Blakey is an aspiring journalist. She is desperate to escape the from the small Australian town in which she lives. Rejection after rejection mean she is stuck in what she sees as a dead,"('Rose', '', 'Blakey')",journalist,F,38,journalist,https://www.netgalley.com/catalog/book/121872,_ is desperate to escape the from the small Australian town in which _ lives. Rejection after rejection mean _ is stuck in what _ sees as a dead-end waitressing job., She is desperate to escape the from the small Australian town in which she lives. Rejection after rejection mean she is stuck in what she sees as a dead-end waitressing job.
```
Output after loading:
```sh
{'link1': 'crawl-data/CC-MAIN-2017-47/segments/1510934806225.78/wet/CC-MAIN-20171120203833-20171120223833-00571.warc.wet.gz', 'text1': 'Rose Blakey is an aspiring journalist. She is desperate to escape the from the small Australian town in which she lives. Rejection after rejection mean she is stuck in what she sees as a dead', 'info1': "('Rose', '', 'Blakey')", 'info2': 'journalist', 'info3': 'F', 'info4': 38, 'info5': 'journalist', 'link2': 'https://www.netgalley.com/catalog/book/121872', 'text2': '_ is desperate to escape the from the small Australian town in which _ lives. Rejection after rejection mean _ is stuck in what _ sees as a dead-end waitressing job.', 'text3': ' She is desperate to escape the from the small Australian town in which she lives. Rejection after rejection mean she is stuck in what she sees as a dead-end waitressing job.'}
```
Can you check once if the tsv works for you directly using the separator argument? The conversion from tsv to csv could create issues, I'm only guessing though.
2021-03-05T15:48:07Z
https://github.com/huggingface/datasets/issues/1989#issuecomment-791503586
https://api.github.com/repos/huggingface/datasets/issues/1989
MDEyOklzc3VlQ29tbWVudDc5MTUwMzU4Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/791503586/reactions
2021-03-05T15:51:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/791503586
https://avatars.githubusercontent.com/u/29076344?v=4
https://api.github.com/users/gchhablani/events{/privacy}
https://api.github.com/users/gchhablani/followers
https://api.github.com/users/gchhablani/following{/other_user}
https://api.github.com/users/gchhablani/gists{/gist_id}
https://github.com/gchhablani
gchhablani
MDQ6VXNlcjI5MDc2MzQ0
https://api.github.com/users/gchhablani/orgs
https://api.github.com/users/gchhablani/received_events
https://api.github.com/users/gchhablani/repos
https://api.github.com/users/gchhablani/starred{/owner}{/repo}
https://api.github.com/users/gchhablani/subscriptions
User
https://api.github.com/users/gchhablani
public
NONE
thanks for the tip. very strange :/ I'll check my datasets version as well. 

I will have more similar experiments soon so I'll let you know if I manage to get rid of this. 
2021-03-05T16:16:56Z
https://github.com/huggingface/datasets/issues/1989#issuecomment-791522319
https://api.github.com/repos/huggingface/datasets/issues/1989
MDEyOklzc3VlQ29tbWVudDc5MTUyMjMxOQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/791522319/reactions
2021-03-05T16:16:56Z
https://api.github.com/repos/huggingface/datasets/issues/comments/791522319
https://avatars.githubusercontent.com/u/17202292?v=4
https://api.github.com/users/ioana-blue/events{/privacy}
https://api.github.com/users/ioana-blue/followers
https://api.github.com/users/ioana-blue/following{/other_user}
https://api.github.com/users/ioana-blue/gists{/gist_id}
https://github.com/ioana-blue
ioana-blue
MDQ6VXNlcjE3MjAyMjky
https://api.github.com/users/ioana-blue/orgs
https://api.github.com/users/ioana-blue/received_events
https://api.github.com/users/ioana-blue/repos
https://api.github.com/users/ioana-blue/starred{/owner}{/repo}
https://api.github.com/users/ioana-blue/subscriptions
User
https://api.github.com/users/ioana-blue
public
CONTRIBUTOR
No problem at all. I thought I'd be able to solve this but I'm unable to replicate the issue :/
2021-03-05T16:48:44Z
https://github.com/huggingface/datasets/issues/1989#issuecomment-791542975
https://api.github.com/repos/huggingface/datasets/issues/1989
MDEyOklzc3VlQ29tbWVudDc5MTU0Mjk3NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/791542975/reactions
2021-03-05T16:49:56Z
https://api.github.com/repos/huggingface/datasets/issues/comments/791542975
https://avatars.githubusercontent.com/u/29076344?v=4
https://api.github.com/users/gchhablani/events{/privacy}
https://api.github.com/users/gchhablani/followers
https://api.github.com/users/gchhablani/following{/other_user}
https://api.github.com/users/gchhablani/gists{/gist_id}
https://github.com/gchhablani
gchhablani
MDQ6VXNlcjI5MDc2MzQ0
https://api.github.com/users/gchhablani/orgs
https://api.github.com/users/gchhablani/received_events
https://api.github.com/users/gchhablani/repos
https://api.github.com/users/gchhablani/starred{/owner}{/repo}
https://api.github.com/users/gchhablani/subscriptions
User
https://api.github.com/users/gchhablani
public
Hi, I'm using a dataset with two labels "nurse" and "not nurse". For whatever reason (that I don't understand), I get an error that I think comes from the datasets package (using csv). Everything works fine if the labels are "nurse" and "surgeon". 

This is the trace I get:

```
File "../../../models/tr-4.3.2/run_puppets.py", line 523, in <module>
    main()
  File "../../../models/tr-4.3.2/run_puppets.py", line 249, in main
    datasets = load_dataset("csv", data_files=data_files)
  File "/dccstor/redrug_ier/envs/last-tr/lib/python3.8/site-packages/datasets/load.py", line 740, in load_dataset
    builder_instance.download_and_prepare(
  File "/dccstor/redrug_ier/envs/last-tr/lib/python3.8/site-packages/datasets/builder.py", line 572, in download_and_prepare
    self._download_and_prepare(
  File "/dccstor/redrug_ier/envs/last-tr/lib/python3.8/site-packages/datasets/builder.py", line 650, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/dccstor/redrug_ier/envs/last-tr/lib/python3.8/site-packages/datasets/builder.py", line 1028, in _prepare_split
    writer.write_table(table)
  File "/dccstor/redrug_ier/envs/last-tr/lib/python3.8/site-packages/datasets/arrow_writer.py", line 292, in write_table
    pa_table = pa_table.cast(self._schema)
  File "pyarrow/table.pxi", line 1311, in pyarrow.lib.Table.cast
  File "pyarrow/table.pxi", line 265, in pyarrow.lib.ChunkedArray.cast
  File "/dccstor/redrug_ier/envs/last-tr/lib/python3.8/site-packages/pyarrow/compute.py", line 87, in cast
    return call_function("cast", [arr], options)
  File "pyarrow/_compute.pyx", line 298, in pyarrow._compute.call_function
  File "pyarrow/_compute.pyx", line 192, in pyarrow._compute.Function.call
  File "pyarrow/error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 84, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Failed to parse string: not nurse
```

Any ideas how to fix this? For now, I'll probably make them numeric. 