invalid data type 'str' at _convert_outputs in arrow_dataset.py
MEMBER
Hi ! Did you try to set the output format to pytorch ? (or tensorflow if you're using tensorflow)
It can be done with `dataset.set_format("torch", columns=columns)`  (or "tensorflow").

Note that for pytorch, string columns can't be converted to `torch.Tensor`, so you have to specify in `columns=` the list of columns you want to keep (`input_ids` for example)
2020-08-03T17:07:39Z
https://github.com/huggingface/datasets/issues/469#issuecomment-668135828
https://api.github.com/repos/huggingface/datasets/issues/469
MDEyOklzc3VlQ29tbWVudDY2ODEzNTgyOA==
https://api.github.com/repos/huggingface/datasets/issues/comments/668135828/reactions
2020-08-03T17:07:39Z
https://api.github.com/repos/huggingface/datasets/issues/comments/668135828
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Hello . Yes, I did set the output format as below for the two columns 

  `train_dataset.set_format('torch',columns=['Text','Label'])`
 
2020-08-04T08:12:42Z
https://github.com/huggingface/datasets/issues/469#issuecomment-668451764
https://api.github.com/repos/huggingface/datasets/issues/469
MDEyOklzc3VlQ29tbWVudDY2ODQ1MTc2NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/668451764/reactions
2020-08-04T08:12:42Z
https://api.github.com/repos/huggingface/datasets/issues/comments/668451764
https://avatars.githubusercontent.com/u/30617486?v=4
https://api.github.com/users/Murgates/events{/privacy}
https://api.github.com/users/Murgates/followers
https://api.github.com/users/Murgates/following{/other_user}
https://api.github.com/users/Murgates/gists{/gist_id}
https://github.com/Murgates
Murgates
MDQ6VXNlcjMwNjE3NDg2
https://api.github.com/users/Murgates/orgs
https://api.github.com/users/Murgates/received_events
https://api.github.com/users/Murgates/repos
https://api.github.com/users/Murgates/starred{/owner}{/repo}
https://api.github.com/users/Murgates/subscriptions
User
https://api.github.com/users/Murgates
public
MEMBER
I think you're having this issue because you try to format strings as pytorch tensors, which is not possible.
Indeed by having "Text" in `columns=['Text','Label']`, you try to convert the text values to pytorch tensors.

Instead I recommend you to first tokenize your dataset using a tokenizer from transformers. For example

```python
from transformers import BertTokenizer
tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")

train_dataset.map(lambda x: tokenizer(x["Text"]), batched=True)
train_dataset.set_format("torch", column=["input_ids"])
```

Another way to fix your issue would be to not set the format to pytorch, and leave the dataset as it is by default. In that case, the strings are returned normally when you get examples from your dataloader. It means that you would have to tokenize the examples in the training loop (or using a data collator) though.

Let me know if you have other questions
2020-08-24T15:22:06Z
https://github.com/huggingface/datasets/issues/469#issuecomment-679193244
https://api.github.com/repos/huggingface/datasets/issues/469
MDEyOklzc3VlQ29tbWVudDY3OTE5MzI0NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/679193244/reactions
2020-08-24T15:22:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/679193244
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Hi, actually the thing is I am getting the same error and even after tokenizing them I am passing them through batch_encode_plus.
I dont know what seems to be the problem is. I even converted it into 'pt' while passing them through batch_encode_plus but when I am evaluating my model , i am getting this error


---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-145-ca218223c9fc> in <module>()
----> 1 val_loss, predictions, true_val = evaluate(dataloader_validation)
      2 val_f1 = f1_score_func(predictions, true_val)
      3 tqdm.write(f'Validation loss: {val_loss}')
      4 tqdm.write(f'F1 Score (Weighted): {val_f1}')

6 frames
/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataset.py in <genexpr>(.0)
    160 
    161     def __getitem__(self, index):
--> 162         return tuple(tensor[index] for tensor in self.tensors)
    163 
    164     def __len__(self):

TypeError: new(): invalid data type 'str' 
2020-10-20T19:09:43Z
https://github.com/huggingface/datasets/issues/469#issuecomment-713078513
https://api.github.com/repos/huggingface/datasets/issues/469
MDEyOklzc3VlQ29tbWVudDcxMzA3ODUxMw==
https://api.github.com/repos/huggingface/datasets/issues/comments/713078513/reactions
2020-10-20T19:09:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/713078513
https://avatars.githubusercontent.com/u/70738578?v=4
https://api.github.com/users/akhilkapil/events{/privacy}
https://api.github.com/users/akhilkapil/followers
https://api.github.com/users/akhilkapil/following{/other_user}
https://api.github.com/users/akhilkapil/gists{/gist_id}
https://github.com/akhilkapil
akhilkapil
MDQ6VXNlcjcwNzM4NTc4
https://api.github.com/users/akhilkapil/orgs
https://api.github.com/users/akhilkapil/received_events
https://api.github.com/users/akhilkapil/repos
https://api.github.com/users/akhilkapil/starred{/owner}{/repo}
https://api.github.com/users/akhilkapil/subscriptions
User
https://api.github.com/users/akhilkapil
public
NONE
> Hi, actually the thing is I am getting the same error and even after tokenizing them I am passing them through batch_encode_plus.
> I dont know what seems to be the problem is. I even converted it into 'pt' while passing them through batch_encode_plus but when I am evaluating my model , i am getting this error
> 
> TypeError Traceback (most recent call last)
> in ()
> ----> 1 val_loss, predictions, true_val = evaluate(dataloader_validation)
> 2 val_f1 = f1_score_func(predictions, true_val)
> 3 tqdm.write(f'Validation loss: {val_loss}')
> 4 tqdm.write(f'F1 Score (Weighted): {val_f1}')
> 
> 6 frames
> /usr/local/lib/python3.6/dist-packages/torch/utils/data/dataset.py in (.0)
> 160
> 161 def **getitem**(self, index):
> --> 162 return tuple(tensor[index] for tensor in self.tensors)
> 163
> 164 def **len**(self):
> 
> TypeError: new(): invalid data type 'str'

I got the same error and fix it .
you can check your input where there may be string contained.
such as
```
a = [1,2,3,4,'<unk>']
torch.tensor(a)
```
2020-10-22T08:33:56Z
https://github.com/huggingface/datasets/issues/469#issuecomment-714328603
https://api.github.com/repos/huggingface/datasets/issues/469
MDEyOklzc3VlQ29tbWVudDcxNDMyODYwMw==
https://api.github.com/repos/huggingface/datasets/issues/comments/714328603/reactions
2020-10-22T08:33:56Z
https://api.github.com/repos/huggingface/datasets/issues/comments/714328603
https://avatars.githubusercontent.com/u/26213546?v=4
https://api.github.com/users/Doragd/events{/privacy}
https://api.github.com/users/Doragd/followers
https://api.github.com/users/Doragd/following{/other_user}
https://api.github.com/users/Doragd/gists{/gist_id}
https://github.com/Doragd
Doragd
MDQ6VXNlcjI2MjEzNTQ2
https://api.github.com/users/Doragd/orgs
https://api.github.com/users/Doragd/received_events
https://api.github.com/users/Doragd/repos
https://api.github.com/users/Doragd/starred{/owner}{/repo}
https://api.github.com/users/Doragd/subscriptions
User
https://api.github.com/users/Doragd
public
MEMBER
I didn't know tokenizers could return strings in the token ids. Which tokenizer are you using to get this @Doragd ?
2020-10-22T08:40:18Z
https://github.com/huggingface/datasets/issues/469#issuecomment-714332393
https://api.github.com/repos/huggingface/datasets/issues/469
MDEyOklzc3VlQ29tbWVudDcxNDMzMjM5Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/714332393/reactions
2020-10-22T08:40:18Z
https://api.github.com/repos/huggingface/datasets/issues/comments/714332393
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
> I didn't know tokenizers could return strings in the token ids. Which tokenizer are you using to get this @Doragd ?

i'm sorry that i met this issue in another place (not in huggingface repo). 
2020-10-22T08:58:54Z
https://github.com/huggingface/datasets/issues/469#issuecomment-714343794
https://api.github.com/repos/huggingface/datasets/issues/469
MDEyOklzc3VlQ29tbWVudDcxNDM0Mzc5NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/714343794/reactions
2020-10-22T08:58:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/714343794
https://avatars.githubusercontent.com/u/26213546?v=4
https://api.github.com/users/Doragd/events{/privacy}
https://api.github.com/users/Doragd/followers
https://api.github.com/users/Doragd/following{/other_user}
https://api.github.com/users/Doragd/gists{/gist_id}
https://github.com/Doragd
Doragd
MDQ6VXNlcjI2MjEzNTQ2
https://api.github.com/users/Doragd/orgs
https://api.github.com/users/Doragd/received_events
https://api.github.com/users/Doragd/repos
https://api.github.com/users/Doragd/starred{/owner}{/repo}
https://api.github.com/users/Doragd/subscriptions
User
https://api.github.com/users/Doragd
public
MEMBER
@akhilkapil do you have strings in your dataset ? When you set the dataset format to "pytorch" you should exclude columns with strings as pytorch can't make tensors out of strings
2020-10-22T09:04:26Z
https://github.com/huggingface/datasets/issues/469#issuecomment-714347398
https://api.github.com/repos/huggingface/datasets/issues/469
MDEyOklzc3VlQ29tbWVudDcxNDM0NzM5OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/714347398/reactions
2020-10-22T09:04:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/714347398
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
COLLABORATOR
Closing due to inactivity.
2023-07-20T15:54:17Z
https://github.com/huggingface/datasets/issues/469#issuecomment-1644181014
https://api.github.com/repos/huggingface/datasets/issues/469
IC_kwDODunzps5iADYW
https://api.github.com/repos/huggingface/datasets/issues/comments/1644181014/reactions
2023-07-20T15:54:17Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1644181014
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
I trying to build multi label text classifier model using Transformers lib. 

I'm using Transformers NLP to load the data set, while calling trainer.train() method. It throws the following error 

File "C:\***\arrow_dataset.py", line 343, in _convert_outputs
    v = command(v)
TypeError: new(): invalid data type 'str'

I'm using pyarrow 1.0.0.  And I have simple custom data set with Text and Integer Label.  
Ex: Data
 Text ,     Label  #Column Header
 I'm facing an Network issue, 1
 I forgot my password, 2

Error StackTrace:

File "C:\**\transformers\trainer.py", line 492, in train
    for step, inputs in enumerate(epoch_iterator):
  File "C:\**\tqdm\std.py", line 1104, in __iter__
    for obj in iterable:
  File "C:\**\torch\utils\data\dataloader.py", line 345, in __next__
    data = self._next_data()
  File "C:\**\torch\utils\data\dataloader.py", line 385, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "C:\**\torch\utils\data\_utils\fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "C:\**\torch\utils\data\_utils\fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "C:\**\nlp\arrow_dataset.py", line 414, in __getitem__
    output_all_columns=self._output_all_columns,
  File "C:\**\nlp\arrow_dataset.py", line 403, in _getitem
    outputs, format_type=format_type, format_columns=format_columns, output_all_columns=output_all_columns
  File "C:\**\nlp\arrow_dataset.py", line 343, in _convert_outputs
    v = command(v)
TypeError: new(): invalid data type 'str'
 
