Dataset.map() turns tensors into lists?
NONE
A solution is to have the tokenizer return a list instead of a tensor, and then use `dataset_tok.set_format(type = 'torch')` to convert that list into a tensor. Still not sure if bug.
2020-12-07T10:54:50Z
https://github.com/huggingface/datasets/issues/1046#issuecomment-739841253
https://api.github.com/repos/huggingface/datasets/issues/1046
MDEyOklzc3VlQ29tbWVudDczOTg0MTI1Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/739841253/reactions
2020-12-07T10:54:50Z
https://api.github.com/repos/huggingface/datasets/issues/comments/739841253
https://avatars.githubusercontent.com/u/5270804?v=4
https://api.github.com/users/tombosc/events{/privacy}
https://api.github.com/users/tombosc/followers
https://api.github.com/users/tombosc/following{/other_user}
https://api.github.com/users/tombosc/gists{/gist_id}
https://github.com/tombosc
tombosc
MDQ6VXNlcjUyNzA4MDQ=
https://api.github.com/users/tombosc/orgs
https://api.github.com/users/tombosc/received_events
https://api.github.com/users/tombosc/repos
https://api.github.com/users/tombosc/starred{/owner}{/repo}
https://api.github.com/users/tombosc/subscriptions
User
https://api.github.com/users/tombosc
public
MEMBER
It is expected behavior, you should set the format to `"torch"` as you mentioned to get pytorch tensors back.
By default datasets returns pure python objects.
2020-12-23T13:51:12Z
https://github.com/huggingface/datasets/issues/1046#issuecomment-750308929
https://api.github.com/repos/huggingface/datasets/issues/1046
MDEyOklzc3VlQ29tbWVudDc1MDMwODkyOQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/750308929/reactions
2020-12-23T13:51:12Z
https://api.github.com/repos/huggingface/datasets/issues/comments/750308929
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
I apply `Dataset.map()` to a function that returns a dict of torch tensors (like a tokenizer from the repo transformers). However, in the mapped dataset, these tensors have turned to lists!

```import datasets
import torch  
from datasets import load_dataset                                                                                                                 
print("version datasets", datasets.__version__)

dataset = load_dataset("snli", split='train[0:50]')  

def tokenizer_fn(example):
    # actually uses a tokenizer which does something like:
    return {'input_ids': torch.tensor([[0, 1, 2]])}

print("First item in dataset:\n", dataset[0])
tokenized = tokenizer_fn(dataset[0])
print("Tokenized hyp:\n", tokenized)
dataset_tok = dataset.map(tokenizer_fn, batched=False,
        remove_columns=['label', 'premise', 'hypothesis'])
print("Tokenized using map:\n", dataset_tok[0])
print(type(tokenized['input_ids']), type(dataset_tok[0]['input_ids']))
dataset_tok = dataset.map(tokenizer_fn, batched=False,
                          remove_columns=['label', 'premise', 'hypothesis'])
print("Tokenized using map:\n", dataset_tok[0])
print(type(tokenized['input_ids']), type(dataset_tok[0]['input_ids']))
```

The output is:

```
version datasets 1.1.3
Reusing dataset snli (/home/tom/.cache/huggingface/datasets/snli/plain_text/1.0.0/bb1102591c6230bd78813e229d5dd4c7fbf4fc478cec28f298761eb69e5b537c)
First item in dataset:
 {'premise': 'A person on a horse jumps over a broken down airplane.', 'hypothesis': 'A person is training his horse for a competition.', 'label': 1}
Tokenized hyp:
 {'input_ids': tensor([[0, 1, 2]])}
Loading cached processed dataset at /home/tom/.cache/huggingface/datasets/snli/plain_text/1.0.0/bb1102591c6230bd78813e229d5dd4c7fbf4fc478cec28f298761eb69e5b537c/cache-fe38f449fe9ac46f.arrow
Tokenized using map:
 {'input_ids': [[0, 1, 2]]}
<class 'torch.Tensor'> <class 'list'>
```

Or am I doing something wrong?
