Why return_tensors='pt' doesn't work？
MEMBER
Hi ! `datasets` uses Arrow as storage backend which is agnostic to deep learning frameworks like torch. If you want to get torch tensors back, you need to do `dataset = dataset.with_format("torch")`
2024-11-18T11:26:44Z
https://github.com/huggingface/datasets/issues/7291#issuecomment-2482768478
https://api.github.com/repos/huggingface/datasets/issues/7291
IC_kwDODunzps6T_Ape
https://api.github.com/repos/huggingface/datasets/issues/comments/2482768478/reactions
2024-11-18T11:26:44Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2482768478
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
> Hi ! `datasets` uses Arrow as storage backend which is agnostic to deep learning frameworks like torch. If you want to get torch tensors back, you need to do `dataset = dataset.with_format("torch")`

It does work! Thanks for your suggestion!
2024-11-18T13:47:06Z
https://github.com/huggingface/datasets/issues/7291#issuecomment-2483099047
https://api.github.com/repos/huggingface/datasets/issues/7291
IC_kwDODunzps6UARWn
https://api.github.com/repos/huggingface/datasets/issues/comments/2483099047/reactions
2024-11-18T13:47:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2483099047
https://avatars.githubusercontent.com/u/86752851?v=4
https://api.github.com/users/bw-wang19/events{/privacy}
https://api.github.com/users/bw-wang19/followers
https://api.github.com/users/bw-wang19/following{/other_user}
https://api.github.com/users/bw-wang19/gists{/gist_id}
https://github.com/bw-wang19
bw-wang19
MDQ6VXNlcjg2NzUyODUx
https://api.github.com/users/bw-wang19/orgs
https://api.github.com/users/bw-wang19/received_events
https://api.github.com/users/bw-wang19/repos
https://api.github.com/users/bw-wang19/starred{/owner}{/repo}
https://api.github.com/users/bw-wang19/subscriptions
User
https://api.github.com/users/bw-wang19
public
### Describe the bug

I tried to add input_ids to dataset with map(), and I used the return_tensors='pt', but why I got the callback with the type of List？
![image](https://github.com/user-attachments/assets/ab046e20-2174-4e91-9cd6-4a296a43e83c)


### Steps to reproduce the bug

![image](https://github.com/user-attachments/assets/5d504d4c-22c7-4742-99a1-9cab78739b17)

### Expected behavior

Sorry for this silly question, I'm noob on using this tool. But I think it should return a tensor value as I have used the protocol？
When I tokenize only one sentence using tokenized_input=tokenizer(input, return_tensors='pt' )，it does return in tensor type. Why doesn't it work in map()？

### Environment info

transformers>=4.41.2,<=4.45.0
datasets>=2.16.0,<=2.21.0
accelerate>=0.30.1,<=0.34.2
peft>=0.11.1,<=0.12.0
trl>=0.8.6,<=0.9.6
gradio>=4.0.0
pandas>=2.0.0
scipy
einops
sentencepiece
tiktoken
protobuf
uvicorn
pydantic
fastapi
sse-starlette
matplotlib>=3.7.0
fire
packaging
pyyaml
numpy<2.0.0
