Saving a dataset after setting format to torch doesn't work, but only if filtering
NONE
Hey, I'm a research engineer working on language modelling wanting to contribute to open source. I was wondering if I could give it a shot?
2023-02-08T14:22:25Z
https://github.com/huggingface/datasets/issues/5508#issuecomment-1422676084
https://api.github.com/repos/huggingface/datasets/issues/5508
IC_kwDODunzps5UzFB0
https://api.github.com/repos/huggingface/datasets/issues/comments/1422676084/reactions
2023-02-08T14:22:25Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1422676084
https://avatars.githubusercontent.com/u/47942453?v=4
https://api.github.com/users/jackapbutler/events{/privacy}
https://api.github.com/users/jackapbutler/followers
https://api.github.com/users/jackapbutler/following{/other_user}
https://api.github.com/users/jackapbutler/gists{/gist_id}
https://github.com/jackapbutler
jackapbutler
MDQ6VXNlcjQ3OTQyNDUz
https://api.github.com/users/jackapbutler/orgs
https://api.github.com/users/jackapbutler/received_events
https://api.github.com/users/jackapbutler/repos
https://api.github.com/users/jackapbutler/starred{/owner}{/repo}
https://api.github.com/users/jackapbutler/subscriptions
User
https://api.github.com/users/jackapbutler
public
COLLABORATOR
Hi! This issue was fixed in https://github.com/huggingface/datasets/pull/4972, so please install `datasets>=2.5.0` to avoid it.
2023-02-09T14:25:12Z
https://github.com/huggingface/datasets/issues/5508#issuecomment-1424271629
https://api.github.com/repos/huggingface/datasets/issues/5508
IC_kwDODunzps5U5KkN
https://api.github.com/repos/huggingface/datasets/issues/comments/1424271629/reactions
2023-02-09T14:25:12Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1424271629
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
### Describe the bug

Saving a dataset after setting format to torch doesn't work, but only if filtering

### Steps to reproduce the bug

```
a = Dataset.from_dict({"b": [1, 2]})
a.set_format('torch')
a.save_to_disk("test_save") # saves successfully
a.filter(None).save_to_disk("test_save_filter") # does not
>> [...] TypeError: Provided `function` which is applied to all elements of table returns a `dict` of types [<class 'torch.Tensor'>]. When using `batched=True`, make sure provided `function` returns a `dict` of types like `(<class 'list'>, <class 'numpy.ndarray'>)`.

# note: skipping the format change to torch lets this work.



### Expected behavior

Saving to work

### Environment info

- `datasets` version: 2.4.0
- Platform: Linux-6.1.9-arch1-1-x86_64-with-glibc2.36
- Python version: 3.10.9
- PyArrow version: 9.0.0
- Pandas version: 1.4.4