Use `torch.nested_tensor` for arrays of varying length in torch formatter
CONTRIBUTOR
What's the current behavior?
2023-07-06T21:15:07Z
https://github.com/huggingface/datasets/issues/4490#issuecomment-1624318901
https://api.github.com/repos/huggingface/datasets/issues/4490
IC_kwDODunzps5g0SO1
https://api.github.com/repos/huggingface/datasets/issues/comments/1624318901/reactions
2023-07-06T21:15:07Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1624318901
https://avatars.githubusercontent.com/u/36224762?v=4
https://api.github.com/users/NightMachinery/events{/privacy}
https://api.github.com/users/NightMachinery/followers
https://api.github.com/users/NightMachinery/following{/other_user}
https://api.github.com/users/NightMachinery/gists{/gist_id}
https://github.com/NightMachinery
NightMachinery
MDQ6VXNlcjM2MjI0NzYy
https://api.github.com/users/NightMachinery/orgs
https://api.github.com/users/NightMachinery/received_events
https://api.github.com/users/NightMachinery/repos
https://api.github.com/users/NightMachinery/starred{/owner}{/repo}
https://api.github.com/users/NightMachinery/subscriptions
User
https://api.github.com/users/NightMachinery
public
COLLABORATOR
Currently, we return a list of Torch tensors if their shapes don't match. If they do, we consolidate them into a single Torch tensor.
2023-07-07T13:02:58Z
https://github.com/huggingface/datasets/issues/4490#issuecomment-1625385431
https://api.github.com/repos/huggingface/datasets/issues/4490
IC_kwDODunzps5g4WnX
https://api.github.com/repos/huggingface/datasets/issues/comments/1625385431/reactions
2023-07-07T13:02:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1625385431
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
Use `torch.nested_tensor` for arrays of varying length in `TorchFormatter`.

The PyTorch API of nested tensors is in the prototype stage, so wait for it to become more mature.