Use pyarrow Tensor dtype
MEMBER
Hi ! We're using the Arrow format for the datasets, and PyArrow tensors are not part of the Arrow format AFAIK:

> There is no direct support in the arrow columnar format to store Tensors as column values.

source: https://github.com/apache/arrow/issues/4802#issuecomment-508494694
2022-11-21T14:42:27Z
https://github.com/huggingface/datasets/issues/5272#issuecomment-1322163494
https://api.github.com/repos/huggingface/datasets/issues/5272
IC_kwDODunzps5Ozp0m
https://api.github.com/repos/huggingface/datasets/issues/comments/1322163494/reactions
2022-11-21T14:42:27Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1322163494
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
@wesm @rok its been around three years. any updates, regarding dataset arrow tensor support? 🙏 I know you must be very busy, would appreciate to learn what is the state of art. I saw the PR is still open [#8510](https://github.com/apache/arrow/pull/8510)
2022-11-21T17:17:01Z
https://github.com/huggingface/datasets/issues/5272#issuecomment-1322399100
https://api.github.com/repos/huggingface/datasets/issues/5272
IC_kwDODunzps5O0jV8
https://api.github.com/repos/huggingface/datasets/issues/comments/1322399100/reactions
2022-11-21T17:27:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1322399100
https://avatars.githubusercontent.com/u/18228395?v=4
https://api.github.com/users/franz101/events{/privacy}
https://api.github.com/users/franz101/followers
https://api.github.com/users/franz101/following{/other_user}
https://api.github.com/users/franz101/gists{/gist_id}
https://github.com/franz101
franz101
MDQ6VXNlcjE4MjI4Mzk1
https://api.github.com/users/franz101/orgs
https://api.github.com/users/franz101/received_events
https://api.github.com/users/franz101/repos
https://api.github.com/users/franz101/starred{/owner}{/repo}
https://api.github.com/users/franz101/subscriptions
User
https://api.github.com/users/franz101
public
NONE
Hey @franz101 & @lhoestq!
There is a plan and a PR to create an [ExtensionArray of Tensors](https://github.com/apache/arrow/pull/8510) of equal sizes as well as a plan to do the same for Tensors of different sizes [ARROW-8714](https://issues.apache.org/jira/browse/ARROW-8714).
2022-11-21T17:28:54Z
https://github.com/huggingface/datasets/issues/5272#issuecomment-1322413885
https://api.github.com/repos/huggingface/datasets/issues/5272
IC_kwDODunzps5O0m89
https://api.github.com/repos/huggingface/datasets/issues/comments/1322413885/reactions
2022-11-21T17:28:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1322413885
https://avatars.githubusercontent.com/u/54589?v=4
https://api.github.com/users/rok/events{/privacy}
https://api.github.com/users/rok/followers
https://api.github.com/users/rok/following{/other_user}
https://api.github.com/users/rok/gists{/gist_id}
https://github.com/rok
rok
MDQ6VXNlcjU0NTg5
https://api.github.com/users/rok/orgs
https://api.github.com/users/rok/received_events
https://api.github.com/users/rok/repos
https://api.github.com/users/rok/starred{/owner}{/repo}
https://api.github.com/users/rok/subscriptions
User
https://api.github.com/users/rok
public
NONE
The work stalled a little because it was not clear where TensorArray would live. However Arrow community recently agreed to make a [well-known-extension-type document](https://lists.apache.org/thread/sxd5fhc42hb6svs79t3fd79gkqj83pfh) and I would like https://github.com/apache/arrow/pull/8510 to land there and add an implementation to C++/Python + another language. Is that something you would find beneficial to you?
2022-11-21T17:37:29Z
https://github.com/huggingface/datasets/issues/5272#issuecomment-1322424043
https://api.github.com/repos/huggingface/datasets/issues/5272
IC_kwDODunzps5O0pbr
https://api.github.com/repos/huggingface/datasets/issues/comments/1322424043/reactions
2022-11-21T17:37:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1322424043
https://avatars.githubusercontent.com/u/54589?v=4
https://api.github.com/users/rok/events{/privacy}
https://api.github.com/users/rok/followers
https://api.github.com/users/rok/following{/other_user}
https://api.github.com/users/rok/gists{/gist_id}
https://github.com/rok
rok
MDQ6VXNlcjU0NTg5
https://api.github.com/users/rok/orgs
https://api.github.com/users/rok/received_events
https://api.github.com/users/rok/repos
https://api.github.com/users/rok/starred{/owner}{/repo}
https://api.github.com/users/rok/subscriptions
User
https://api.github.com/users/rok
public
NONE
that is a great update, thank you.
it looks like this feature would benefit datasets implementation of [ArrayExtensionArray](https://github.com/huggingface/datasets/blob/9f2ff14673cac1f1ad56d80221a793f5938b68c7/src/datasets/features/features.py#L585-L641). Is that correct @eladsegal @lhoestq?


2022-11-21T17:43:50Z
https://github.com/huggingface/datasets/issues/5272#issuecomment-1322431344
https://api.github.com/repos/huggingface/datasets/issues/5272
IC_kwDODunzps5O0rNw
https://api.github.com/repos/huggingface/datasets/issues/comments/1322431344/reactions
2022-11-21T17:44:12Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1322431344
https://avatars.githubusercontent.com/u/18228395?v=4
https://api.github.com/users/franz101/events{/privacy}
https://api.github.com/users/franz101/followers
https://api.github.com/users/franz101/following{/other_user}
https://api.github.com/users/franz101/gists{/gist_id}
https://github.com/franz101
franz101
MDQ6VXNlcjE4MjI4Mzk1
https://api.github.com/users/franz101/orgs
https://api.github.com/users/franz101/received_events
https://api.github.com/users/franz101/repos
https://api.github.com/users/franz101/starred{/owner}{/repo}
https://api.github.com/users/franz101/subscriptions
User
https://api.github.com/users/franz101
public
MEMBER
TensorArray sounds great ! Looking forward to it :)

We've had our own ExtensionArray for fixed shape tensors for a while now, hoping to see something more standardized by the arrow community.

Also super interested in the extension array for tensors of different sizes cc @mariosasko 
2022-11-21T17:57:05Z
https://github.com/huggingface/datasets/issues/5272#issuecomment-1322445244
https://api.github.com/repos/huggingface/datasets/issues/5272
IC_kwDODunzps5O0um8
https://api.github.com/repos/huggingface/datasets/issues/comments/1322445244/reactions
2022-11-21T17:57:55Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1322445244
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
[FixedShapeTensor ExtensionType](https://github.com/apache/arrow/pull/8510) was merged and will be in Arrow 12.0.0 (release is planned mid April).

2023-04-04T16:02:11Z
https://github.com/huggingface/datasets/issues/5272#issuecomment-1496232796
https://api.github.com/repos/huggingface/datasets/issues/5272
IC_kwDODunzps5ZLrNc
https://api.github.com/repos/huggingface/datasets/issues/comments/1496232796/reactions
2023-04-04T16:02:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1496232796
https://avatars.githubusercontent.com/u/54589?v=4
https://api.github.com/users/rok/events{/privacy}
https://api.github.com/users/rok/followers
https://api.github.com/users/rok/following{/other_user}
https://api.github.com/users/rok/gists{/gist_id}
https://github.com/rok
rok
MDQ6VXNlcjU0NTg5
https://api.github.com/users/rok/orgs
https://api.github.com/users/rok/received_events
https://api.github.com/users/rok/repos
https://api.github.com/users/rok/starred{/owner}{/repo}
https://api.github.com/users/rok/subscriptions
User
https://api.github.com/users/rok
public
COLLABORATOR
@rok Thanks for keeping us updated! I think it's best to introduce a new feature type that would use this extension type under the hood. I'll create an issue to discuss the design with the community in the coming days.

Also, is there a tentative time frame for the variable-shape Tensor extension type?
2023-04-07T18:29:02Z
https://github.com/huggingface/datasets/issues/5272#issuecomment-1500530792
https://api.github.com/repos/huggingface/datasets/issues/5272
IC_kwDODunzps5ZcEho
https://api.github.com/repos/huggingface/datasets/issues/comments/1500530792/reactions
2023-04-07T18:29:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1500530792
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
@mariosasko please tag me in the discussion, perhaps I can contribute.

As for the [variable shape tensor array](https://github.com/apache/arrow/issues/24868) - I'd be interested in working on it but didn't see much interest in community yet. Are you saying `huggingface/datasets` could use it?
2023-04-07T19:06:16Z
https://github.com/huggingface/datasets/issues/5272#issuecomment-1500557935
https://api.github.com/repos/huggingface/datasets/issues/5272
IC_kwDODunzps5ZcLJv
https://api.github.com/repos/huggingface/datasets/issues/comments/1500557935/reactions
2023-04-07T19:06:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1500557935
https://avatars.githubusercontent.com/u/54589?v=4
https://api.github.com/users/rok/events{/privacy}
https://api.github.com/users/rok/followers
https://api.github.com/users/rok/following{/other_user}
https://api.github.com/users/rok/gists{/gist_id}
https://github.com/rok
rok
MDQ6VXNlcjU0NTg5
https://api.github.com/users/rok/orgs
https://api.github.com/users/rok/received_events
https://api.github.com/users/rok/repos
https://api.github.com/users/rok/starred{/owner}{/repo}
https://api.github.com/users/rok/subscriptions
User
https://api.github.com/users/rok
public
NONE
pyarrow 12 is out 🎉, will have a look if I can work on it for the ExtensionArray
2023-05-03T23:20:07Z
https://github.com/huggingface/datasets/issues/5272#issuecomment-1533873357
https://api.github.com/repos/huggingface/datasets/issues/5272
IC_kwDODunzps5bbQzN
https://api.github.com/repos/huggingface/datasets/issues/comments/1533873357/reactions
2023-05-03T23:20:07Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1533873357
https://avatars.githubusercontent.com/u/18228395?v=4
https://api.github.com/users/franz101/events{/privacy}
https://api.github.com/users/franz101/followers
https://api.github.com/users/franz101/following{/other_user}
https://api.github.com/users/franz101/gists{/gist_id}
https://github.com/franz101
franz101
MDQ6VXNlcjE4MjI4Mzk1
https://api.github.com/users/franz101/orgs
https://api.github.com/users/franz101/received_events
https://api.github.com/users/franz101/repos
https://api.github.com/users/franz101/starred{/owner}{/repo}
https://api.github.com/users/franz101/subscriptions
User
https://api.github.com/users/franz101
public
COLLABORATOR
I think these two issues need to be fixed first on the Arrow side before adding the tensor feature type here: https://github.com/apache/arrow/issues/35573 and https://github.com/apache/arrow/issues/35599.

@rok We've had a couple of requests for supporting variable-shape tensors on the forum/GH, but I did not manage to find the concrete issues using the search. TF/TFDS (and PyTorch with the `nested_tensor` API) support them, so it makes sense for us to do the same eventually (the Ray project has an [extension](https://github.com/ray-project/ray/blob/42a8d1489b37243f203120899a23d919dc85bf2a/python/ray/air/util/tensor_extensions/arrow.py#L634) type to support this case)
2023-05-16T18:59:17Z
https://github.com/huggingface/datasets/issues/5272#issuecomment-1550200824
https://api.github.com/repos/huggingface/datasets/issues/5272
IC_kwDODunzps5cZi_4
https://api.github.com/repos/huggingface/datasets/issues/comments/1550200824/reactions
2023-05-16T18:59:17Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1550200824
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
> @rok We've had a couple of requests for supporting variable-shape tensors on the forum/GH, but I did not manage to find the concrete issues using the search. TF/TFDS (and PyTorch with the `nested_tensor` API) support them, so it makes sense for us to do the same eventually (the Ray project has an [extension](https://github.com/ray-project/ray/blob/42a8d1489b37243f203120899a23d919dc85bf2a/python/ray/air/util/tensor_extensions/arrow.py#L634) type to support this case)

That does make sense indeed. We should probably also be careful about memory layout to enable zero-copy interface to TF/PyTorch.
2023-05-16T20:21:50Z
https://github.com/huggingface/datasets/issues/5272#issuecomment-1550308374
https://api.github.com/repos/huggingface/datasets/issues/5272
IC_kwDODunzps5cZ9QW
https://api.github.com/repos/huggingface/datasets/issues/comments/1550308374/reactions
2023-05-16T20:21:50Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1550308374
https://avatars.githubusercontent.com/u/54589?v=4
https://api.github.com/users/rok/events{/privacy}
https://api.github.com/users/rok/followers
https://api.github.com/users/rok/following{/other_user}
https://api.github.com/users/rok/gists{/gist_id}
https://github.com/rok
rok
MDQ6VXNlcjU0NTg5
https://api.github.com/users/rok/orgs
https://api.github.com/users/rok/received_events
https://api.github.com/users/rok/repos
https://api.github.com/users/rok/starred{/owner}{/repo}
https://api.github.com/users/rok/subscriptions
User
https://api.github.com/users/rok
public
CONTRIBUTOR
So there is no way we can use [pyarrow.Tensor](https://arrow.apache.org/docs/python/generated/pyarrow.Tensor.html#pyarrow.Tensor) ?
2023-06-29T07:34:31Z
https://github.com/huggingface/datasets/issues/5272#issuecomment-1612558121
https://api.github.com/repos/huggingface/datasets/issues/5272
IC_kwDODunzps5gHa8p
https://api.github.com/repos/huggingface/datasets/issues/comments/1612558121/reactions
2023-06-29T07:34:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1612558121
https://avatars.githubusercontent.com/u/29229602?v=4
https://api.github.com/users/hfawaz/events{/privacy}
https://api.github.com/users/hfawaz/followers
https://api.github.com/users/hfawaz/following{/other_user}
https://api.github.com/users/hfawaz/gists{/gist_id}
https://github.com/hfawaz
hfawaz
MDQ6VXNlcjI5MjI5NjAy
https://api.github.com/users/hfawaz/orgs
https://api.github.com/users/hfawaz/received_events
https://api.github.com/users/hfawaz/repos
https://api.github.com/users/hfawaz/starred{/owner}{/repo}
https://api.github.com/users/hfawaz/subscriptions
User
https://api.github.com/users/hfawaz
public
MEMBER
Not with with the Arrow format, and therefore not in `datasets`. But they released a new [FixedShapeTensorArray](https://arrow.apache.org/docs/python/extending_types.html#fixed-size-tensor) to store tensors in Arrow format. We plan to support this in `datasets` at one point !
2023-06-29T09:48:37Z
https://github.com/huggingface/datasets/issues/5272#issuecomment-1612732037
https://api.github.com/repos/huggingface/datasets/issues/5272
IC_kwDODunzps5gIFaF
https://api.github.com/repos/huggingface/datasets/issues/comments/1612732037/reactions
2023-06-29T09:48:37Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1612732037
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
There is also an open issue to enable the conversion of `pyarrow.Tensor` to `pyarrow.FixedShapeTensorType`: https://github.com/apache/arrow/issues/35068. This way one could indirectly use `pyarrow.Tensor` in Arrow format.
2023-07-04T04:57:50Z
https://github.com/huggingface/datasets/issues/5272#issuecomment-1619488342
https://api.github.com/repos/huggingface/datasets/issues/5272
IC_kwDODunzps5gh25W
https://api.github.com/repos/huggingface/datasets/issues/comments/1619488342/reactions
2023-07-04T04:57:50Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1619488342
https://avatars.githubusercontent.com/u/16418547?v=4
https://api.github.com/users/AlenkaF/events{/privacy}
https://api.github.com/users/AlenkaF/followers
https://api.github.com/users/AlenkaF/following{/other_user}
https://api.github.com/users/AlenkaF/gists{/gist_id}
https://github.com/AlenkaF
AlenkaF
MDQ6VXNlcjE2NDE4NTQ3
https://api.github.com/users/AlenkaF/orgs
https://api.github.com/users/AlenkaF/received_events
https://api.github.com/users/AlenkaF/repos
https://api.github.com/users/AlenkaF/starred{/owner}{/repo}
https://api.github.com/users/AlenkaF/subscriptions
User
https://api.github.com/users/AlenkaF
public
NONE
We started a [mailing list discussion](https://lists.apache.org/thread/qc9qho0fg5ph1dns4hjq56hp4tj7rk1k) about potential `VariableShapeTensor` extension array, please check it out and give feedback. For more details here's also a PR https://github.com/apache/arrow/pull/37166.
2023-08-17T21:09:11Z
https://github.com/huggingface/datasets/issues/5272#issuecomment-1682985213
https://api.github.com/repos/huggingface/datasets/issues/5272
IC_kwDODunzps5kUFD9
https://api.github.com/repos/huggingface/datasets/issues/comments/1682985213/reactions
2023-08-17T21:09:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1682985213
https://avatars.githubusercontent.com/u/54589?v=4
https://api.github.com/users/rok/events{/privacy}
https://api.github.com/users/rok/followers
https://api.github.com/users/rok/following{/other_user}
https://api.github.com/users/rok/gists{/gist_id}
https://github.com/rok
rok
MDQ6VXNlcjU0NTg5
https://api.github.com/users/rok/orgs
https://api.github.com/users/rok/received_events
https://api.github.com/users/rok/repos
https://api.github.com/users/rok/starred{/owner}{/repo}
https://api.github.com/users/rok/subscriptions
User
https://api.github.com/users/rok
public
CONTRIBUTOR
Kindly ask what's the recent progress?
2024-11-11T03:03:16Z
https://github.com/huggingface/datasets/issues/5272#issuecomment-2467139765
https://api.github.com/repos/huggingface/datasets/issues/5272
IC_kwDODunzps6TDZC1
https://api.github.com/repos/huggingface/datasets/issues/comments/2467139765/reactions
2024-11-11T03:03:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2467139765
https://avatars.githubusercontent.com/u/11533479?v=4
https://api.github.com/users/npuichigo/events{/privacy}
https://api.github.com/users/npuichigo/followers
https://api.github.com/users/npuichigo/following{/other_user}
https://api.github.com/users/npuichigo/gists{/gist_id}
https://github.com/npuichigo
npuichigo
MDQ6VXNlcjExNTMzNDc5
https://api.github.com/users/npuichigo/orgs
https://api.github.com/users/npuichigo/received_events
https://api.github.com/users/npuichigo/repos
https://api.github.com/users/npuichigo/starred{/owner}{/repo}
https://api.github.com/users/npuichigo/subscriptions
User
https://api.github.com/users/npuichigo
public
### Feature request

I was going the discussion of converting tensors to lists.
Is there a way to leverage pyarrow's Tensors for nested arrays / embeddings?

For example:
```python
import pyarrow as pa
import numpy as np
x = np.array([[2, 2, 4], [4, 5, 100]], np.int32)
pa.Tensor.from_numpy(x, dim_names=["dim1","dim2"])
```

[Apache docs](https://arrow.apache.org/docs/python/generated/pyarrow.Tensor.html)

Maybe this belongs into the pyarrow features / repo.

### Motivation

Working with big data, we need to make sure to use the best data structures and IO out there

### Your contribution

Can try to a PR if code changes necessary