Column of arrays (2D+) are using unreasonably high memory
CONTRIBUTOR
note i have tried the same code with `datasets` version 2.4.0, the outcome is the very same as described above.
2022-09-10T13:08:17Z
https://github.com/huggingface/datasets/issues/4964#issuecomment-1242724946
https://api.github.com/repos/huggingface/datasets/issues/4964
IC_kwDODunzps5KEnpS
https://api.github.com/repos/huggingface/datasets/issues/comments/1242724946/reactions
2022-09-10T13:08:17Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1242724946
https://avatars.githubusercontent.com/u/30353?v=4
https://api.github.com/users/vigsterkr/events{/privacy}
https://api.github.com/users/vigsterkr/followers
https://api.github.com/users/vigsterkr/following{/other_user}
https://api.github.com/users/vigsterkr/gists{/gist_id}
https://github.com/vigsterkr
vigsterkr
MDQ6VXNlcjMwMzUz
https://api.github.com/users/vigsterkr/orgs
https://api.github.com/users/vigsterkr/received_events
https://api.github.com/users/vigsterkr/repos
https://api.github.com/users/vigsterkr/starred{/owner}{/repo}
https://api.github.com/users/vigsterkr/subscriptions
User
https://api.github.com/users/vigsterkr
public
NONE
Seems related to issues #4623 and #4802 so it would appear this issue has been around for a few months.
2022-09-19T12:09:27Z
https://github.com/huggingface/datasets/issues/4964#issuecomment-1250934160
https://api.github.com/repos/huggingface/datasets/issues/4964
IC_kwDODunzps5Kj72Q
https://api.github.com/repos/huggingface/datasets/issues/comments/1250934160/reactions
2022-09-19T12:09:27Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1250934160
https://avatars.githubusercontent.com/u/1417499?v=4
https://api.github.com/users/Slater-Victoroff/events{/privacy}
https://api.github.com/users/Slater-Victoroff/followers
https://api.github.com/users/Slater-Victoroff/following{/other_user}
https://api.github.com/users/Slater-Victoroff/gists{/gist_id}
https://github.com/Slater-Victoroff
Slater-Victoroff
MDQ6VXNlcjE0MTc0OTk=
https://api.github.com/users/Slater-Victoroff/orgs
https://api.github.com/users/Slater-Victoroff/received_events
https://api.github.com/users/Slater-Victoroff/repos
https://api.github.com/users/Slater-Victoroff/starred{/owner}{/repo}
https://api.github.com/users/Slater-Victoroff/subscriptions
User
https://api.github.com/users/Slater-Victoroff
public
MEMBER
Hi ! `Dataset.from_dict` keeps the data in memory. You can write on disk and reload them with
```python
dataset.save_to_disk("path/to/local")
dataset = load_from_disk("path/to/local")
```
this way you'll end up with a dataset loaded from your disk using memory mapping, and it won't fill up your RAM :)

related to https://github.com/huggingface/datasets/issues/4861
2022-09-20T15:40:49Z
https://github.com/huggingface/datasets/issues/4964#issuecomment-1252543916
https://api.github.com/repos/huggingface/datasets/issues/4964
IC_kwDODunzps5KqE2s
https://api.github.com/repos/huggingface/datasets/issues/comments/1252543916/reactions
2022-09-20T15:42:05Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1252543916
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
@lhoestq thnx for getting back to me! i've tested the suggested method, but unfortunately the memory consumption is the very same:

```
from datasets import Dataset, Features, Array2D, Array3D, load_from_disk
import numpy as np

column_name = "a"
array_shape = (64, 64, 3)

data = np.random.random((10000,) + array_shape)
dataset = Dataset.from_dict({column_name: data}, features=Features({column_name: Array3D(shape=array_shape, dtype="float64")}))
dataset.save_to_disk("foo")

foo_db = load_from_disk("foo")
colum_value = foo_db[column_name]
```

the very same happens when you create the dataset, but dont specify the feature type.

i've tried running this on different envs (macOS, linux) and it's behaving the very same way.
2022-09-21T08:27:27Z
https://github.com/huggingface/datasets/issues/4964#issuecomment-1253374670
https://api.github.com/repos/huggingface/datasets/issues/4964
IC_kwDODunzps5KtPrO
https://api.github.com/repos/huggingface/datasets/issues/comments/1253374670/reactions
2022-09-21T08:51:37Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1253374670
https://avatars.githubusercontent.com/u/30353?v=4
https://api.github.com/users/vigsterkr/events{/privacy}
https://api.github.com/users/vigsterkr/followers
https://api.github.com/users/vigsterkr/following{/other_user}
https://api.github.com/users/vigsterkr/gists{/gist_id}
https://github.com/vigsterkr
vigsterkr
MDQ6VXNlcjMwMzUz
https://api.github.com/users/vigsterkr/orgs
https://api.github.com/users/vigsterkr/received_events
https://api.github.com/users/vigsterkr/repos
https://api.github.com/users/vigsterkr/starred{/owner}{/repo}
https://api.github.com/users/vigsterkr/subscriptions
User
https://api.github.com/users/vigsterkr
public
MEMBER
When you call `colum_value = foo_db[column_name]`, you load the full column in memory.

If you want to avoid filling up your memory, you can access chunks of data instead
```python
embeddings = dataset[i:i + chunk_size]["embeddings"]
```
2022-09-21T10:16:55Z
https://github.com/huggingface/datasets/issues/4964#issuecomment-1253498740
https://api.github.com/repos/huggingface/datasets/issues/4964
IC_kwDODunzps5Ktt90
https://api.github.com/repos/huggingface/datasets/issues/comments/1253498740/reactions
2022-09-21T10:16:55Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1253498740
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
@lhoestq yeah that's intentional, i.e. i really want to load the whole column into the memory. but as said above there's an unreasonable amount of overhead for the memory. the np array itself is using about 1G of memory:
```
>>> getsizeof(data)/1024/1024
937.5001525878906
```
that accessing of column above is using 10x memory compared to the original numpy array.
2022-09-21T10:22:31Z
https://github.com/huggingface/datasets/issues/4964#issuecomment-1253504167
https://api.github.com/repos/huggingface/datasets/issues/4964
IC_kwDODunzps5KtvSn
https://api.github.com/repos/huggingface/datasets/issues/comments/1253504167/reactions
2022-09-21T10:22:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1253504167
https://avatars.githubusercontent.com/u/30353?v=4
https://api.github.com/users/vigsterkr/events{/privacy}
https://api.github.com/users/vigsterkr/followers
https://api.github.com/users/vigsterkr/following{/other_user}
https://api.github.com/users/vigsterkr/gists{/gist_id}
https://github.com/vigsterkr
vigsterkr
MDQ6VXNlcjMwMzUz
https://api.github.com/users/vigsterkr/orgs
https://api.github.com/users/vigsterkr/received_events
https://api.github.com/users/vigsterkr/repos
https://api.github.com/users/vigsterkr/starred{/owner}{/repo}
https://api.github.com/users/vigsterkr/subscriptions
User
https://api.github.com/users/vigsterkr
public
MEMBER
The dataset must be twice as big because we use regular arrow ListArray under the hood and not FixedSizeListArray. Basically we store unnecessary offsets.

And this should affect performance as well. When we developed this, FixedSizeListArray still had some issues but they should be resolved on the PyArrow side now
2022-09-21T13:24:37Z
https://github.com/huggingface/datasets/issues/4964#issuecomment-1253707147
https://api.github.com/repos/huggingface/datasets/issues/4964
IC_kwDODunzps5Kug2L
https://api.github.com/repos/huggingface/datasets/issues/comments/1253707147/reactions
2022-09-21T13:24:37Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1253707147
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
A doubling would be fine. My very basic understanding of PyArrow is that using ListArray is probably related to the issue though. Using a multi-dimensional array in datasets is storing everything as strange nested 1d object arrays, which I imagine is creating the massive overhead.

I think it should be a PyArrow Tensor, no?
2022-09-21T15:21:54Z
https://github.com/huggingface/datasets/issues/4964#issuecomment-1253864451
https://api.github.com/repos/huggingface/datasets/issues/4964
IC_kwDODunzps5KvHQD
https://api.github.com/repos/huggingface/datasets/issues/comments/1253864451/reactions
2022-09-21T15:21:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1253864451
https://avatars.githubusercontent.com/u/1417499?v=4
https://api.github.com/users/Slater-Victoroff/events{/privacy}
https://api.github.com/users/Slater-Victoroff/followers
https://api.github.com/users/Slater-Victoroff/following{/other_user}
https://api.github.com/users/Slater-Victoroff/gists{/gist_id}
https://github.com/Slater-Victoroff
Slater-Victoroff
MDQ6VXNlcjE0MTc0OTk=
https://api.github.com/users/Slater-Victoroff/orgs
https://api.github.com/users/Slater-Victoroff/received_events
https://api.github.com/users/Slater-Victoroff/repos
https://api.github.com/users/Slater-Victoroff/starred{/owner}{/repo}
https://api.github.com/users/Slater-Victoroff/subscriptions
User
https://api.github.com/users/Slater-Victoroff
public
MEMBER
PyArrow tensors are not part of the Arrow format AFAIK:

> There is no direct support in the arrow columnar format to store Tensors as column values.

source: https://github.com/apache/arrow/issues/4802#issuecomment-508494694
2022-09-21T15:48:27Z
https://github.com/huggingface/datasets/issues/4964#issuecomment-1253897502
https://api.github.com/repos/huggingface/datasets/issues/4964
IC_kwDODunzps5KvPUe
https://api.github.com/repos/huggingface/datasets/issues/comments/1253897502/reactions
2022-09-21T16:02:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1253897502
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
That's... unfortunate. I didn't realize that.
2022-09-22T18:29:22Z
https://github.com/huggingface/datasets/issues/4964#issuecomment-1255398274
https://api.github.com/repos/huggingface/datasets/issues/4964
IC_kwDODunzps5K09uC
https://api.github.com/repos/huggingface/datasets/issues/comments/1255398274/reactions
2022-09-22T18:29:22Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1255398274
https://avatars.githubusercontent.com/u/1417499?v=4
https://api.github.com/users/Slater-Victoroff/events{/privacy}
https://api.github.com/users/Slater-Victoroff/followers
https://api.github.com/users/Slater-Victoroff/following{/other_user}
https://api.github.com/users/Slater-Victoroff/gists{/gist_id}
https://github.com/Slater-Victoroff
Slater-Victoroff
MDQ6VXNlcjE0MTc0OTk=
https://api.github.com/users/Slater-Victoroff/orgs
https://api.github.com/users/Slater-Victoroff/received_events
https://api.github.com/users/Slater-Victoroff/repos
https://api.github.com/users/Slater-Victoroff/starred{/owner}{/repo}
https://api.github.com/users/Slater-Victoroff/subscriptions
User
https://api.github.com/users/Slater-Victoroff
public
## Describe the bug
When trying to store `Array2D, Array3D, etc` as column values in a dataset, accessing that column (or creating depending on how you create it, see code below) will cause more than 10 fold of memory usage.

## Steps to reproduce the bug
```python
from datasets import Dataset, Features, Array2D, Array3D
import numpy as np

column_name = "a"
array_shape = (64, 64, 3)

data = np.random.random((10000,) + array_shape)
dataset = Dataset.from_dict({column_name: data}, features=Features({column_name: Array3D(shape=array_shape, dtype="float64")}))
```

the code above will use about 10Gb of RAM while constructing the `dataset` object.

The code below will use roughly the same amount of memory (and time) when trying to actually access the data itself of that column.
```python
from datasets import Dataset
import numpy as np

column_name = "a"
array_shape = (64, 64, 3)

data = np.random.random((10000,) + array_shape)
dataset = Dataset.from_dict({column_name: data})
dataset[column_name]
```

## Expected results
Some memory overhead, but not like as it is now and certainly not an overhead of such runtime that is currently happening.

## Actual results
Enormous memory- and runtime overhead.

## Environment info
- `datasets` version: 2.3.2
- Platform: macOS-12.5.1-arm64-arm-64bit
- Python version: 3.8.13
- PyArrow version: 9.0.0
- Pandas version: 1.4.4