Dataset file size on disk is very large with 3D Array
MEMBER
Hi ! In the arrow file we store all the integers as uint8.
So your arrow file should weigh around `height x width x n_channels x n_images` bytes.

What feature type do your TFDS dataset have ?

If it uses a `tfds.features.Image` type, then what is stored is the encoded data (as png or jpg for example). Since these encodings are made for compression, the resulting tfrecord is smaller that the arrow file.

We are working on adding a similar feature in `datasets`: the ability to store the encoded data instead of the raw integers for images, but also for audio data. This way, arrow files will have similar sizes as tfrecords for images.
2021-03-30T16:06:36Z
https://github.com/huggingface/datasets/issues/2146#issuecomment-810387651
https://api.github.com/repos/huggingface/datasets/issues/2146
MDEyOklzc3VlQ29tbWVudDgxMDM4NzY1MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/810387651/reactions
2021-03-30T16:06:36Z
https://api.github.com/repos/huggingface/datasets/issues/comments/810387651
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Thanks for the prompt response. You're right about the encoding, I have the `tfds.features.Image` feature type you mentioned.
However, as described in the `dataset_info.json`, my dataset is made of 1479 (224x224x3) images. 1479 x 224 x 224 x 3 = 222630912 bytes which is far from the actual size 520803408 bytes. 

Anyway I look forward to the Image feature type in `datasets`. 
2021-03-30T17:01:49Z
https://github.com/huggingface/datasets/issues/2146#issuecomment-810425460
https://api.github.com/repos/huggingface/datasets/issues/2146
MDEyOklzc3VlQ29tbWVudDgxMDQyNTQ2MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/810425460/reactions
2021-03-30T17:02:52Z
https://api.github.com/repos/huggingface/datasets/issues/comments/810425460
https://avatars.githubusercontent.com/u/22685854?v=4
https://api.github.com/users/jblemoine/events{/privacy}
https://api.github.com/users/jblemoine/followers
https://api.github.com/users/jblemoine/following{/other_user}
https://api.github.com/users/jblemoine/gists{/gist_id}
https://github.com/jblemoine
jblemoine
MDQ6VXNlcjIyNjg1ODU0
https://api.github.com/users/jblemoine/orgs
https://api.github.com/users/jblemoine/received_events
https://api.github.com/users/jblemoine/repos
https://api.github.com/users/jblemoine/starred{/owner}{/repo}
https://api.github.com/users/jblemoine/subscriptions
User
https://api.github.com/users/jblemoine
public
NONE
@lhoestq I changed the data structure so I have a  2D Array feature type instead of a 3D Array by grouping the two last dimensions ( a 224x672 2D Array instead of a 224x224x3 3D Array). The file size is now  223973964 bytes, nearly half the previous size! Which is around of what I would expect.
I found similar behavior in existing `datasets` collection, when comparing black and white vs color image, for example MNIST vs CIFAR. 
2021-03-31T08:48:13Z
https://github.com/huggingface/datasets/issues/2146#issuecomment-810894127
https://api.github.com/repos/huggingface/datasets/issues/2146
MDEyOklzc3VlQ29tbWVudDgxMDg5NDEyNw==
https://api.github.com/repos/huggingface/datasets/issues/comments/810894127/reactions
2021-03-31T08:48:13Z
https://api.github.com/repos/huggingface/datasets/issues/comments/810894127
https://avatars.githubusercontent.com/u/22685854?v=4
https://api.github.com/users/jblemoine/events{/privacy}
https://api.github.com/users/jblemoine/followers
https://api.github.com/users/jblemoine/following{/other_user}
https://api.github.com/users/jblemoine/gists{/gist_id}
https://github.com/jblemoine
jblemoine
MDQ6VXNlcjIyNjg1ODU0
https://api.github.com/users/jblemoine/orgs
https://api.github.com/users/jblemoine/received_events
https://api.github.com/users/jblemoine/repos
https://api.github.com/users/jblemoine/starred{/owner}{/repo}
https://api.github.com/users/jblemoine/subscriptions
User
https://api.github.com/users/jblemoine
public
MEMBER
Interesting !
This may be because of the offsets that are stored with the array data.

Currently the offsets are stored even if the `shape` of the arrays is fixed. This was needed because of some issues with pyarrow a few months ago. I think these issues have been addressed now, so we can probably try to remove them to make the file lighter.

Ideally in your case the floats data should be 220 MB for both Array2D and Array3D
2021-03-31T10:27:16Z
https://github.com/huggingface/datasets/issues/2146#issuecomment-810957618
https://api.github.com/repos/huggingface/datasets/issues/2146
MDEyOklzc3VlQ29tbWVudDgxMDk1NzYxOA==
https://api.github.com/repos/huggingface/datasets/issues/comments/810957618/reactions
2021-03-31T10:28:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/810957618
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Yeah for sure, can you be a bit more specific about where the offset is stored in the code base ? And any reference to pyarrow issues if you have some. I would be very interested in contributing to `datasets` by trying to fix this issue. 
2021-04-01T07:41:21Z
https://github.com/huggingface/datasets/issues/2146#issuecomment-811714818
https://api.github.com/repos/huggingface/datasets/issues/2146
MDEyOklzc3VlQ29tbWVudDgxMTcxNDgxOA==
https://api.github.com/repos/huggingface/datasets/issues/comments/811714818/reactions
2021-04-01T07:49:24Z
https://api.github.com/repos/huggingface/datasets/issues/comments/811714818
https://avatars.githubusercontent.com/u/22685854?v=4
https://api.github.com/users/jblemoine/events{/privacy}
https://api.github.com/users/jblemoine/followers
https://api.github.com/users/jblemoine/following{/other_user}
https://api.github.com/users/jblemoine/gists{/gist_id}
https://github.com/jblemoine
jblemoine
MDQ6VXNlcjIyNjg1ODU0
https://api.github.com/users/jblemoine/orgs
https://api.github.com/users/jblemoine/received_events
https://api.github.com/users/jblemoine/repos
https://api.github.com/users/jblemoine/starred{/owner}{/repo}
https://api.github.com/users/jblemoine/subscriptions
User
https://api.github.com/users/jblemoine
public
MEMBER
Pyarrow has two types of lists: variable length lists and fixed size lists.
Currently we store the ArrayXD data as variable length lists. They take more disk space because they must store both actual data and offsets.
In the `datasets` code this is done here:

https://github.com/huggingface/nlp/blob/dbac87c8a083f806467f5afc4ec9b401a7e4c15c/src/datasets/features.py#L346-L352

To use a fixed length list, one should use the `list_size` argument of `pyarrow.list_()`.
I believe this would work directly modulo some changes in the numpy conversion here:

https://github.com/huggingface/nlp/blob/dbac87c8a083f806467f5afc4ec9b401a7e4c15c/src/datasets/features.py#L381-L395
2021-04-01T08:19:55Z
https://github.com/huggingface/datasets/issues/2146#issuecomment-811738069
https://api.github.com/repos/huggingface/datasets/issues/2146
MDEyOklzc3VlQ29tbWVudDgxMTczODA2OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/811738069/reactions
2021-04-01T08:19:55Z
https://api.github.com/repos/huggingface/datasets/issues/comments/811738069
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
Hi, 

I have created my own dataset using the provided dataset loading script. It is an image dataset where images are stored as 3D Array with dtype=uint8. 

The actual size on disk is surprisingly large. It takes 520 MB. Here is some info from `dataset_info.json`. 

`{
    "description": "",
    "citation": "",
    "homepage": "",
    "license": "",
    "features": {
        "image": {
            "shape": [224, 224, 3],
            "dtype": "uint8",
            "id": null,
            "_type": "Array3D",
        }
    },
    "post_processed": null,
    "supervised_keys": null,
    "builder_name": "shot_type_image_dataset",
    "config_name": "default",
    "version": {
        "version_str": "0.0.0",
        "description": null,
        "major": 0,
        "minor": 0,
        "patch": 0,
    },
    "splits": {
        "train": {
            "name": "train",
            "num_bytes": 520803408,
            "num_examples": 1479,
            "dataset_name": "shot_type_image_dataset",
        }
    },
    "download_checksums": {
        "": {
            "num_bytes": 16940447118,
            "checksum": "5854035705efe08b0ed8f3cf3da7b4d29cba9055c2d2d702c79785350d72ee03",
        }
    },
    "download_size": 16940447118,
    "post_processing_size": null,
    "dataset_size": 520803408,
    "size_in_bytes": 17461250526,
}`

I have created the same dataset with tensorflow_dataset and it takes only 125MB on disk.

I am wondering, is it normal behavior ? I understand `Datasets` uses Arrow for serialization wheres tf uses TF Records.

This might be a problem for large dataset. 

Thanks for your help. 
