Errror when saving to disk a dataset of images
CONTRIBUTOR
Looks like as long as the number of shards makes a batch lower than 1000 images it works. In my training set I have 40K images. If I use `num_shards=40` (batch of 1000 images) I get the error, but if I update it to `num_shards=50` (batch of 800 images) it works.

I will be happy to share my dataset privately if it can help to better debug.
2023-04-07T15:54:29Z
https://github.com/huggingface/datasets/issues/5717#issuecomment-1500410424
https://api.github.com/repos/huggingface/datasets/issues/5717
IC_kwDODunzps5ZbnI4
https://api.github.com/repos/huggingface/datasets/issues/comments/1500410424/reactions
2023-04-07T15:54:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1500410424
https://avatars.githubusercontent.com/u/959590?v=4
https://api.github.com/users/jplu/events{/privacy}
https://api.github.com/users/jplu/followers
https://api.github.com/users/jplu/following{/other_user}
https://api.github.com/users/jplu/gists{/gist_id}
https://github.com/jplu
jplu
MDQ6VXNlcjk1OTU5MA==
https://api.github.com/users/jplu/orgs
https://api.github.com/users/jplu/received_events
https://api.github.com/users/jplu/repos
https://api.github.com/users/jplu/starred{/owner}{/repo}
https://api.github.com/users/jplu/subscriptions
User
https://api.github.com/users/jplu
public
COLLABORATOR
Hi! I didn't manage to reproduce this behavior, so sharing the dataset with us would help a lot. 

> My dataset is around 50K images, is this error might be due to a bad image?

This shouldn't be the case as we save raw data to disk without decoding it.
2023-04-14T17:32:22Z
https://github.com/huggingface/datasets/issues/5717#issuecomment-1508999941
https://api.github.com/repos/huggingface/datasets/issues/5717
IC_kwDODunzps5Z8YMF
https://api.github.com/repos/huggingface/datasets/issues/comments/1508999941/reactions
2023-04-14T17:32:22Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1508999941
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
CONTRIBUTOR
OK, thanks! The dataset is currently hosted on a gcs bucket. How would you like to proceed for sharing the link? 
2023-04-14T17:52:36Z
https://github.com/huggingface/datasets/issues/5717#issuecomment-1509021090
https://api.github.com/repos/huggingface/datasets/issues/5717
IC_kwDODunzps5Z8dWi
https://api.github.com/repos/huggingface/datasets/issues/comments/1509021090/reactions
2023-04-14T17:52:36Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1509021090
https://avatars.githubusercontent.com/u/959590?v=4
https://api.github.com/users/jplu/events{/privacy}
https://api.github.com/users/jplu/followers
https://api.github.com/users/jplu/following{/other_user}
https://api.github.com/users/jplu/gists{/gist_id}
https://github.com/jplu
jplu
MDQ6VXNlcjk1OTU5MA==
https://api.github.com/users/jplu/orgs
https://api.github.com/users/jplu/received_events
https://api.github.com/users/jplu/repos
https://api.github.com/users/jplu/starred{/owner}{/repo}
https://api.github.com/users/jplu/subscriptions
User
https://api.github.com/users/jplu
public
COLLABORATOR
You could follow [this](https://cloud.google.com/storage/docs/collaboration#browser) procedure or upload the dataset to Google Drive (50K images is not that much unless high-res) and send me an email with the link.
2023-04-14T18:55:03Z
https://github.com/huggingface/datasets/issues/5717#issuecomment-1509083919
https://api.github.com/repos/huggingface/datasets/issues/5717
IC_kwDODunzps5Z8ssP
https://api.github.com/repos/huggingface/datasets/issues/comments/1509083919/reactions
2023-04-14T18:55:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1509083919
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
CONTRIBUTOR
Thanks @mariosasko. I just sent you the GDrive link.
2023-04-17T09:47:46Z
https://github.com/huggingface/datasets/issues/5717#issuecomment-1511032677
https://api.github.com/repos/huggingface/datasets/issues/5717
IC_kwDODunzps5aEIdl
https://api.github.com/repos/huggingface/datasets/issues/comments/1511032677/reactions
2023-04-17T09:47:46Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1511032677
https://avatars.githubusercontent.com/u/959590?v=4
https://api.github.com/users/jplu/events{/privacy}
https://api.github.com/users/jplu/followers
https://api.github.com/users/jplu/following{/other_user}
https://api.github.com/users/jplu/gists{/gist_id}
https://github.com/jplu
jplu
MDQ6VXNlcjk1OTU5MA==
https://api.github.com/users/jplu/orgs
https://api.github.com/users/jplu/received_events
https://api.github.com/users/jplu/repos
https://api.github.com/users/jplu/starred{/owner}{/repo}
https://api.github.com/users/jplu/subscriptions
User
https://api.github.com/users/jplu
public
COLLABORATOR
Thanks @jplu! I managed to reproduce the `TypeError` - it stems from [this](https://github.com/huggingface/datasets/blob/e3f4f124a1b118a5bfff5bae76b25a68aedbebbc/src/datasets/features/image.py#L258-L264) line, which can return a `ChunkedArray` (its mask is also chunked then, which Arrow does not allow) when the embedded data is too big to fit in a standard `Array`.

I'm working on a fix.
2023-05-09T17:14:49Z
https://github.com/huggingface/datasets/issues/5717#issuecomment-1540563141
https://api.github.com/repos/huggingface/datasets/issues/5717
IC_kwDODunzps5b0yDF
https://api.github.com/repos/huggingface/datasets/issues/comments/1540563141/reactions
2023-05-09T17:14:49Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1540563141
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
COLLABORATOR
@yairl-dn You should be able to bypass this issue by reducing `datasets.config.DEFAULT_MAX_BATCH_SIZE` (1000 by default)

In Datasets 3.0, the Image storage format will be simplified, so this should be easier to fix then.
2023-09-25T13:43:14Z
https://github.com/huggingface/datasets/issues/5717#issuecomment-1733740819
https://api.github.com/repos/huggingface/datasets/issues/5717
IC_kwDODunzps5nVskT
https://api.github.com/repos/huggingface/datasets/issues/comments/1733740819/reactions
2023-09-25T13:43:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1733740819
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
The same error occurs with my save_to_disk() of Audio() items. I still get it with:
```python
import datasets
datasets.config.DEFAULT_MAX_BATCH_SIZE=35
from datasets import Features, Array2D, Value, Dataset, Sequence, Audio
```

```
Saving the dataset (41/47 shards):  88%|██████████████████████████████████████████▉      | 297/339 [01:21<00:11,  3.65 examples/s]
Traceback (most recent call last):
File "/mnt/ddrive/prj/voice/voice-training-dataset-create/./dataset.py", line 155, in <module>
create_dataset(args)
File "/mnt/ddrive/prj/voice/voice-training-dataset-create/./dataset.py", line 137, in create_dataset
hf_dataset.save_to_disk(args.outds)
File "/home/j/src/py/datasets/src/datasets/arrow_dataset.py", line 1532, in save_to_disk
for job_id, done, content in Dataset._save_to_disk_single(**kwargs):
File "/home/j/src/py/datasets/src/datasets/arrow_dataset.py", line 1563, in _save_to_disk_single
writer.write_table(pa_table)
File "/home/j/src/py/datasets/src/datasets/arrow_writer.py", line 574, in write_table
pa_table = embed_table_storage(pa_table)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/j/src/py/datasets/src/datasets/table.py", line 2307, in embed_table_storage
arrays = [
^
File "/home/j/src/py/datasets/src/datasets/table.py", line 2308, in <listcomp>
embed_array_storage(table[name], feature) if require_storage_embed(feature) else table[name]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/j/src/py/datasets/src/datasets/table.py", line 1831, in wrapper
return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/j/src/py/datasets/src/datasets/table.py", line 1831, in <listcomp>
return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/j/src/py/datasets/src/datasets/table.py", line 2177, in embed_array_storage
return feature.embed_storage(array)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/j/src/py/datasets/src/datasets/features/audio.py", line 276, in embed_storage
storage = pa.StructArray.from_arrays([bytes_array, path_array], ["bytes", "path"], mask=bytes_array.is_null())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "pyarrow/array.pxi", line 2850, in pyarrow.lib.StructArray.from_arrays
File "pyarrow/array.pxi", line 3290, in pyarrow.lib.c_mask_inverted_from_obj
TypeError: Mask must be a pyarrow.Array of type boolean
```
2023-11-08T10:39:52Z
https://github.com/huggingface/datasets/issues/5717#issuecomment-1801596120
https://api.github.com/repos/huggingface/datasets/issues/5717
IC_kwDODunzps5rYizY
https://api.github.com/repos/huggingface/datasets/issues/comments/1801596120/reactions
2023-11-08T11:08:18Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1801596120
https://avatars.githubusercontent.com/u/20318973?v=4
https://api.github.com/users/jaggzh/events{/privacy}
https://api.github.com/users/jaggzh/followers
https://api.github.com/users/jaggzh/following{/other_user}
https://api.github.com/users/jaggzh/gists{/gist_id}
https://github.com/jaggzh
jaggzh
MDQ6VXNlcjIwMzE4OTcz
https://api.github.com/users/jaggzh/orgs
https://api.github.com/users/jaggzh/received_events
https://api.github.com/users/jaggzh/repos
https://api.github.com/users/jaggzh/starred{/owner}{/repo}
https://api.github.com/users/jaggzh/subscriptions
User
https://api.github.com/users/jaggzh
public
NONE
Similar to @jaggzh, setting `datasets.config.DEFAULT_MAX_BATCH_SIZE` did not help in my case (same error here but for different dataset: https://github.com/Stanford-AIMI/RRG24/issues/2).

This is also reproducible with this open dataset: https://huggingface.co/datasets/nlphuji/winogavil/discussions/1

Here's some code to do so:
```python
import datasets

datasets.config.DEFAULT_MAX_BATCH_SIZE = 1

from datasets import load_dataset

ds = load_dataset("nlphuji/winogavil")

ds.save_to_disk("temp")
```

I've done some more debugging with `datasets==2.18.0` (which incorporates PR #6283 as suggested by @lhoestq in the above dataset discussion), and it seems like the culprit might now be these lines: https://github.com/huggingface/datasets/blob/ca8409a8bec4508255b9c3e808d0751eb1005260/src/datasets/table.py#L2111-L2115

From what I understand (and apologies I'm new to pyarrow), for an Image or Audio feature, these lines recursively call `embed_array_storage` for a list of either feature, ending up in the feature's `embed_storage` function. For all values in the list, `embed_storage` reads the bytes if they're not already loaded. The issue is the list being passed to the first recursive call is `array.values` which are the underlying values of `array` regardless of `array`'s slicing (as influenced by parameters such as `datasets.config.DEFAULT_MAX_BATCH_SIZE`). This results in the same overflowing list of bytes that result in the ChunkedArray being returned in `embed_storage`. Even if the array weren't to overflow and this code ran without throwing an exception, it still seems incorrect to load all values if you ultimately only want some subset with `ListArray.from_arrays(offsets, values)`; it seems some wasted effort if those values thrown out will get loaded again in the next batch and vice versa for the current batch of values during later batches.

Maybe there's a fix where you could pass a mask to `embed_storage` such that it only loads the values you ultimately want for the current batch? Curious to see if you agree with this diagnosis of the problem and if you think this fix is viable @mariosasko?
2024-03-12T03:37:25Z
https://github.com/huggingface/datasets/issues/5717#issuecomment-1990071181
https://api.github.com/repos/huggingface/datasets/issues/5717
IC_kwDODunzps52nhON
https://api.github.com/repos/huggingface/datasets/issues/comments/1990071181/reactions
2024-03-12T03:41:22Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1990071181
https://avatars.githubusercontent.com/u/26208374?v=4
https://api.github.com/users/StevenSong/events{/privacy}
https://api.github.com/users/StevenSong/followers
https://api.github.com/users/StevenSong/following{/other_user}
https://api.github.com/users/StevenSong/gists{/gist_id}
https://github.com/StevenSong
StevenSong
MDQ6VXNlcjI2MjA4Mzc0
https://api.github.com/users/StevenSong/orgs
https://api.github.com/users/StevenSong/received_events
https://api.github.com/users/StevenSong/repos
https://api.github.com/users/StevenSong/starred{/owner}{/repo}
https://api.github.com/users/StevenSong/subscriptions
User
https://api.github.com/users/StevenSong
public
NONE
Would be nice if they have something similar to Dagshub's S3 sync; it worked like a charm for my bigger datasets.
2024-03-12T03:40:51Z
https://github.com/huggingface/datasets/issues/5717#issuecomment-1990086854
https://api.github.com/repos/huggingface/datasets/issues/5717
IC_kwDODunzps52nlDG
https://api.github.com/repos/huggingface/datasets/issues/comments/1990086854/reactions
2024-03-12T03:40:51Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1990086854
https://avatars.githubusercontent.com/u/3602968?v=4
https://api.github.com/users/yairl/events{/privacy}
https://api.github.com/users/yairl/followers
https://api.github.com/users/yairl/following{/other_user}
https://api.github.com/users/yairl/gists{/gist_id}
https://github.com/yairl
yairl
MDQ6VXNlcjM2MDI5Njg=
https://api.github.com/users/yairl/orgs
https://api.github.com/users/yairl/received_events
https://api.github.com/users/yairl/repos
https://api.github.com/users/yairl/starred{/owner}{/repo}
https://api.github.com/users/yairl/subscriptions
User
https://api.github.com/users/yairl
public
NONE
I guess also the proposed masking solution simply enables `datasets.config.DEFAULT_MAX_BATCH_SIZE` by reducing the number of elements loaded, it does not address the underlying problem of trying to load all the images as bytes into a pyarrow array.

I'm happy to turn this into an actual PR but here's what I've implemented locally at `tables.py:embed_array_storage` to fix the above test case (`nlphuji/winogavil`) and my own use case:
```python
    elif pa.types.is_list(array.type):
        # feature must be either [subfeature] or Sequence(subfeature)
        # Merge offsets with the null bitmap to avoid the "Null bitmap with offsets slice not supported" ArrowNotImplementedError
        array_offsets = _combine_list_array_offsets_with_mask(array)

        # mask underlying struct array so array_values.to_pylist()
        # fills None (see feature.embed_storage)
        idxs = np.arange(len(array.values))
        idxs = pa.ListArray.from_arrays(array_offsets, idxs).flatten()
        mask = np.ones(len(array.values)).astype(bool)
        mask[idxs] = False
        mask = pa.array(mask)
        # indexing 0 might be problematic but not sure
        # how else to get arbitrary keys from a struct array
        array_keys = array.values[0].keys()
        # is array.values always a struct array?
        array_values = pa.StructArray.from_arrays(
            arrays=[array.values.field(k) for k in array_keys],
            names=array_keys,
            mask=mask,
        )
        if isinstance(feature, list):
            return pa.ListArray.from_arrays(array_offsets, _e(array_values, feature[0]))
        if isinstance(feature, Sequence) and feature.length == -1:
            return pa.ListArray.from_arrays(array_offsets, _e(array_values, feature.feature))
```

Again though I'm new to pyarrow so this might not be the cleanest implementation, also I'm really not sure if there are other cases where this solution doesn't work. Would love to get some feedback from the hf folks!
2024-03-12T14:14:24Z
https://github.com/huggingface/datasets/issues/5717#issuecomment-1991751814
https://api.github.com/repos/huggingface/datasets/issues/5717
IC_kwDODunzps52t7iG
https://api.github.com/repos/huggingface/datasets/issues/comments/1991751814/reactions
2024-03-12T14:15:59Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1991751814
https://avatars.githubusercontent.com/u/26208374?v=4
https://api.github.com/users/StevenSong/events{/privacy}
https://api.github.com/users/StevenSong/followers
https://api.github.com/users/StevenSong/following{/other_user}
https://api.github.com/users/StevenSong/gists{/gist_id}
https://github.com/StevenSong
StevenSong
MDQ6VXNlcjI2MjA4Mzc0
https://api.github.com/users/StevenSong/orgs
https://api.github.com/users/StevenSong/received_events
https://api.github.com/users/StevenSong/repos
https://api.github.com/users/StevenSong/starred{/owner}{/repo}
https://api.github.com/users/StevenSong/subscriptions
User
https://api.github.com/users/StevenSong
public
NONE
I have the same issue, with an audio dataset where file sizes vary significantly (~0.2-200 mb). Reducing `datasets.config.DEFAULT_MAX_BATCH_SIZE` doesn't help.
2024-03-18T08:26:18Z
https://github.com/huggingface/datasets/issues/5717#issuecomment-2003183683
https://api.github.com/repos/huggingface/datasets/issues/5717
IC_kwDODunzps53ZihD
https://api.github.com/repos/huggingface/datasets/issues/comments/2003183683/reactions
2024-03-18T08:26:18Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2003183683
https://avatars.githubusercontent.com/u/38854604?v=4
https://api.github.com/users/AJDERS/events{/privacy}
https://api.github.com/users/AJDERS/followers
https://api.github.com/users/AJDERS/following{/other_user}
https://api.github.com/users/AJDERS/gists{/gist_id}
https://github.com/AJDERS
AJDERS
MDQ6VXNlcjM4ODU0NjA0
https://api.github.com/users/AJDERS/orgs
https://api.github.com/users/AJDERS/received_events
https://api.github.com/users/AJDERS/repos
https://api.github.com/users/AJDERS/starred{/owner}{/repo}
https://api.github.com/users/AJDERS/subscriptions
User
https://api.github.com/users/AJDERS
public
NONE
Still the problem is occured.
Huggingface is sucks 🤮🤮🤮🤮
2024-07-04T07:25:37Z
https://github.com/huggingface/datasets/issues/5717#issuecomment-2208294965
https://api.github.com/repos/huggingface/datasets/issues/5717
IC_kwDODunzps6Dn-g1
https://api.github.com/repos/huggingface/datasets/issues/comments/2208294965/reactions
2024-07-04T07:25:37Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2208294965
https://avatars.githubusercontent.com/u/89951934?v=4
https://api.github.com/users/20141888/events{/privacy}
https://api.github.com/users/20141888/followers
https://api.github.com/users/20141888/following{/other_user}
https://api.github.com/users/20141888/gists{/gist_id}
https://github.com/20141888
20141888
MDQ6VXNlcjg5OTUxOTM0
https://api.github.com/users/20141888/orgs
https://api.github.com/users/20141888/received_events
https://api.github.com/users/20141888/repos
https://api.github.com/users/20141888/starred{/owner}{/repo}
https://api.github.com/users/20141888/subscriptions
User
https://api.github.com/users/20141888
public
NONE
Came across this issue myself, with the same symptoms and reasons as everyone else; `pa.array` is returning a `ChunkedArray` in `features.audio.Audio.embed_storage` for my audio which varies between ~1MB and ~10MB in size.

I would rather remove a troublesome file from my dataset than have to switch off this library, but it would be difficult to identify which file(s) caused the issue, and it may just shift the issue down to another shard or another file anyway. So, I took the path of least resistance and simply **dropped** anything beyond the first chunk when this issue occurred, and added a warning to indicate what was dropped.

In the end I lost **one** file out of 105,024 samples and was able to complete the 1,479 shard dataset after only the one issue on shard 228.

While this is certainly not an ideal solution, it does represent a much better user experience, and was acceptable for my use case. I'm going to test the Image portion and then open a pull request to propose this "lossy" behavior become the way these edge cases are handled (maybe behind an environment flag?) until someone like @mariosasko or others can formulate a more holistic solution.

My work-in-progress "fix": https://github.com/huggingface/datasets/compare/main...painebenjamin:datasets:main (https://github.com/painebenjamin/datasets)
2024-09-05T23:49:10Z
https://github.com/huggingface/datasets/issues/5717#issuecomment-2332911206
https://api.github.com/repos/huggingface/datasets/issues/5717
IC_kwDODunzps6LDWZm
https://api.github.com/repos/huggingface/datasets/issues/comments/2332911206/reactions
2024-09-05T23:56:53Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2332911206
https://avatars.githubusercontent.com/u/57536852?v=4
https://api.github.com/users/painebenjamin/events{/privacy}
https://api.github.com/users/painebenjamin/followers
https://api.github.com/users/painebenjamin/following{/other_user}
https://api.github.com/users/painebenjamin/gists{/gist_id}
https://github.com/painebenjamin
painebenjamin
MDQ6VXNlcjU3NTM2ODUy
https://api.github.com/users/painebenjamin/orgs
https://api.github.com/users/painebenjamin/received_events
https://api.github.com/users/painebenjamin/repos
https://api.github.com/users/painebenjamin/starred{/owner}{/repo}
https://api.github.com/users/painebenjamin/subscriptions
User
https://api.github.com/users/painebenjamin
public
MEMBER
Another option could be to use `pa.large_binary` instead of `pa.binary` in certain cases ?
2024-09-06T13:09:00Z
https://github.com/huggingface/datasets/issues/5717#issuecomment-2334015546
https://api.github.com/repos/huggingface/datasets/issues/5717
IC_kwDODunzps6LHkA6
https://api.github.com/repos/huggingface/datasets/issues/comments/2334015546/reactions
2024-09-06T13:09:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2334015546
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
For my large audio dataset, what seems to work for me is to locally change `pa_type` to `pa.large_binary` in both

https://github.com/huggingface/datasets/blob/01f91bae037c98f2e05456287bab21470adb8f07/src/datasets/features/audio.py#L71

and

https://github.com/huggingface/datasets/blob/01f91bae037c98f2e05456287bab21470adb8f07/src/datasets/features/audio.py#L270

prior to uploading the dataset. Before downloading it, I just remove both changes to make sure any user with latest `datasets` can use it.

As a side note, the other proposed workarounds did not work for me.
2024-11-06T14:51:24Z
https://github.com/huggingface/datasets/issues/5717#issuecomment-2459959869
https://api.github.com/repos/huggingface/datasets/issues/5717
IC_kwDODunzps6SoAI9
https://api.github.com/repos/huggingface/datasets/issues/comments/2459959869/reactions
2024-11-06T14:52:13Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2459959869
https://avatars.githubusercontent.com/u/39233597?v=4
https://api.github.com/users/fdschmidt93/events{/privacy}
https://api.github.com/users/fdschmidt93/followers
https://api.github.com/users/fdschmidt93/following{/other_user}
https://api.github.com/users/fdschmidt93/gists{/gist_id}
https://github.com/fdschmidt93
fdschmidt93
MDQ6VXNlcjM5MjMzNTk3
https://api.github.com/users/fdschmidt93/orgs
https://api.github.com/users/fdschmidt93/received_events
https://api.github.com/users/fdschmidt93/repos
https://api.github.com/users/fdschmidt93/starred{/owner}{/repo}
https://api.github.com/users/fdschmidt93/subscriptions
User
https://api.github.com/users/fdschmidt93
public
NONE
Hey @fdschmidt93 I am not sure to follow. Can users downloading your dataset from the hub read it if you created the files with large_binary? It sounds like it will not be casted properly for them? 
2024-12-07T18:56:25Z
https://github.com/huggingface/datasets/issues/5717#issuecomment-2525278248
https://api.github.com/repos/huggingface/datasets/issues/5717
IC_kwDODunzps6WhLAo
https://api.github.com/repos/huggingface/datasets/issues/comments/2525278248/reactions
2024-12-07T18:56:25Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2525278248
https://avatars.githubusercontent.com/u/11910731?v=4
https://api.github.com/users/TParcollet/events{/privacy}
https://api.github.com/users/TParcollet/followers
https://api.github.com/users/TParcollet/following{/other_user}
https://api.github.com/users/TParcollet/gists{/gist_id}
https://github.com/TParcollet
TParcollet
MDQ6VXNlcjExOTEwNzMx
https://api.github.com/users/TParcollet/orgs
https://api.github.com/users/TParcollet/received_events
https://api.github.com/users/TParcollet/repos
https://api.github.com/users/TParcollet/starred{/owner}{/repo}
https://api.github.com/users/TParcollet/subscriptions
User
https://api.github.com/users/TParcollet
public
CONTRIBUTOR
Yes, that should work. In full detail -

I have two separate conda environments. 

1. The one I prepare the data with for which I apply the above changes.
2. Another one I actually run my experiments with that uses latest available `datasets` from pip

This seems to work just fine. More concretely, I'm downloading my own data uploaded with environment 1 from a private HF datasets repo in environment 2 (i.e., `load_dataset(...)`) and run the experiments.

E: The private HF repo I was referring to now is public: https://huggingface.co/datasets/WueNLP/belebele-fleurs
2024-12-07T19:18:46Z
https://github.com/huggingface/datasets/issues/5717#issuecomment-2525284128
https://api.github.com/repos/huggingface/datasets/issues/5717
IC_kwDODunzps6WhMcg
https://api.github.com/repos/huggingface/datasets/issues/comments/2525284128/reactions
2024-12-15T09:39:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2525284128
https://avatars.githubusercontent.com/u/39233597?v=4
https://api.github.com/users/fdschmidt93/events{/privacy}
https://api.github.com/users/fdschmidt93/followers
https://api.github.com/users/fdschmidt93/following{/other_user}
https://api.github.com/users/fdschmidt93/gists{/gist_id}
https://github.com/fdschmidt93
fdschmidt93
MDQ6VXNlcjM5MjMzNTk3
https://api.github.com/users/fdschmidt93/orgs
https://api.github.com/users/fdschmidt93/received_events
https://api.github.com/users/fdschmidt93/repos
https://api.github.com/users/fdschmidt93/starred{/owner}{/repo}
https://api.github.com/users/fdschmidt93/subscriptions
User
https://api.github.com/users/fdschmidt93
public
NONE
Interesting ... so it's not even relevant at reading time, only writting ... Thanks I'll try this out.
2024-12-07T19:26:58Z
https://github.com/huggingface/datasets/issues/5717#issuecomment-2525286293
https://api.github.com/repos/huggingface/datasets/issues/5717
IC_kwDODunzps6WhM-V
https://api.github.com/repos/huggingface/datasets/issues/comments/2525286293/reactions
2024-12-07T19:26:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2525286293
https://avatars.githubusercontent.com/u/11910731?v=4
https://api.github.com/users/TParcollet/events{/privacy}
https://api.github.com/users/TParcollet/followers
https://api.github.com/users/TParcollet/following{/other_user}
https://api.github.com/users/TParcollet/gists{/gist_id}
https://github.com/TParcollet
TParcollet
MDQ6VXNlcjExOTEwNzMx
https://api.github.com/users/TParcollet/orgs
https://api.github.com/users/TParcollet/received_events
https://api.github.com/users/TParcollet/repos
https://api.github.com/users/TParcollet/starred{/owner}{/repo}
https://api.github.com/users/TParcollet/subscriptions
User
https://api.github.com/users/TParcollet
public
NONE
I'm experiencing the same issue with the image dataset. Has this problem been resolved? Neither reducing the number of images per batch nor decreasing the DEFAULT_MAX_BATCH_SIZE has solved the issue.

What I discovered is that in [line 272](https://github.com/huggingface/datasets/blob/3a4e74a9ace62ecd5c9cde7dcb6bcabd65cc7857/src/datasets/features/image.py#L272), when the storage length becomes large (probably over 1000), it returns a chunked array, which causes an error. 

 Therefore, I tried to explicitly flatten it using `.combine_chunks()`, but encountered errors like `pyarrow.lib.ArrowInvalid: offset overflow while concatenating arrays`. 
```python
        bytes_array = pa.array(
            [
                (path_to_bytes(x["path"]) if x["bytes"] is None else x["bytes"]) if x is not None else None
                for x in storage.to_pylist()
            ],
            type=pa.binary(),
        ).combine_chunks() # <- HERE
```

This makes me feel there are limitations in processing images with arrow.

When constructing an image dataset, I think storing image paths as strings might be the most straightforward bypass rather than using `Image()`. I'm curious if there would be any performance disadvantages to this approach.

2025-02-09T07:13:22Z
https://github.com/huggingface/datasets/issues/5717#issuecomment-2646102865
https://api.github.com/repos/huggingface/datasets/issues/5717
IC_kwDODunzps6duFNR
https://api.github.com/repos/huggingface/datasets/issues/comments/2646102865/reactions
2025-02-09T07:52:12Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2646102865
https://avatars.githubusercontent.com/u/21365990?v=4
https://api.github.com/users/Ryoo72/events{/privacy}
https://api.github.com/users/Ryoo72/followers
https://api.github.com/users/Ryoo72/following{/other_user}
https://api.github.com/users/Ryoo72/gists{/gist_id}
https://github.com/Ryoo72
Ryoo72
MDQ6VXNlcjIxMzY1OTkw
https://api.github.com/users/Ryoo72/orgs
https://api.github.com/users/Ryoo72/received_events
https://api.github.com/users/Ryoo72/repos
https://api.github.com/users/Ryoo72/starred{/owner}{/repo}
https://api.github.com/users/Ryoo72/subscriptions
User
https://api.github.com/users/Ryoo72
public
NONE
Absurd, after months I encountered the same error with datasets 3.6.0 and an Image-Text dataset.
Any solution?
2025-07-12T12:33:50Z
https://github.com/huggingface/datasets/issues/5717#issuecomment-3065245468
https://api.github.com/repos/huggingface/datasets/issues/5717
IC_kwDODunzps62s-8c
https://api.github.com/repos/huggingface/datasets/issues/comments/3065245468/reactions
2025-07-12T12:33:50Z
https://api.github.com/repos/huggingface/datasets/issues/comments/3065245468
https://avatars.githubusercontent.com/u/5716727?v=4
https://api.github.com/users/sbrzz/events{/privacy}
https://api.github.com/users/sbrzz/followers
https://api.github.com/users/sbrzz/following{/other_user}
https://api.github.com/users/sbrzz/gists{/gist_id}
https://github.com/sbrzz
sbrzz
MDQ6VXNlcjU3MTY3Mjc=
https://api.github.com/users/sbrzz/orgs
https://api.github.com/users/sbrzz/received_events
https://api.github.com/users/sbrzz/repos
https://api.github.com/users/sbrzz/starred{/owner}{/repo}
https://api.github.com/users/sbrzz/subscriptions
User
https://api.github.com/users/sbrzz
public
NONE
I end up by splitting the dataset in chunks, and uploading them with different commits and config names. Starting from ~ 30k images, each chunk has ~ 2.5k images. 
This is the only effective solution I found.

I wonder how the HF team has successfully uploaded datasets like the_cauldron [localized_narratives] with more then 200k lines of image-text. I mean, for sure they can do more then push_to_hub, eventually they can explain @lhoestq ? (thanks in advance) 
2025-07-13T08:27:47Z
https://github.com/huggingface/datasets/issues/5717#issuecomment-3066718166
https://api.github.com/repos/huggingface/datasets/issues/5717
IC_kwDODunzps62ymfW
https://api.github.com/repos/huggingface/datasets/issues/comments/3066718166/reactions
2025-07-13T08:27:47Z
https://api.github.com/repos/huggingface/datasets/issues/comments/3066718166
https://avatars.githubusercontent.com/u/5716727?v=4
https://api.github.com/users/sbrzz/events{/privacy}
https://api.github.com/users/sbrzz/followers
https://api.github.com/users/sbrzz/following{/other_user}
https://api.github.com/users/sbrzz/gists{/gist_id}
https://github.com/sbrzz
sbrzz
MDQ6VXNlcjU3MTY3Mjc=
https://api.github.com/users/sbrzz/orgs
https://api.github.com/users/sbrzz/received_events
https://api.github.com/users/sbrzz/repos
https://api.github.com/users/sbrzz/starred{/owner}{/repo}
https://api.github.com/users/sbrzz/subscriptions
User
https://api.github.com/users/sbrzz
public
### Describe the bug

Hello!

I have an issue when I try to save on disk my dataset of images. The error I get is:

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 1442, in save_to_disk
    for job_id, done, content in Dataset._save_to_disk_single(**kwargs):
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 1473, in _save_to_disk_single
    writer.write_table(pa_table)
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/arrow_writer.py", line 570, in write_table
    pa_table = embed_table_storage(pa_table)
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/table.py", line 2268, in embed_table_storage
    arrays = [
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/table.py", line 2269, in <listcomp>
    embed_array_storage(table[name], feature) if require_storage_embed(feature) else table[name]
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/table.py", line 1817, in wrapper
    return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/table.py", line 1817, in <listcomp>
    return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/table.py", line 2142, in embed_array_storage
    return feature.embed_storage(array)
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/features/image.py", line 269, in embed_storage
    storage = pa.StructArray.from_arrays([bytes_array, path_array], ["bytes", "path"], mask=bytes_array.is_null())
  File "pyarrow/array.pxi", line 2766, in pyarrow.lib.StructArray.from_arrays
  File "pyarrow/array.pxi", line 2961, in pyarrow.lib.c_mask_inverted_from_obj
TypeError: Mask must be a pyarrow.Array of type boolean
```

My dataset is around 50K images, is this error might be due to a bad image?

Thanks for the help.

### Steps to reproduce the bug

```python
from datasets import load_dataset
dataset = load_dataset("imagefolder", data_dir="/path/to/dataset")
dataset["train"].save_to_disk("./myds", num_shards=40)
```

### Expected behavior

Having my dataset properly saved to disk.

### Environment info

- `datasets` version: 2.11.0
- Platform: Linux-5.15.90.1-microsoft-standard-WSL2-x86_64-with-glibc2.35
- Python version: 3.10.10
- Huggingface_hub version: 0.13.3
- PyArrow version: 11.0.0
- Pandas version: 2.0.0