Dataset.from_generator consistently freezes at ~1000 rows
COLLABORATOR
By default, we write data to disk (so it can be memory-mapped) every 1000 rows/samples. You can control this with the `writer_batch_size` parameter. Also, when working with fixed-size arrays, the `ArrayXD` feature types yield better performance (e.g., in your case, `features=datasets.Features({"i": datasets.Array3D(shape=(512,512,3), dtype="float32")})` should be faster).

Our support for multi-dim arrays could be better, and we plan to improve it as part of https://github.com/huggingface/datasets/issues/5272.
2023-07-05T19:09:38Z
https://github.com/huggingface/datasets/issues/6008#issuecomment-1622320308
https://api.github.com/repos/huggingface/datasets/issues/6008
IC_kwDODunzps5gsqS0
https://api.github.com/repos/huggingface/datasets/issues/comments/1622320308/reactions
2023-07-05T19:09:38Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1622320308
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
> By default, we write data to disk (so it can be memory-mapped) every 1000 rows/samples. You can control this with the `writer_batch_size` parameter. Also, when working with fixed-size arrays, the `ArrayXD` feature types yield better performance (e.g., in your case, `features=datasets.Features({"i": datasets.Array3D(shape=(512,512,3), dtype="float32")})` should be faster).
> 
> Our support for multi-dim arrays could be better, and we plan to improve it as part of #5272.

Thanks for the explanation! The Image array was just for demonstration, I use PIL Images in practice. Does that make a difference? What's the best approach for a dataset with PIL Images as rows?
2023-07-05T21:44:33Z
https://github.com/huggingface/datasets/issues/6008#issuecomment-1622567452
https://api.github.com/repos/huggingface/datasets/issues/6008
IC_kwDODunzps5gtmoc
https://api.github.com/repos/huggingface/datasets/issues/comments/1622567452/reactions
2023-07-05T21:44:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1622567452
https://avatars.githubusercontent.com/u/27695722?v=4
https://api.github.com/users/andreemic/events{/privacy}
https://api.github.com/users/andreemic/followers
https://api.github.com/users/andreemic/following{/other_user}
https://api.github.com/users/andreemic/gists{/gist_id}
https://github.com/andreemic
andreemic
MDQ6VXNlcjI3Njk1NzIy
https://api.github.com/users/andreemic/orgs
https://api.github.com/users/andreemic/received_events
https://api.github.com/users/andreemic/repos
https://api.github.com/users/andreemic/starred{/owner}{/repo}
https://api.github.com/users/andreemic/subscriptions
User
https://api.github.com/users/andreemic
public
COLLABORATOR
It's best to use the `datasets.Image()` feature type for PIL images (to save space) :)
2023-07-06T16:29:25Z
https://github.com/huggingface/datasets/issues/6008#issuecomment-1623972849
https://api.github.com/repos/huggingface/datasets/issues/6008
IC_kwDODunzps5gy9vx
https://api.github.com/repos/huggingface/datasets/issues/comments/1623972849/reactions
2023-07-06T16:32:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1623972849
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
### Describe the bug

Whenever I try to create a dataset which contains images using `Dataset.from_generator`, it freezes around 996 rows. I suppose it has something to do with memory consumption, but there's more memory available. I

Somehow it worked a few times but mostly this makes the datasets library much more cumbersome to work with because generators are the easiest way to turn an existing dataset into a Hugging Face dataset.

I've let it run in the frozen state for way longer than it can possibly take to load the actual dataset.

Let me know if you have ideas how to resolve it!

### Steps to reproduce the bug

```python
from datasets import Dataset
import numpy as np

def gen():
    for row in range(10000):
        yield {"i": np.random.rand(512, 512, 3)}
        
Dataset.from_generator(gen)
# -> 90% of the time gets stuck around 1000 rows
```

### Expected behavior

Should continue and go through all the examples yielded by the generator, or at least throw an error or somehow communicate what's going on.

### Environment info

- `datasets` version: 2.8.0
- Platform: Linux-5.15.0-52-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 12.0.1
- Pandas version: 1.5.1
