`from_list` fails while `from_generator` works for large datasets
NONE
@lhoestq any thoughts on this? 
2025-06-18T09:29:24Z
https://github.com/huggingface/datasets/issues/7619#issuecomment-2983410342
https://api.github.com/repos/huggingface/datasets/issues/7619
IC_kwDODunzps6x0zqm
https://api.github.com/repos/huggingface/datasets/issues/comments/2983410342/reactions
2025-06-18T09:29:24Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2983410342
https://avatars.githubusercontent.com/u/4028948?v=4
https://api.github.com/users/abdulfatir/events{/privacy}
https://api.github.com/users/abdulfatir/followers
https://api.github.com/users/abdulfatir/following{/other_user}
https://api.github.com/users/abdulfatir/gists{/gist_id}
https://github.com/abdulfatir
abdulfatir
MDQ6VXNlcjQwMjg5NDg=
https://api.github.com/users/abdulfatir/orgs
https://api.github.com/users/abdulfatir/received_events
https://api.github.com/users/abdulfatir/repos
https://api.github.com/users/abdulfatir/starred{/owner}{/repo}
https://api.github.com/users/abdulfatir/subscriptions
User
https://api.github.com/users/abdulfatir
public
CONTRIBUTOR
Thanks for the report! This behavior is expected due to how `from_list()` and `from_generator()` differ internally.

- `from_list()` builds the entire dataset in memory at once, which can easily exceed limits (especially with variable-length arrays or millions of rows). The Arrow error you're seeing (`Value too large to fit in C integer type`) is related to that memory overload.
- `from_generator()` avoids this issue by batching and streaming data incrementally, which is much more memory-efficient.

So for large datasets like time series or NLP data with large arrays, `from_generator()` (or `datasets.IterableDataset`) is the recommended approach.

Hope this helps clarify the behavior — let me know if you'd like me to point to prior issues/discussions where similar tradeoffs came up!

2025-06-29T07:36:25Z
https://github.com/huggingface/datasets/issues/7619#issuecomment-3016379235
https://api.github.com/repos/huggingface/datasets/issues/7619
IC_kwDODunzps6zyktj
https://api.github.com/repos/huggingface/datasets/issues/comments/3016379235/reactions
2025-06-29T07:36:52Z
https://api.github.com/repos/huggingface/datasets/issues/comments/3016379235
https://avatars.githubusercontent.com/u/142811259?v=4
https://api.github.com/users/ArjunJagdale/events{/privacy}
https://api.github.com/users/ArjunJagdale/followers
https://api.github.com/users/ArjunJagdale/following{/other_user}
https://api.github.com/users/ArjunJagdale/gists{/gist_id}
https://github.com/ArjunJagdale
ArjunJagdale
U_kgDOCIMgew
https://api.github.com/users/ArjunJagdale/orgs
https://api.github.com/users/ArjunJagdale/received_events
https://api.github.com/users/ArjunJagdale/repos
https://api.github.com/users/ArjunJagdale/starred{/owner}{/repo}
https://api.github.com/users/ArjunJagdale/subscriptions
User
https://api.github.com/users/ArjunJagdale
public
NONE
@ArjunJagdale Yes, it is related to using large dataset but not in the way that you have described. As I understand, the problem here is that `datasets` does not use `LargeList` with 64-bit offsets from PyArrow when using `from_list`. However, with `from_generator` this seems to work okay, likely due to batching. As such, this is more like a bug than an expected outcome. If this is indeed "expected", `datasets` should fail more gracefully in these cases with a recommendation to use `from_generator`. 
2025-06-29T10:31:46Z
https://github.com/huggingface/datasets/issues/7619#issuecomment-3016572725
https://api.github.com/repos/huggingface/datasets/issues/7619
IC_kwDODunzps6zzT81
https://api.github.com/repos/huggingface/datasets/issues/comments/3016572725/reactions
2025-06-29T10:31:46Z
https://api.github.com/repos/huggingface/datasets/issues/comments/3016572725
https://avatars.githubusercontent.com/u/4028948?v=4
https://api.github.com/users/abdulfatir/events{/privacy}
https://api.github.com/users/abdulfatir/followers
https://api.github.com/users/abdulfatir/following{/other_user}
https://api.github.com/users/abdulfatir/gists{/gist_id}
https://github.com/abdulfatir
abdulfatir
MDQ6VXNlcjQwMjg5NDg=
https://api.github.com/users/abdulfatir/orgs
https://api.github.com/users/abdulfatir/received_events
https://api.github.com/users/abdulfatir/repos
https://api.github.com/users/abdulfatir/starred{/owner}{/repo}
https://api.github.com/users/abdulfatir/subscriptions
User
https://api.github.com/users/abdulfatir
public
CONTRIBUTOR
Thanks for the clarification — you're absolutely right, this seems tied to the use of 32-bit list offsets in from_list() under the hood. That distinction between List and LargeList in PyArrow is a crucial one, and definitely worth highlighting in the docs or error message. Happy to help if a check or fallback to LargeList makes sense here.
2025-06-29T16:34:44Z
https://github.com/huggingface/datasets/issues/7619#issuecomment-3016839227
https://api.github.com/repos/huggingface/datasets/issues/7619
IC_kwDODunzps6z0VA7
https://api.github.com/repos/huggingface/datasets/issues/comments/3016839227/reactions
2025-06-29T16:34:44Z
https://api.github.com/repos/huggingface/datasets/issues/comments/3016839227
https://avatars.githubusercontent.com/u/142811259?v=4
https://api.github.com/users/ArjunJagdale/events{/privacy}
https://api.github.com/users/ArjunJagdale/followers
https://api.github.com/users/ArjunJagdale/following{/other_user}
https://api.github.com/users/ArjunJagdale/gists{/gist_id}
https://github.com/ArjunJagdale
ArjunJagdale
U_kgDOCIMgew
https://api.github.com/users/ArjunJagdale/orgs
https://api.github.com/users/ArjunJagdale/received_events
https://api.github.com/users/ArjunJagdale/repos
https://api.github.com/users/ArjunJagdale/starred{/owner}{/repo}
https://api.github.com/users/ArjunJagdale/subscriptions
User
https://api.github.com/users/ArjunJagdale
public
### Describe the bug

I am constructing a large time series dataset and observed that first constructing a list of entries and then using `Dataset.from_list` led to a crash as the number of items became large. However, this is not a problem when using `Dataset.from_generator`.

### Steps to reproduce the bug

#### Snippet A (crashes)

```py
from tqdm.auto import tqdm
import numpy as np
import datasets


def data_generator():
    for i in tqdm(range(10_000_000)):
        length = np.random.randint(2048)
        series = np.random.rand(length)
        yield {"target": series, "item_id": str(i), "start": np.datetime64("2000", "ms")}

data_list = list(data_generator())
ds = datasets.Dataset.from_list(data_list)
```
The last line crashes with
```
ArrowInvalid: Value 2147483761 too large to fit in C integer type
```

#### Snippet B (works)

```py
from tqdm.auto import tqdm
import numpy as np
import datasets


def data_generator():
    for i in tqdm(range(10_000_000)):
        length = np.random.randint(2048)
        series = np.random.rand(length)
        yield {"target": series, "item_id": str(i), "start": np.datetime64("2000", "ms")}

ds = datasets.Dataset.from_generator(data_generator)
```

### Expected behavior

I expected both the approaches to work or to fail similarly. 

### Environment info

```
- `datasets` version: 3.6.0
- Platform: Linux-6.8.0-1029-aws-x86_64-with-glibc2.35
- Python version: 3.11.11
- `huggingface_hub` version: 0.32.2
- PyArrow version: 19.0.1
- Pandas version: 2.2.3
- `fsspec` version: 2025.3.0
```