Index error when data is large
NONE
I'm facing this problem while doing my translation of [mteb/stackexchange-clustering](https://huggingface.co/datasets/mteb/stackexchange-clustering). each row has lots of samples (up to 100k samples), because in this dataset, each row represent multiple clusters.
my hack is to setting `max_shard_size` to 20Gb or even larger
```py
final_dataset.push_to_hub(
        output_dataset, 
        private=True,
        max_shard_size="20GB"  # This will ensure appropriate sharding based on data size
    )
```
It will work, but depends on your data size. 
2025-04-16T04:13:00Z
https://github.com/huggingface/datasets/issues/6602#issuecomment-2808189920
https://api.github.com/repos/huggingface/datasets/issues/6602
IC_kwDODunzps6nYZPg
https://api.github.com/repos/huggingface/datasets/issues/comments/2808189920/reactions
2025-04-16T04:13:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2808189920
https://avatars.githubusercontent.com/u/67360122?v=4
https://api.github.com/users/BaoLocPham/events{/privacy}
https://api.github.com/users/BaoLocPham/followers
https://api.github.com/users/BaoLocPham/following{/other_user}
https://api.github.com/users/BaoLocPham/gists{/gist_id}
https://github.com/BaoLocPham
BaoLocPham
MDQ6VXNlcjY3MzYwMTIy
https://api.github.com/users/BaoLocPham/orgs
https://api.github.com/users/BaoLocPham/received_events
https://api.github.com/users/BaoLocPham/repos
https://api.github.com/users/BaoLocPham/starred{/owner}{/repo}
https://api.github.com/users/BaoLocPham/subscriptions
User
https://api.github.com/users/BaoLocPham
public
### Describe the bug

At `save_to_disk` step, the `max_shard_size` by default is `500MB`. However, one row of the dataset might be larger than `500MB` then the saving will throw an index error. Without looking at the source code, the bug is due to wrong calculation of number of shards which i think is 
`total_size / min(max_shard_size, row_size)` which should be `total_size / max(max_shard_size, row_size)`

The fix is setting a larger `max_shard_size`

### Steps to reproduce the bug

1. create a dataset with large dense tensors per row
2. set a small `max_shard_size` say 1MB
3. `save_to_disk`

### Expected behavior

```
raise IndexError(f"Index {index} out of range for dataset of size {size}.")

IndexError: Index 10 out of range for dataset of size 10.
```

### Environment info

- `datasets` version: 2.16.0
- Platform: Linux-5.10.201-168.748.amzn2int.x86_64-x86_64-with-glibc2.26
- Python version: 3.10.13
- `huggingface_hub` version: 0.20.2
- PyArrow version: 14.0.2
- Pandas version: 2.1.4
- `fsspec` version: 2023.12.2