save_to_disk() freezes when saving on s3 bucket with multiprocessing
NONE
I got the same issue. Any updates so far for this issue?
2024-07-22T23:08:41Z
https://github.com/huggingface/datasets/issues/6936#issuecomment-2243960302
https://api.github.com/repos/huggingface/datasets/issues/6936
IC_kwDODunzps6FwB3u
https://api.github.com/repos/huggingface/datasets/issues/comments/2243960302/reactions
2024-07-22T23:08:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2243960302
https://avatars.githubusercontent.com/u/169724841?v=4
https://api.github.com/users/sfc-gh-ywei/events{/privacy}
https://api.github.com/users/sfc-gh-ywei/followers
https://api.github.com/users/sfc-gh-ywei/following{/other_user}
https://api.github.com/users/sfc-gh-ywei/gists{/gist_id}
https://github.com/sfc-gh-ywei
sfc-gh-ywei
U_kgDOCh3LqQ
https://api.github.com/users/sfc-gh-ywei/orgs
https://api.github.com/users/sfc-gh-ywei/received_events
https://api.github.com/users/sfc-gh-ywei/repos
https://api.github.com/users/sfc-gh-ywei/starred{/owner}{/repo}
https://api.github.com/users/sfc-gh-ywei/subscriptions
User
https://api.github.com/users/sfc-gh-ywei
public
NONE
Same here. Any updates?
2024-11-29T21:41:58Z
https://github.com/huggingface/datasets/issues/6936#issuecomment-2508684902
https://api.github.com/repos/huggingface/datasets/issues/6936
IC_kwDODunzps6Vh35m
https://api.github.com/repos/huggingface/datasets/issues/comments/2508684902/reactions
2024-11-29T21:41:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2508684902
https://avatars.githubusercontent.com/u/60658814?v=4
https://api.github.com/users/solanovisitor/events{/privacy}
https://api.github.com/users/solanovisitor/followers
https://api.github.com/users/solanovisitor/following{/other_user}
https://api.github.com/users/solanovisitor/gists{/gist_id}
https://github.com/solanovisitor
solanovisitor
MDQ6VXNlcjYwNjU4ODE0
https://api.github.com/users/solanovisitor/orgs
https://api.github.com/users/solanovisitor/received_events
https://api.github.com/users/solanovisitor/repos
https://api.github.com/users/solanovisitor/starred{/owner}{/repo}
https://api.github.com/users/solanovisitor/subscriptions
User
https://api.github.com/users/solanovisitor
public
NONE
+1, experiencing this as well
2025-02-06T22:12:50Z
https://github.com/huggingface/datasets/issues/6936#issuecomment-2641178416
https://api.github.com/repos/huggingface/datasets/issues/6936
IC_kwDODunzps6dbS8w
https://api.github.com/repos/huggingface/datasets/issues/comments/2641178416/reactions
2025-02-06T22:12:50Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2641178416
https://avatars.githubusercontent.com/u/4479137?v=4
https://api.github.com/users/grapefroot/events{/privacy}
https://api.github.com/users/grapefroot/followers
https://api.github.com/users/grapefroot/following{/other_user}
https://api.github.com/users/grapefroot/gists{/gist_id}
https://github.com/grapefroot
grapefroot
MDQ6VXNlcjQ0NzkxMzc=
https://api.github.com/users/grapefroot/orgs
https://api.github.com/users/grapefroot/received_events
https://api.github.com/users/grapefroot/repos
https://api.github.com/users/grapefroot/starred{/owner}{/repo}
https://api.github.com/users/grapefroot/subscriptions
User
https://api.github.com/users/grapefroot
public
### Describe the bug

I'm trying to save a `Dataset` using the `save_to_disk()` function with:
- `num_proc > 1`
- `dataset_path` being a s3 bucket path e.g. "s3://{bucket_name}/{dataset_folder}/"

The hf progress bar shows up but the saving does not seem to start. 
When using one processor only (`num_proc=1`), everything works fine.
When saving the dataset on local disk (as opposed to s3 bucket) with `num_proc > 1`, everything works fine.

Thank you for your help! :)

### Steps to reproduce the bug

I tried without any storage options:

```
from datasets import load_dataset

sandbox_ds = load_dataset("openai_humaneval")
sandbox_ds["test"].save_to_disk(
    "s3://bucket-name/test_multiprocessing_saving/",
    num_proc=4,
)
```


and with the specific s3fs storage options: 
```
from datasets import load_dataset
from s3fs import S3FileSystem

def get_s3fs():
    return S3FileSystem()

sandbox_ds = load_dataset("openai_humaneval")
sandbox_ds["test"].save_to_disk(
    "s3://bucket-name/test_multiprocessing_saving/",
    num_proc=4,
    storage_options=get_s3fs().storage_options, # also tried: storage_options=S3FileSystem().storage_options
)
```

I'm guessing I might use `storage_options` parameter wrongly, but I didn't find anything online that made it work.

**NB**: Behavior is the same when trying to save the whole `DatasetDict`.

### Expected behavior

Progress bar fills in and saving is carried out.

### Environment info

`datasets==2.18.0`