[2.12.0] DatasetDict.save_to_disk not saving to S3
MEMBER
Hi ! Can you try adding the `s3://` prefix ?
```python
f"s3://{s3_bucket}/{s3_dir}/{dataset_name}"
```
2023-05-05T12:43:16Z
https://github.com/huggingface/datasets/issues/5823#issuecomment-1536206488
https://api.github.com/repos/huggingface/datasets/issues/5823
IC_kwDODunzps5bkKaY
https://api.github.com/repos/huggingface/datasets/issues/comments/1536206488/reactions
2023-05-05T12:43:28Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1536206488
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Ugh, yeah that was it. Thank you!
2023-05-05T15:01:17Z
https://github.com/huggingface/datasets/issues/5823#issuecomment-1536391793
https://api.github.com/repos/huggingface/datasets/issues/5823
IC_kwDODunzps5bk3px
https://api.github.com/repos/huggingface/datasets/issues/comments/1536391793/reactions
2023-05-05T15:01:17Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1536391793
https://avatars.githubusercontent.com/u/5233185?v=4
https://api.github.com/users/thejamesmarq/events{/privacy}
https://api.github.com/users/thejamesmarq/followers
https://api.github.com/users/thejamesmarq/following{/other_user}
https://api.github.com/users/thejamesmarq/gists{/gist_id}
https://github.com/thejamesmarq
thejamesmarq
MDQ6VXNlcjUyMzMxODU=
https://api.github.com/users/thejamesmarq/orgs
https://api.github.com/users/thejamesmarq/received_events
https://api.github.com/users/thejamesmarq/repos
https://api.github.com/users/thejamesmarq/starred{/owner}{/repo}
https://api.github.com/users/thejamesmarq/subscriptions
User
https://api.github.com/users/thejamesmarq
public
NONE
Hi @thejamesmarq, by any chance, did you use multiprocessing `num_proc > 1` when saving your dataset on the s3 bucket ? I'm struggling making it work in a multiprocessing setting while everything works fine with one processor.
2024-05-30T16:11:22Z
https://github.com/huggingface/datasets/issues/5823#issuecomment-2140105875
https://api.github.com/repos/huggingface/datasets/issues/5823
IC_kwDODunzps5_j2yT
https://api.github.com/repos/huggingface/datasets/issues/comments/2140105875/reactions
2024-05-30T16:11:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2140105875
https://avatars.githubusercontent.com/u/54974949?v=4
https://api.github.com/users/ycattan/events{/privacy}
https://api.github.com/users/ycattan/followers
https://api.github.com/users/ycattan/following{/other_user}
https://api.github.com/users/ycattan/gists{/gist_id}
https://github.com/ycattan
ycattan
MDQ6VXNlcjU0OTc0OTQ5
https://api.github.com/users/ycattan/orgs
https://api.github.com/users/ycattan/received_events
https://api.github.com/users/ycattan/repos
https://api.github.com/users/ycattan/starred{/owner}{/repo}
https://api.github.com/users/ycattan/subscriptions
User
https://api.github.com/users/ycattan
public
### Describe the bug

When trying to save a `DatasetDict` to a private S3 bucket using `save_to_disk`, the artifacts are instead saved locally, and not in the S3 bucket.

I have tried using the deprecated `fs` as well as the `storage_options` arguments and I get the same results.

### Steps to reproduce the bug

1. Create a DatsetDict `dataset`
2. Create a S3FileSystem object
`s3 = datasets.filesystems.S3FileSystem(key=aws_access_key_id, secret=aws_secret_access_key)`
3. Save using `dataset_dict.save_to_disk(f"{s3_bucket}/{s3_dir}/{dataset_name}", storage_options=s3.storage_options)` or `dataset_dict.save_to_disk(f"{s3_bucket}/{s3_dir}/{dataset_name}", fs=s3)`
4. Check the corresponding S3 bucket and verify nothing has been uploaded
5. Check the path at f"{s3_bucket}/{s3_dir}/{dataset_name}" and verify that files have been saved there

### Expected behavior

Artifacts are uploaded at the f"{s3_bucket}/{s3_dir}/{dataset_name}" S3 location.

### Environment info

- `datasets` version: 2.12.0
- Platform: macOS-13.3.1-x86_64-i386-64bit
- Python version: 3.11.2
- Huggingface_hub version: 0.14.1
- PyArrow version: 12.0.0
- Pandas version: 2.0.1