Force JSON format regardless of file naming on S3
MEMBER
Hi ! Support for URIs like `s3://...` is not implemented yet in `data_files=`. You can use the HTTP URL instead if your data is public in the meantime
2022-09-26T09:31:38Z
https://github.com/huggingface/datasets/issues/5012#issuecomment-1257758120
https://api.github.com/repos/huggingface/datasets/issues/5012
IC_kwDODunzps5K992o
https://api.github.com/repos/huggingface/datasets/issues/comments/1257758120/reactions
2022-09-26T09:31:38Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1257758120
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Hi,
I want to make sure I understand this response. I have a set of files on S3 that are private for security reasons. Because they are not public files  I cannot read those files (many are parquet) into my hf notebooks in Kaggle? That can't be correct, can it? 
2023-04-03T20:18:15Z
https://github.com/huggingface/datasets/issues/5012#issuecomment-1494925150
https://api.github.com/repos/huggingface/datasets/issues/5012
IC_kwDODunzps5ZGr9e
https://api.github.com/repos/huggingface/datasets/issues/comments/1494925150/reactions
2023-04-03T20:19:28Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1494925150
https://avatars.githubusercontent.com/u/4406671?v=4
https://api.github.com/users/rdpratti/events{/privacy}
https://api.github.com/users/rdpratti/followers
https://api.github.com/users/rdpratti/following{/other_user}
https://api.github.com/users/rdpratti/gists{/gist_id}
https://github.com/rdpratti
rdpratti
MDQ6VXNlcjQ0MDY2NzE=
https://api.github.com/users/rdpratti/orgs
https://api.github.com/users/rdpratti/received_events
https://api.github.com/users/rdpratti/repos
https://api.github.com/users/rdpratti/starred{/owner}{/repo}
https://api.github.com/users/rdpratti/subscriptions
User
https://api.github.com/users/rdpratti
public
MEMBER
Hi ! There is a discussion at https://github.com/huggingface/datasets/issues/5281

Using the latest `datasets` 2.11 you can try passing fsspec URLs to private buckets to `data_files` in `load_dataset()`. Though this is still experimental and undocumented, so feedback is welcome. You may not have the best experience though, since anything related to performance and caching hasn't been tested properly yet.
2023-04-04T09:33:41Z
https://github.com/huggingface/datasets/issues/5012#issuecomment-1495651312
https://api.github.com/repos/huggingface/datasets/issues/5012
IC_kwDODunzps5ZJdPw
https://api.github.com/repos/huggingface/datasets/issues/comments/1495651312/reactions
2023-04-05T08:59:35Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1495651312
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
closing this one since data_files supports fsspec (still experimental/untested/undocumented for s3 though)
2023-08-16T09:58:34Z
https://github.com/huggingface/datasets/issues/5012#issuecomment-1680314042
https://api.github.com/repos/huggingface/datasets/issues/5012
IC_kwDODunzps5kJ466
https://api.github.com/repos/huggingface/datasets/issues/comments/1680314042/reactions
2023-08-16T09:58:34Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1680314042
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
I have a file on S3 created by Data Version Control, it looks like `s3://dvc/ac/badff5b134382a0f25248f1b45d7b2` but contains a json file. If I run 
```python
dataset = load_dataset(
    "json",
    data_files='s3://dvc/ac/badff5b134382a0f25248f1b45d7b2' 
)
```
It gives me 
```
InvalidSchema: No connection adapters were found for 's3://dvc/ac/badff5b134382a0f25248f1b45d7b2'
```
However, I cannot go ahead and change the names of the s3 file. Is there a way to "force" load a S3 url with certain decoder (JSON, CSV, etc.) regardless of s3 URL naming?