Adding argument to `_get_data_files_patterns`
MEMBER
Hi ! The pattern can be specified in advance in YAML in the README.md of the dataset :)

For example

```
---
configs:
- config_name: default
  data_files:
  - split: train
    path: "train/*"
  - split: test
    path: "test/*"
---
```

See the docs at https://huggingface.co/docs/hub/en/datasets-manual-configuration
2025-03-25T10:11:06Z
https://github.com/huggingface/datasets/issues/7471#issuecomment-2750748701
https://api.github.com/repos/huggingface/datasets/issues/7471
IC_kwDODunzps6j9Rgd
https://api.github.com/repos/huggingface/datasets/issues/comments/2750748701/reactions
2025-03-25T10:11:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2750748701
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
@lhoestq How can we choose in this case ? https://huggingface.co/datasets/datasets-examples/doc-image-5

2025-03-26T07:37:52Z
https://github.com/huggingface/datasets/issues/7471#issuecomment-2753479436
https://api.github.com/repos/huggingface/datasets/issues/7471
IC_kwDODunzps6kHsMM
https://api.github.com/repos/huggingface/datasets/issues/comments/2753479436/reactions
2025-03-26T07:37:52Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2753479436
https://avatars.githubusercontent.com/u/34004152?v=4
https://api.github.com/users/SangbumChoi/events{/privacy}
https://api.github.com/users/SangbumChoi/followers
https://api.github.com/users/SangbumChoi/following{/other_user}
https://api.github.com/users/SangbumChoi/gists{/gist_id}
https://github.com/SangbumChoi
SangbumChoi
MDQ6VXNlcjM0MDA0MTUy
https://api.github.com/users/SangbumChoi/orgs
https://api.github.com/users/SangbumChoi/received_events
https://api.github.com/users/SangbumChoi/repos
https://api.github.com/users/SangbumChoi/starred{/owner}{/repo}
https://api.github.com/users/SangbumChoi/subscriptions
User
https://api.github.com/users/SangbumChoi
public
MEMBER
choose what ? sorry I didn't get it ^^'
2025-03-27T12:30:51Z
https://github.com/huggingface/datasets/issues/7471#issuecomment-2757872148
https://api.github.com/repos/huggingface/datasets/issues/7471
IC_kwDODunzps6kYcoU
https://api.github.com/repos/huggingface/datasets/issues/comments/2757872148/reactions
2025-03-27T12:30:51Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2757872148
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
### Feature request

How about adding if the user already know about the pattern?

https://github.com/huggingface/datasets/blob/a256b85cbc67aa3f0e75d32d6586afc507cf535b/src/datasets/data_files.py#L252

### Motivation

While using this load_dataset people might use 10M of images for the local files.
However, due to searching all the appropriate file pattern in fsspec, purely searching this pattern takes more than 10 hours (real use-case).

### Your contribution

Yeah I can make this happen if this seems valid. @lhoestq WDYT?
such like
```
def _get_data_files_patterns(pattern_resolver: Callable[[str], list[str]], patterns: PATTERNS) -> dict[str, list[str]]:
```