Increase speed of data files resolution
NONE
#self-assign
2023-03-31T08:24:21Z
https://github.com/huggingface/datasets/issues/5537#issuecomment-1491518352
https://api.github.com/repos/huggingface/datasets/issues/5537
IC_kwDODunzps5Y5sOQ
https://api.github.com/repos/huggingface/datasets/issues/comments/1491518352/reactions
2023-03-31T08:24:21Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1491518352
https://avatars.githubusercontent.com/u/35013374?v=4
https://api.github.com/users/semajyllek/events{/privacy}
https://api.github.com/users/semajyllek/followers
https://api.github.com/users/semajyllek/following{/other_user}
https://api.github.com/users/semajyllek/gists{/gist_id}
https://github.com/semajyllek
semajyllek
MDQ6VXNlcjM1MDEzMzc0
https://api.github.com/users/semajyllek/orgs
https://api.github.com/users/semajyllek/received_events
https://api.github.com/users/semajyllek/repos
https://api.github.com/users/semajyllek/starred{/owner}{/repo}
https://api.github.com/users/semajyllek/subscriptions
User
https://api.github.com/users/semajyllek
public
NONE
You were right, if `self.dir_cache` is not None in glob, it is exactly the same as what is returned by find, at least for all the tests we have, and some extended evaluation I did across a random sample of about 1000 datasets. 

Thanks for the nice hints, and let me know if this is not exactly what we want here!

see PR: https://github.com/huggingface/datasets/pull/5704


2023-04-04T09:07:59Z
https://github.com/huggingface/datasets/issues/5537#issuecomment-1495615491
https://api.github.com/repos/huggingface/datasets/issues/5537
IC_kwDODunzps5ZJUgD
https://api.github.com/repos/huggingface/datasets/issues/comments/1495615491/reactions
2023-04-04T09:07:59Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1495615491
https://avatars.githubusercontent.com/u/35013374?v=4
https://api.github.com/users/semajyllek/events{/privacy}
https://api.github.com/users/semajyllek/followers
https://api.github.com/users/semajyllek/following{/other_user}
https://api.github.com/users/semajyllek/gists{/gist_id}
https://github.com/semajyllek
semajyllek
MDQ6VXNlcjM1MDEzMzc0
https://api.github.com/users/semajyllek/orgs
https://api.github.com/users/semajyllek/received_events
https://api.github.com/users/semajyllek/repos
https://api.github.com/users/semajyllek/starred{/owner}{/repo}
https://api.github.com/users/semajyllek/subscriptions
User
https://api.github.com/users/semajyllek
public
COLLABORATOR
I think we can make the data files resolution (significantly) faster in 2 steps:

1. `glob` calls `find` (which in turn calls `ls`), so we need `find` to be fast, and this can be achieved by fetching all the entries in a single API call and avoiding calls to `ls`. Implementing this for `HfFileSystem.find` (the one in `huggingface_hub`) is on my TO-DO list.
2. caching the repeated `find` calls in `_get_data_files_patterns` when the `data_files` patterns are not provided in `load_dataset`. To address this, we can introduce a `_resolve_single_pattern` function that would accept a filesystem object and a list of regex patterns to resolve. Then we can wrap this filesystem object in `_get_data_files_patterns` with an object that would cache the find calls before resolving the patterns with `_resolve_single_pattern`. (Feel free to suggest a cleaner implementation)

WDYT?
2023-04-07T16:26:36Z
https://github.com/huggingface/datasets/issues/5537#issuecomment-1500438819
https://api.github.com/repos/huggingface/datasets/issues/5537
IC_kwDODunzps5ZbuEj
https://api.github.com/repos/huggingface/datasets/issues/comments/1500438819/reactions
2023-04-07T16:26:36Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1500438819
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
MEMBER
Good idea :) 

For 2:

That would work ! It's also possible to have a FileSystem with a cache on `.find` and use it inside the resolver passed to `_get_data_files_patterns`. Right now they're pretty simple:

```python
# for remote repositories
resolver = partial(_resolve_single_pattern_in_dataset_repository, dataset_info, base_path=base_path)
# for local
resolver = partial(_resolve_single_pattern_locally, base_path)
```
2023-04-07T16:45:36Z
https://github.com/huggingface/datasets/issues/5537#issuecomment-1500452922
https://api.github.com/repos/huggingface/datasets/issues/5537
IC_kwDODunzps5Zbxg6
https://api.github.com/repos/huggingface/datasets/issues/comments/1500452922/reactions
2023-04-07T16:45:36Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1500452922
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
something like this maybe (with Quentin's reimplementation of `HfFilesystem.find`)?

 ```
 @lru_cache(max_size=None)
 def _find(self, path, maxdepth=None, withdirs=False, detail=False, **kwargs):
```

In any case please let me know if I can help in any way!
2023-04-07T17:32:44Z
https://github.com/huggingface/datasets/issues/5537#issuecomment-1500491576
https://api.github.com/repos/huggingface/datasets/issues/5537
IC_kwDODunzps5Zb684
https://api.github.com/repos/huggingface/datasets/issues/comments/1500491576/reactions
2023-04-07T17:32:44Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1500491576
https://avatars.githubusercontent.com/u/35013374?v=4
https://api.github.com/users/semajyllek/events{/privacy}
https://api.github.com/users/semajyllek/followers
https://api.github.com/users/semajyllek/following{/other_user}
https://api.github.com/users/semajyllek/gists{/gist_id}
https://github.com/semajyllek
semajyllek
MDQ6VXNlcjM1MDEzMzc0
https://api.github.com/users/semajyllek/orgs
https://api.github.com/users/semajyllek/received_events
https://api.github.com/users/semajyllek/repos
https://api.github.com/users/semajyllek/starred{/owner}{/repo}
https://api.github.com/users/semajyllek/subscriptions
User
https://api.github.com/users/semajyllek
public
Certain datasets like `bigcode/the-stack-dedup` have so many files that loading them takes forever right from the data files resolution step.

`datasets` uses file patterns to check the structure of the repository but it takes too much time to iterate over and over again on all the data files.

This comes from `resolve_patterns_in_dataset_repository` which calls `_resolve_single_pattern_in_dataset_repository`, which iterates on all the files at
```python
glob_iter = [PurePath(filepath) for filepath in fs.glob(PurePath(pattern).as_posix()) if fs.isfile(filepath)]
```
but calling `glob` on such a dataset is too expensive. Indeed it calls `ls()` in `hffilesystem.py` too many times.

Maybe `glob` can be more optimized in `hffilesystem.py`, or the data files resolution can directly be implemented in the filesystem by checking its `dir_cache` ?