Stuck in "Resolving data files..."
COLLABORATOR
Most likely, the data file inference logic is the problem here.

You can run the following code to verify this:
```python
import time
from datasets.data_files import get_data_patterns
start_time = time.time()
get_data_patterns("/path/to/img_dir")
end_time = time.time()
print(f"Elapsed time: {end_time - start_time:.2f}s")
```
 
We plan to optimize this for the next version (or version after that). In the meantime, specifying the split patterns manually should give better performance:
```python
ds = load_dataset("imagefolder", data_files={"train": "path/to/img_dir/train/**", ...}, split="train")
```
2023-10-27T17:28:23Z
https://github.com/huggingface/datasets/issues/6359#issuecomment-1783267865
https://api.github.com/repos/huggingface/datasets/issues/6359
IC_kwDODunzps5qSoIZ
https://api.github.com/repos/huggingface/datasets/issues/comments/1783267865/reactions
2023-10-27T17:28:23Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1783267865
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
Hi, @mariosasko, you are right; data file inference logic is extremely slow.

I have done a similar test, that is I modify the source code of datasets/load.py to measure the cost of two suspicious operations:
```python
def get_module(self) -> DatasetModule:
        base_path = Path(self.data_dir or "").expanduser().resolve().as_posix()
        start = time.time()
        patterns = sanitize_patterns(self.data_files) if self.data_files is not None else get_data_patterns(base_path)
        print(f"patterns: {time.time() - start}")
        start = time.time()
        data_files = DataFilesDict.from_patterns(
            patterns,
            download_config=self.download_config,
            base_path=base_path,
        )
        print(f"data_files: {time.time() - start}")
```
It gaves:
patterns: 3062.2050700187683
data_files: 413.9576675891876

Thus, these two operations contribute to almost all of load time. What's going on in them?
2023-10-28T01:36:13Z
https://github.com/huggingface/datasets/issues/6359#issuecomment-1783655565
https://api.github.com/repos/huggingface/datasets/issues/6359
IC_kwDODunzps5qUGyN
https://api.github.com/repos/huggingface/datasets/issues/comments/1783655565/reactions
2023-10-28T01:36:13Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1783655565
https://avatars.githubusercontent.com/u/20135317?v=4
https://api.github.com/users/Luciennnnnnn/events{/privacy}
https://api.github.com/users/Luciennnnnnn/followers
https://api.github.com/users/Luciennnnnnn/following{/other_user}
https://api.github.com/users/Luciennnnnnn/gists{/gist_id}
https://github.com/Luciennnnnnn
Luciennnnnnn
MDQ6VXNlcjIwMTM1MzE3
https://api.github.com/users/Luciennnnnnn/orgs
https://api.github.com/users/Luciennnnnnn/received_events
https://api.github.com/users/Luciennnnnnn/repos
https://api.github.com/users/Luciennnnnnn/starred{/owner}{/repo}
https://api.github.com/users/Luciennnnnnn/subscriptions
User
https://api.github.com/users/Luciennnnnnn
public
NONE
Furthermore, what's my current workaround about this problem? Should I save it by `save_to_disk()` and load dataset through `load_from_disk`?
2023-10-28T01:38:21Z
https://github.com/huggingface/datasets/issues/6359#issuecomment-1783656064
https://api.github.com/repos/huggingface/datasets/issues/6359
IC_kwDODunzps5qUG6A
https://api.github.com/repos/huggingface/datasets/issues/comments/1783656064/reactions
2023-10-28T01:38:21Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1783656064
https://avatars.githubusercontent.com/u/20135317?v=4
https://api.github.com/users/Luciennnnnnn/events{/privacy}
https://api.github.com/users/Luciennnnnnn/followers
https://api.github.com/users/Luciennnnnnn/following{/other_user}
https://api.github.com/users/Luciennnnnnn/gists{/gist_id}
https://github.com/Luciennnnnnn
Luciennnnnnn
MDQ6VXNlcjIwMTM1MzE3
https://api.github.com/users/Luciennnnnnn/orgs
https://api.github.com/users/Luciennnnnnn/received_events
https://api.github.com/users/Luciennnnnnn/repos
https://api.github.com/users/Luciennnnnnn/starred{/owner}{/repo}
https://api.github.com/users/Luciennnnnnn/subscriptions
User
https://api.github.com/users/Luciennnnnnn
public
NONE
were you able to solve this issue?, I am facing the same issue
2024-01-24T15:02:05Z
https://github.com/huggingface/datasets/issues/6359#issuecomment-1908310723
https://api.github.com/repos/huggingface/datasets/issues/6359
IC_kwDODunzps5xvoLD
https://api.github.com/repos/huggingface/datasets/issues/comments/1908310723/reactions
2024-01-24T15:02:05Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1908310723
https://avatars.githubusercontent.com/u/68569076?v=4
https://api.github.com/users/Ahmed-Roushdy/events{/privacy}
https://api.github.com/users/Ahmed-Roushdy/followers
https://api.github.com/users/Ahmed-Roushdy/following{/other_user}
https://api.github.com/users/Ahmed-Roushdy/gists{/gist_id}
https://github.com/Ahmed-Roushdy
Ahmed-Roushdy
MDQ6VXNlcjY4NTY5MDc2
https://api.github.com/users/Ahmed-Roushdy/orgs
https://api.github.com/users/Ahmed-Roushdy/received_events
https://api.github.com/users/Ahmed-Roushdy/repos
https://api.github.com/users/Ahmed-Roushdy/starred{/owner}{/repo}
https://api.github.com/users/Ahmed-Roushdy/subscriptions
User
https://api.github.com/users/Ahmed-Roushdy
public
NONE
Still suffering from this issue. For me, I cannot download Emilia dataset, and stucked at 'Resolving data files' forever.
``` e = load_dataset('amphion/Emilia-Dataset', token=my_token)
Resolving data files:   0%|                                                                                  | 1/4343 [00:00<11:27,  6.32it/s]
Resolving data files: 100%|█████████████████████████████████████████████████████████████████████████████| 4343/4343 [00:01<00:00, 3844.15it/s]
```
2025-03-09T02:18:18Z
https://github.com/huggingface/datasets/issues/6359#issuecomment-2708626060
https://api.github.com/repos/huggingface/datasets/issues/6359
IC_kwDODunzps6hclqM
https://api.github.com/repos/huggingface/datasets/issues/comments/2708626060/reactions
2025-03-09T02:18:18Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2708626060
https://avatars.githubusercontent.com/u/36876966?v=4
https://api.github.com/users/junjun3518/events{/privacy}
https://api.github.com/users/junjun3518/followers
https://api.github.com/users/junjun3518/following{/other_user}
https://api.github.com/users/junjun3518/gists{/gist_id}
https://github.com/junjun3518
junjun3518
MDQ6VXNlcjM2ODc2OTY2
https://api.github.com/users/junjun3518/orgs
https://api.github.com/users/junjun3518/received_events
https://api.github.com/users/junjun3518/repos
https://api.github.com/users/junjun3518/starred{/owner}{/repo}
https://api.github.com/users/junjun3518/subscriptions
User
https://api.github.com/users/junjun3518
public
### Describe the bug

I have an image dataset with 300k images, the size of image is 768 * 768.

When I run `dataset = load_dataset("imagefolder", data_dir="/path/to/img_dir", split='train')` in second time, it takes 50 minutes to finish "Resolving data files" part, what's going on in this part?

From my understand, after Arrow files been created in the first run, the second run should not take time longer than one or two minutes.

### Steps to reproduce the bug

# Run following code two times
dataset = load_dataset("imagefolder", data_dir="/path/to/img_dir", split='train')

### Expected behavior

Fast dataset building

### Environment info

- `datasets` version: 2.14.5
- Platform: Linux-5.15.0-60-generic-x86_64-with-glibc2.35
- Python version: 3.10.11
- Huggingface_hub version: 0.17.3
- PyArrow version: 10.0.1
- Pandas version: 1.5.3