Loading dataset from large GCS bucket very slow since 2.14
NONE
I've also encountered this issue recently and want to ask if this has been seen.

@albertvillanova for visibility - I'm not sure who the right person is to tag, but I noticed you were active recently so perhaps you can direct this to the right person.

Thanks!
2024-09-03T18:42:32Z
https://github.com/huggingface/datasets/issues/6323#issuecomment-2327184334
https://api.github.com/repos/huggingface/datasets/issues/6323
IC_kwDODunzps6KtgPO
https://api.github.com/repos/huggingface/datasets/issues/comments/2327184334/reactions
2024-09-03T18:42:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2327184334
https://avatars.githubusercontent.com/u/57027354?v=4
https://api.github.com/users/connermanuel/events{/privacy}
https://api.github.com/users/connermanuel/followers
https://api.github.com/users/connermanuel/following{/other_user}
https://api.github.com/users/connermanuel/gists{/gist_id}
https://github.com/connermanuel
connermanuel
MDQ6VXNlcjU3MDI3MzU0
https://api.github.com/users/connermanuel/orgs
https://api.github.com/users/connermanuel/received_events
https://api.github.com/users/connermanuel/repos
https://api.github.com/users/connermanuel/starred{/owner}{/repo}
https://api.github.com/users/connermanuel/subscriptions
User
https://api.github.com/users/connermanuel
public
### Describe the bug

Since updating to >2.14 we have very slow access to our parquet files on GCS when loading a dataset (>30 min vs 3s). Our GCS bucket has many objects and resolving globs is very slow. I could track down the problem to this change:
https://github.com/huggingface/datasets/blame/bade7af74437347a760830466eb74f7a8ce0d799/src/datasets/data_files.py#L348 
The underlying implementation with gcsfs is really slow. Could you go back to the old way if we are simply giving the parquet files and no glob pattern?

Thank you.




### Steps to reproduce the bug

Load a dataset from a GCS bucket that has many files.

### Expected behavior

Used to be fast (3s) in 2.13

### Environment info

datasets==2.14.5