load_dataset('bigcode/the-stack-dedup', streaming=True) very slow!
COLLABORATOR
This is due to the slow resolution of the data files: https://github.com/huggingface/datasets/issues/5537.

We plan to switch to `huggingface_hub`'s `HfFileSystem` soon to make the resolution faster (will be up to 20x faster once we merge https://github.com/huggingface/huggingface_hub/pull/1443)


2023-05-12T14:28:15Z
https://github.com/huggingface/datasets/issues/5846#issuecomment-1545834996
https://api.github.com/repos/huggingface/datasets/issues/5846
IC_kwDODunzps5cI5H0
https://api.github.com/repos/huggingface/datasets/issues/comments/1545834996/reactions
2023-05-12T14:28:23Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1545834996
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
You're right, when I try to parse more than 50GB of text data, I also get very slow, usually taking hours or even tens of hours.
2023-05-15T05:27:12Z
https://github.com/huggingface/datasets/issues/5846#issuecomment-1547214281
https://api.github.com/repos/huggingface/datasets/issues/5846
IC_kwDODunzps5cOJ3J
https://api.github.com/repos/huggingface/datasets/issues/comments/1547214281/reactions
2023-05-15T05:27:12Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1547214281
https://avatars.githubusercontent.com/u/14285786?v=4
https://api.github.com/users/enze5088/events{/privacy}
https://api.github.com/users/enze5088/followers
https://api.github.com/users/enze5088/following{/other_user}
https://api.github.com/users/enze5088/gists{/gist_id}
https://github.com/enze5088
enze5088
MDQ6VXNlcjE0Mjg1Nzg2
https://api.github.com/users/enze5088/orgs
https://api.github.com/users/enze5088/received_events
https://api.github.com/users/enze5088/repos
https://api.github.com/users/enze5088/starred{/owner}{/repo}
https://api.github.com/users/enze5088/subscriptions
User
https://api.github.com/users/enze5088
public
NONE
> You're right, when I try to parse more than 50GB of text data, I also get very slow, usually taking hours or even tens of hours.

That's unrelated to the problem discussed in this issue. 
2023-05-15T15:07:34Z
https://github.com/huggingface/datasets/issues/5846#issuecomment-1548047064
https://api.github.com/repos/huggingface/datasets/issues/5846
IC_kwDODunzps5cRVLY
https://api.github.com/repos/huggingface/datasets/issues/comments/1548047064/reactions
2023-05-15T15:07:34Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1548047064
https://avatars.githubusercontent.com/u/4241811?v=4
https://api.github.com/users/tbenthompson/events{/privacy}
https://api.github.com/users/tbenthompson/followers
https://api.github.com/users/tbenthompson/following{/other_user}
https://api.github.com/users/tbenthompson/gists{/gist_id}
https://github.com/tbenthompson
tbenthompson
MDQ6VXNlcjQyNDE4MTE=
https://api.github.com/users/tbenthompson/orgs
https://api.github.com/users/tbenthompson/received_events
https://api.github.com/users/tbenthompson/repos
https://api.github.com/users/tbenthompson/starred{/owner}{/repo}
https://api.github.com/users/tbenthompson/subscriptions
User
https://api.github.com/users/tbenthompson
public
NONE
> > You're right, when I try to parse more than 50GB of text data, I also get very slow, usually taking hours or even tens of hours.
> 
> That's unrelated to the problem discussed in this issue.

Sorry, I misunderstood it.
2023-05-16T03:23:46Z
https://github.com/huggingface/datasets/issues/5846#issuecomment-1548921076
https://api.github.com/repos/huggingface/datasets/issues/5846
IC_kwDODunzps5cUqj0
https://api.github.com/repos/huggingface/datasets/issues/comments/1548921076/reactions
2023-05-16T03:23:46Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1548921076
https://avatars.githubusercontent.com/u/14285786?v=4
https://api.github.com/users/enze5088/events{/privacy}
https://api.github.com/users/enze5088/followers
https://api.github.com/users/enze5088/following{/other_user}
https://api.github.com/users/enze5088/gists{/gist_id}
https://github.com/enze5088
enze5088
MDQ6VXNlcjE0Mjg1Nzg2
https://api.github.com/users/enze5088/orgs
https://api.github.com/users/enze5088/received_events
https://api.github.com/users/enze5088/repos
https://api.github.com/users/enze5088/starred{/owner}{/repo}
https://api.github.com/users/enze5088/subscriptions
User
https://api.github.com/users/enze5088
public
COLLABORATOR
Closing this issue as it has been addressed in `huggingface_hub`!

(This now takes 25s to execute on my machine.)
2024-04-05T12:28:58Z
https://github.com/huggingface/datasets/issues/5846#issuecomment-2039671723
https://api.github.com/repos/huggingface/datasets/issues/5846
IC_kwDODunzps55kuur
https://api.github.com/repos/huggingface/datasets/issues/comments/2039671723/reactions
2024-04-05T12:28:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2039671723
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
Thanks for the improvements! 🎉🎉

25 seconds is better but still about 2500x slower than this _should_ be! Loading a tiny 1-2KB metadata file is all that would be necessary with a better design.
2024-04-05T13:45:08Z
https://github.com/huggingface/datasets/issues/5846#issuecomment-2039844937
https://api.github.com/repos/huggingface/datasets/issues/5846
IC_kwDODunzps55lZBJ
https://api.github.com/repos/huggingface/datasets/issues/comments/2039844937/reactions
2024-04-05T13:45:08Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2039844937
https://avatars.githubusercontent.com/u/4241811?v=4
https://api.github.com/users/tbenthompson/events{/privacy}
https://api.github.com/users/tbenthompson/followers
https://api.github.com/users/tbenthompson/following{/other_user}
https://api.github.com/users/tbenthompson/gists{/gist_id}
https://github.com/tbenthompson
tbenthompson
MDQ6VXNlcjQyNDE4MTE=
https://api.github.com/users/tbenthompson/orgs
https://api.github.com/users/tbenthompson/received_events
https://api.github.com/users/tbenthompson/repos
https://api.github.com/users/tbenthompson/starred{/owner}{/repo}
https://api.github.com/users/tbenthompson/subscriptions
User
https://api.github.com/users/tbenthompson
public
COLLABORATOR
Once we merge https://github.com/huggingface/huggingface_hub/pull/2103, this should only take a few seconds. 

For the 2500x speed-up (without metadata files with pre-cached results), we wouldn't even be allowed to use `os.path` functions or `requests`/`aiohttp` for HTTP requests, so I don't think this is feasible for us as it would make the code unreadable.

The HF Datasets Hub is (almost) platform-agnostic, so you are free to implement your own library (in a faster language than Python) to achieve this kind of performance, and we would be happy to support it 🙂.  
2024-04-08T12:50:55Z
https://github.com/huggingface/datasets/issues/5846#issuecomment-2042675887
https://api.github.com/repos/huggingface/datasets/issues/5846
IC_kwDODunzps55wMKv
https://api.github.com/repos/huggingface/datasets/issues/comments/2042675887/reactions
2024-04-08T12:53:17Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2042675887
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
### Describe the bug

Running

```
import datasets
ds = datasets.load_dataset('bigcode/the-stack-dedup', streaming=True)
```

takes about 2.5 minutes! 

I would expect this to be near instantaneous. With other datasets, the runtime is one or two seconds.


### Environment info

- `datasets` version: 2.11.0
- Platform: macOS-13.3.1-arm64-arm-64bit
- Python version: 3.10.10
- Huggingface_hub version: 0.13.4
- PyArrow version: 11.0.0
- Pandas version: 2.0.0