fsspec: support fsspec>=2023.12.0 glob changes
NONE
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6687). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
2024-02-22T16:13:20Z
https://github.com/huggingface/datasets/pull/6687#issuecomment-1959783723
https://api.github.com/repos/huggingface/datasets/issues/6687
IC_kwDODunzps50z-0r
https://api.github.com/repos/huggingface/datasets/issues/comments/1959783723/reactions
2024-02-22T16:13:20Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1959783723
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
CONTRIBUTOR
Looking into the CI failure, this PR is incompatible with `huggingface-hub>=0.20.0`. It looks like there were several changes made to HfFileSystem in 0.20.0, @lhoestq any ideas on what the issue might be in particular?

a bisect indicates that it's related to https://github.com/huggingface/huggingface_hub/pull/1815
2024-02-28T05:59:47Z
https://github.com/huggingface/datasets/pull/6687#issuecomment-1968290091
https://api.github.com/repos/huggingface/datasets/issues/6687
IC_kwDODunzps51Ubkr
https://api.github.com/repos/huggingface/datasets/issues/comments/1968290091/reactions
2024-02-28T06:11:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1968290091
https://avatars.githubusercontent.com/u/651988?v=4
https://api.github.com/users/pmrowla/events{/privacy}
https://api.github.com/users/pmrowla/followers
https://api.github.com/users/pmrowla/following{/other_user}
https://api.github.com/users/pmrowla/gists{/gist_id}
https://github.com/pmrowla
pmrowla
MDQ6VXNlcjY1MTk4OA==
https://api.github.com/users/pmrowla/orgs
https://api.github.com/users/pmrowla/received_events
https://api.github.com/users/pmrowla/repos
https://api.github.com/users/pmrowla/starred{/owner}{/repo}
https://api.github.com/users/pmrowla/subscriptions
User
https://api.github.com/users/pmrowla
public
CONTRIBUTOR
It looks like huggingface-hub's `HfFileSystem.glob` is broken for exact string matches (that don't contain glob wildcards) when combining `huggingface-hub>=0.20.0` and `fsspec>=2023.12.0`.

I did a quick test with huggingface-hub `main`, and adding this test case to `tests/test_hf_filesystem::HfFileSystemTests::test_glob` (https://github.com/huggingface/huggingface_hub/blob/main/tests/test_hf_file_system.py) passes with `fsspec==2023.10.0` and fails with `fsspec==2023.12.0`
```python
        self.assertEqual(
            sorted(self.hffs.glob(self.hf_path + "/.gitattributes")),
            sorted([self.hf_path + "/.gitattributes"]),
        )

```

the `hffs.glob()` call with a pattern that does not contain any wildcards returns an empty list:
```
E       AssertionError: Lists differ: [] != ['datasets/__DUMMY_TRANSFORMERS_USER__/rep[35 chars]tes']
E
E       Second list contains 1 additional elements.
E       First extra element 0:
E       'datasets/__DUMMY_TRANSFORMERS_USER__/repo-7d0ae9-17091013467064/.gitattributes'
E
E       - []
E       + ['datasets/__DUMMY_TRANSFORMERS_USER__/repo-7d0ae9-17091013467064/.gitattributes']
```
(and with the compatible/passing older fsspec versions the glob call returns the single exact file match as expected)

So it looks like the CI failure here isn't directly related to this PR. The failing patterns that don't contain any `*` wildcards are generated by `datasets` with or without this PR, but now that this PR installs the incompatible fsspec version, the underlying `HfFileSystem.glob()` call ends up failing.
2024-02-28T06:33:00Z
https://github.com/huggingface/datasets/pull/6687#issuecomment-1968322475
https://api.github.com/repos/huggingface/datasets/issues/6687
IC_kwDODunzps51Ujer
https://api.github.com/repos/huggingface/datasets/issues/comments/1968322475/reactions
2024-02-28T06:34:30Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1968322475
https://avatars.githubusercontent.com/u/651988?v=4
https://api.github.com/users/pmrowla/events{/privacy}
https://api.github.com/users/pmrowla/followers
https://api.github.com/users/pmrowla/following{/other_user}
https://api.github.com/users/pmrowla/gists{/gist_id}
https://github.com/pmrowla
pmrowla
MDQ6VXNlcjY1MTk4OA==
https://api.github.com/users/pmrowla/orgs
https://api.github.com/users/pmrowla/received_events
https://api.github.com/users/pmrowla/repos
https://api.github.com/users/pmrowla/starred{/owner}{/repo}
https://api.github.com/users/pmrowla/subscriptions
User
https://api.github.com/users/pmrowla
public
MEMBER
I just opened https://github.com/huggingface/huggingface_hub/pull/2056 to fix this.

Do you mind if I continue this PR to run the CI against `huggingface_hub@main` until the fix is released ?

EDIT: the fix has been released in `huggingface_hub` 0.21.2 - I removed my commits that were using `huggingface_hub@main`
2024-02-28T15:08:54Z
https://github.com/huggingface/datasets/pull/6687#issuecomment-1969189366
https://api.github.com/repos/huggingface/datasets/issues/6687
IC_kwDODunzps51X3H2
https://api.github.com/repos/huggingface/datasets/issues/comments/1969189366/reactions
2024-02-28T16:03:49Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1969189366
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
I just added two additional patterns to cover cases like `test-data/xxx.csv` and `data-test/xxx.csv`
2024-02-29T10:57:05Z
https://github.com/huggingface/datasets/pull/6687#issuecomment-1970887074
https://api.github.com/repos/huggingface/datasets/issues/6687
IC_kwDODunzps51eVmi
https://api.github.com/repos/huggingface/datasets/issues/comments/1970887074/reactions
2024-02-29T10:57:05Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1970887074
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005461 / 0.011353 (-0.005892) | 0.003861 / 0.011008 (-0.007148) | 0.063252 / 0.038508 (0.024744) | 0.031474 / 0.023109 (0.008364) | 0.250321 / 0.275898 (-0.025577) | 0.275198 / 0.323480 (-0.048282) | 0.003275 / 0.007986 (-0.004710) | 0.002874 / 0.004328 (-0.001454) | 0.049499 / 0.004250 (0.045248) | 0.045334 / 0.037052 (0.008282) | 0.266347 / 0.258489 (0.007858) | 0.308974 / 0.293841 (0.015133) | 0.027742 / 0.128546 (-0.100804) | 0.010274 / 0.075646 (-0.065373) | 0.207516 / 0.419271 (-0.211755) | 0.036538 / 0.043533 (-0.006995) | 0.247949 / 0.255139 (-0.007190) | 0.268986 / 0.283200 (-0.014214) | 0.019842 / 0.141683 (-0.121841) | 1.117547 / 1.452155 (-0.334607) | 1.175813 / 1.492716 (-0.316903) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.103661 / 0.018006 (0.085655) | 0.331023 / 0.000490 (0.330534) | 0.000240 / 0.000200 (0.000040) | 0.000041 / 0.000054 (-0.000013) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019767 / 0.037411 (-0.017645) | 0.061500 / 0.014526 (0.046974) | 0.075899 / 0.176557 (-0.100658) | 0.122240 / 0.737135 (-0.614895) | 0.074621 / 0.296338 (-0.221717) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.287501 / 0.215209 (0.072292) | 2.794737 / 2.077655 (0.717082) | 1.505362 / 1.504120 (0.001242) | 1.379481 / 1.541195 (-0.161713) | 1.394836 / 1.468490 (-0.073654) | 0.545803 / 4.584777 (-4.038974) | 2.364167 / 3.745712 (-1.381545) | 2.800923 / 5.269862 (-2.468939) | 1.723910 / 4.565676 (-2.841766) | 0.061270 / 0.424275 (-0.363005) | 0.005006 / 0.007607 (-0.002601) | 0.334952 / 0.226044 (0.108908) | 3.367122 / 2.268929 (1.098194) | 1.839822 / 55.444624 (-53.604803) | 1.553774 / 6.876477 (-5.322703) | 1.583585 / 2.142072 (-0.558488) | 0.624680 / 4.805227 (-4.180547) | 0.116364 / 6.500664 (-6.384300) | 0.042412 / 0.075469 (-0.033057) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.975207 / 1.841788 (-0.866580) | 11.843126 / 8.074308 (3.768818) | 9.418537 / 10.191392 (-0.772855) | 0.130648 / 0.680424 (-0.549775) | 0.013747 / 0.534201 (-0.520454) | 0.288195 / 0.579283 (-0.291088) | 0.269861 / 0.434364 (-0.164503) | 0.326732 / 0.540337 (-0.213606) | 0.441256 / 1.386936 (-0.945680) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005185 / 0.011353 (-0.006168) | 0.003836 / 0.011008 (-0.007172) | 0.050057 / 0.038508 (0.011549) | 0.030929 / 0.023109 (0.007820) | 0.263558 / 0.275898 (-0.012340) | 0.284553 / 0.323480 (-0.038927) | 0.004331 / 0.007986 (-0.003655) | 0.002815 / 0.004328 (-0.001513) | 0.050187 / 0.004250 (0.045936) | 0.048431 / 0.037052 (0.011379) | 0.271005 / 0.258489 (0.012515) | 0.304749 / 0.293841 (0.010908) | 0.029286 / 0.128546 (-0.099260) | 0.010598 / 0.075646 (-0.065048) | 0.058111 / 0.419271 (-0.361160) | 0.053665 / 0.043533 (0.010132) | 0.257574 / 0.255139 (0.002436) | 0.285802 / 0.283200 (0.002602) | 0.018917 / 0.141683 (-0.122766) | 1.206517 / 1.452155 (-0.245638) | 1.220572 / 1.492716 (-0.272144) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.122466 / 0.018006 (0.104460) | 0.567887 / 0.000490 (0.567397) | 0.000321 / 0.000200 (0.000121) | 0.000049 / 0.000054 (-0.000006) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022120 / 0.037411 (-0.015292) | 0.075456 / 0.014526 (0.060931) | 0.086385 / 0.176557 (-0.090171) | 0.126045 / 0.737135 (-0.611091) | 0.087502 / 0.296338 (-0.208837) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.304847 / 0.215209 (0.089638) | 3.008095 / 2.077655 (0.930441) | 1.726178 / 1.504120 (0.222058) | 1.592332 / 1.541195 (0.051138) | 1.603714 / 1.468490 (0.135224) | 0.576875 / 4.584777 (-4.007902) | 2.450884 / 3.745712 (-1.294828) | 2.719073 / 5.269862 (-2.550789) | 1.775261 / 4.565676 (-2.790415) | 0.063144 / 0.424275 (-0.361131) | 0.005122 / 0.007607 (-0.002485) | 0.350004 / 0.226044 (0.123960) | 3.467146 / 2.268929 (1.198218) | 2.062907 / 55.444624 (-53.381717) | 1.798793 / 6.876477 (-5.077684) | 1.921204 / 2.142072 (-0.220868) | 0.651832 / 4.805227 (-4.153396) | 0.122326 / 6.500664 (-6.378338) | 0.041396 / 0.075469 (-0.034073) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.024859 / 1.841788 (-0.816928) | 12.569744 / 8.074308 (4.495436) | 10.448487 / 10.191392 (0.257095) | 0.131529 / 0.680424 (-0.548895) | 0.014853 / 0.534201 (-0.519348) | 0.287683 / 0.579283 (-0.291600) | 0.289814 / 0.434364 (-0.144550) | 0.323935 / 0.540337 (-0.216403) | 0.425208 / 1.386936 (-0.961728) |

</details>
</details>

![](https://cml.dev/watermark.png#ba71e92c59c9bd9d1ee6168691977f0c4728ed6e "CML watermark")

2024-02-29T15:18:31Z
https://github.com/huggingface/datasets/pull/6687#issuecomment-1971364583
https://api.github.com/repos/huggingface/datasets/issues/6687
IC_kwDODunzps51gKLn
Iv1.05c79e9ad1f6bdfa
2018-07-30T09:30:17Z
Automate your workflow from idea to production
branch_protection_rule check_run check_suite create delete deployment deployment_status discussion discussion_comment fork gollum issues issue_comment label merge_group milestone page_build project project_card project_column public pull_request pull_request_review pull_request_review_comment push registry_package release repository repository_dispatch status watch workflow_dispatch workflow_run
https://help.github.com/en/actions
https://github.com/apps/github-actions
GitHub Actions
MDM6QXBwMTUzNjg=
https://avatars.githubusercontent.com/u/9919?v=4
https://api.github.com/users/github/events{/privacy}
https://api.github.com/users/github/followers
https://api.github.com/users/github/following{/other_user}
https://api.github.com/users/github/gists{/gist_id}
https://github.com/github
github
MDEyOk9yZ2FuaXphdGlvbjk5MTk=
https://api.github.com/users/github/orgs
https://api.github.com/users/github/received_events
https://api.github.com/users/github/repos
https://api.github.com/users/github/starred{/owner}{/repo}
https://api.github.com/users/github/subscriptions
Organization
https://api.github.com/users/github
public
write
read
write
write
write
write
write
write
write
read
read
write
write
write
write
write
write
write
read
github-actions
2025-03-07T16:35:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1971364583/reactions
2024-02-29T15:18:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1971364583
https://avatars.githubusercontent.com/in/15368?v=4
https://api.github.com/users/github-actions%5Bbot%5D/events{/privacy}
https://api.github.com/users/github-actions%5Bbot%5D/followers
https://api.github.com/users/github-actions%5Bbot%5D/following{/other_user}
https://api.github.com/users/github-actions%5Bbot%5D/gists{/gist_id}
https://github.com/apps/github-actions
github-actions[bot]
MDM6Qm90NDE4OTgyODI=
https://api.github.com/users/github-actions%5Bbot%5D/orgs
https://api.github.com/users/github-actions%5Bbot%5D/received_events
https://api.github.com/users/github-actions%5Bbot%5D/repos
https://api.github.com/users/github-actions%5Bbot%5D/starred{/owner}{/repo}
https://api.github.com/users/github-actions%5Bbot%5D/subscriptions
Bot
https://api.github.com/users/github-actions%5Bbot%5D
public
MEMBER
> EDIT: the fix has been released in `huggingface_hub` 0.21.2 - I removed my commits that were using `huggingface_hub@main`

Please note that people using `huggingface_hub` < 0.21.2 and latest `fsspec` will have issues when using `datasets`:
- https://github.com/huggingface/lighteval/actions/runs/8139147047/job/22241658122?pr=86
- https://github.com/huggingface/lighteval/pull/84

CC: @clefourrier 

2024-03-04T12:38:46Z
https://github.com/huggingface/datasets/pull/6687#issuecomment-1976493336
https://api.github.com/repos/huggingface/datasets/issues/6687
IC_kwDODunzps51zuUY
https://api.github.com/repos/huggingface/datasets/issues/comments/1976493336/reactions
2024-03-04T12:38:46Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1976493336
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
- adds support for the `fs.glob` changes introduced in `fsspec==2023.12.0` and unpins the current upper bound

Should close #6644
Should close #6645

The `test_data_files` glob/pattern tests pass for me in:
- `fsspec==2023.10.0` (the pinned max version in datasets `main`)
- `fsspec==2023.12.0` (#6644)
- `fsspec==2024.2.0` (#6645)