fsspec.exceptions.FSTimeoutError when downloading dataset
MEMBER
Hi ! If you check the dataset loading script [here](https://huggingface.co/datasets/openslr/librispeech_asr/blob/main/librispeech_asr.py) you'll see that it downloads the data from OpenSLR, and apparently their storage has timeout issues. It would be great to ultimately host the dataset on Hugging Face instead.

In the meantime I can only recommend to try again later :/
2024-09-24T14:36:21Z
https://github.com/huggingface/datasets/issues/7164#issuecomment-2371491116
https://api.github.com/repos/huggingface/datasets/issues/7164
IC_kwDODunzps6NWhUs
https://api.github.com/repos/huggingface/datasets/issues/comments/2371491116/reactions
2024-09-24T14:36:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2371491116
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Ok, still many thanks!
2024-09-24T15:35:55Z
https://github.com/huggingface/datasets/issues/7164#issuecomment-2371654596
https://api.github.com/repos/huggingface/datasets/issues/7164
IC_kwDODunzps6NXJPE
https://api.github.com/repos/huggingface/datasets/issues/comments/2371654596/reactions
2024-09-24T15:35:55Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2371654596
https://avatars.githubusercontent.com/u/38216460?v=4
https://api.github.com/users/timonmerk/events{/privacy}
https://api.github.com/users/timonmerk/followers
https://api.github.com/users/timonmerk/following{/other_user}
https://api.github.com/users/timonmerk/gists{/gist_id}
https://github.com/timonmerk
timonmerk
MDQ6VXNlcjM4MjE2NDYw
https://api.github.com/users/timonmerk/orgs
https://api.github.com/users/timonmerk/received_events
https://api.github.com/users/timonmerk/repos
https://api.github.com/users/timonmerk/starred{/owner}{/repo}
https://api.github.com/users/timonmerk/subscriptions
User
https://api.github.com/users/timonmerk
public
NONE
I'm also getting this same error but for `CSTR-Edinburgh/vctk`, so I don't think it's the remote host that's timing out, since I also time out at exactly 5 minutes.  It seems there is a universal fsspec timeout that's getting hit starting in v3.
2024-10-23T19:15:16Z
https://github.com/huggingface/datasets/issues/7164#issuecomment-2433236143
https://api.github.com/repos/huggingface/datasets/issues/7164
IC_kwDODunzps6RCDyv
https://api.github.com/repos/huggingface/datasets/issues/comments/2433236143/reactions
2024-10-23T19:15:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2433236143
https://avatars.githubusercontent.com/u/4621169?v=4
https://api.github.com/users/Epiphero/events{/privacy}
https://api.github.com/users/Epiphero/followers
https://api.github.com/users/Epiphero/following{/other_user}
https://api.github.com/users/Epiphero/gists{/gist_id}
https://github.com/Epiphero
Epiphero
MDQ6VXNlcjQ2MjExNjk=
https://api.github.com/users/Epiphero/orgs
https://api.github.com/users/Epiphero/received_events
https://api.github.com/users/Epiphero/repos
https://api.github.com/users/Epiphero/starred{/owner}{/repo}
https://api.github.com/users/Epiphero/subscriptions
User
https://api.github.com/users/Epiphero
public
MEMBER
in v3 we cleaned the download parts of the library to make it more robust for HF downloads and to simplify support of script-based datasets. As a side effect it's not the same code that is used for other hosts, maybe time out handling changed. Anyway it should be possible to tweak fsspec to use retries

For example using [aiohttp_retry](https://github.com/inyutin/aiohttp_retry) maybe (haven't tried) ?

```python
import fsspec
from aiohttp_retry import RetryClient

fsspec.filesystem("http")._session = RetryClient()
```

related topic : https://github.com/huggingface/datasets/issues/7175
2024-10-24T13:14:38Z
https://github.com/huggingface/datasets/issues/7164#issuecomment-2435267713
https://api.github.com/repos/huggingface/datasets/issues/7164
IC_kwDODunzps6RJzyB
https://api.github.com/repos/huggingface/datasets/issues/comments/2435267713/reactions
2024-10-24T13:17:23Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2435267713
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Adding a timeout argument to the `fs.get_file` call in `fsspec_get` in `datasets/utils/file_utils.py` might fix this ([source code](https://github.com/huggingface/datasets/blob/65f6eb54aa0e8bb44cea35deea28e0e8fecc25b9/src/datasets/utils/file_utils.py#L330)):

```python
fs.get_file(path, temp_file.name, callback=callback, timeout=3600)
```

Setting `timeout=1` fails after about one second, so setting it to 3600 should give us 1h. Havn't really tested this though. I'm also not sure what implications this has and if it causes errors for other `fs` implementations/configurations.

This is using `datasets==3.0.1` and Python 3.11.6.

---

Edit: This doesn't seem to change the timeout time, but add a second timeout counter (probably in `fsspec/asyn.py/sync`). So one can reduce the time for downloading like this, but not expand.

---

Edit 2: `fs` is of type `fsspec.implementations.http.HTTPFileSystem` which initializes a `aiohttp.ClientSession` using `client_kwargs`. We can pass these when calling `load_dataset`.

**TLDR; This fixes it:**

```python
import datasets, aiohttp
dataset = datasets.load_dataset(
    dataset_name,
    storage_options={'client_kwargs': {'timeout': aiohttp.ClientTimeout(total=3600)}}
)
```
2024-10-26T13:47:46Z
https://github.com/huggingface/datasets/issues/7164#issuecomment-2439589751
https://api.github.com/repos/huggingface/datasets/issues/7164
IC_kwDODunzps6RaS93
https://api.github.com/repos/huggingface/datasets/issues/comments/2439589751/reactions
2024-10-26T15:11:13Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2439589751
https://avatars.githubusercontent.com/u/33965649?v=4
https://api.github.com/users/JonasLoos/events{/privacy}
https://api.github.com/users/JonasLoos/followers
https://api.github.com/users/JonasLoos/following{/other_user}
https://api.github.com/users/JonasLoos/gists{/gist_id}
https://github.com/JonasLoos
JonasLoos
MDQ6VXNlcjMzOTY1NjQ5
https://api.github.com/users/JonasLoos/orgs
https://api.github.com/users/JonasLoos/received_events
https://api.github.com/users/JonasLoos/repos
https://api.github.com/users/JonasLoos/starred{/owner}{/repo}
https://api.github.com/users/JonasLoos/subscriptions
User
https://api.github.com/users/JonasLoos
public
NONE
I've handled the issue like this to ensure smoother downloads when using the `datasets` library.  
If modifying the library is not too inconvenient, this approach could be a good (but tentative) solution.

### Changes Made

Modified `datasets.utils.file_utils.fsspec_get` to handle storage options and set a timeout:

```python
def fsspec_get(url, temp_file, storage_options=None, desc=None, disable_tqdm=False):

    # ---> [ADD]
    if storage_options is None:
        storage_options = {}
    if "client_kwargs" not in storage_options:
        storage_options["client_kwargs"] = {}
    storage_options["client_kwargs"]["timeout"] = aiohttp.ClientTimeout(total=3600)
    # <---

    # The rest of the original code remains unchanged
2025-01-14T09:48:22Z
https://github.com/huggingface/datasets/issues/7164#issuecomment-2589462969
https://api.github.com/repos/huggingface/datasets/issues/7164
IC_kwDODunzps6aWBG5
https://api.github.com/repos/huggingface/datasets/issues/comments/2589462969/reactions
2025-01-14T09:48:22Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2589462969
https://avatars.githubusercontent.com/u/33245427?v=4
https://api.github.com/users/eleventh83/events{/privacy}
https://api.github.com/users/eleventh83/followers
https://api.github.com/users/eleventh83/following{/other_user}
https://api.github.com/users/eleventh83/gists{/gist_id}
https://github.com/eleventh83
eleventh83
MDQ6VXNlcjMzMjQ1NDI3
https://api.github.com/users/eleventh83/orgs
https://api.github.com/users/eleventh83/received_events
https://api.github.com/users/eleventh83/repos
https://api.github.com/users/eleventh83/starred{/owner}{/repo}
https://api.github.com/users/eleventh83/subscriptions
User
https://api.github.com/users/eleventh83
public
MEMBER
Librispeech_asr is now hosted on HF, which fixes this issue 🎉

closing this one now :)
2025-07-28T14:58:49Z
https://github.com/huggingface/datasets/issues/7164#issuecomment-3127633430
https://api.github.com/repos/huggingface/datasets/issues/7164
IC_kwDODunzps66a-YW
https://api.github.com/repos/huggingface/datasets/issues/comments/3127633430/reactions
2025-07-28T14:58:49Z
https://api.github.com/repos/huggingface/datasets/issues/comments/3127633430
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
### Describe the bug

I am trying to download the `librispeech_asr` `clean` dataset, which results in a `FSTimeoutError` exception after downloading around 61% of the data.

### Steps to reproduce the bug

```
import datasets
datasets.load_dataset("librispeech_asr", "clean")
```

The output is as follows:

> Downloading data:  61%|██████████████▋         | 3.92G/6.39G [05:00<03:06, 13.2MB/s]Traceback (most recent call last):
>   File "/Users/Timon/Documents/iEEG_deeplearning/wav2vec_pretrain/.venv/lib/python3.12/site-packages/fsspec/asyn.py", line 56, in _runner
>     result[0] = await coro
>                 ^^^^^^^^^^
>   File "/Users/Timon/Documents/iEEG_deeplearning/wav2vec_pretrain/.venv/lib/python3.12/site-packages/fsspec/implementations/http.py", line 262, in _get_file
>     chunk = await r.content.read(chunk_size)
>             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/Users/Timon/Documents/iEEG_deeplearning/wav2vec_pretrain/.venv/lib/python3.12/site-packages/aiohttp/streams.py", line 393, in read
>     await self._wait("read")
>   File "/Users/Timon/Documents/iEEG_deeplearning/wav2vec_pretrain/.venv/lib/python3.12/site-packages/aiohttp/streams.py", line 311, in _wait
>     with self._timer:
>          ^^^^^^^^^^^
>   File "/Users/Timon/Documents/iEEG_deeplearning/wav2vec_pretrain/.venv/lib/python3.12/site-packages/aiohttp/helpers.py", line 713, in __exit__
>     raise asyncio.TimeoutError from None
> TimeoutError
> 
> The above exception was the direct cause of the following exception:
> 
> Traceback (most recent call last):
>   File "/Users/Timon/Documents/iEEG_deeplearning/wav2vec_pretrain/load_dataset.py", line 3, in <module>
>     datasets.load_dataset("librispeech_asr", "clean")
>   File "/Users/Timon/Documents/iEEG_deeplearning/wav2vec_pretrain/.venv/lib/python3.12/site-packages/datasets/load.py", line 2096, in load_dataset
>     builder_instance.download_and_prepare(
>   File "/Users/Timon/Documents/iEEG_deeplearning/wav2vec_pretrain/.venv/lib/python3.12/site-packages/datasets/builder.py", line 924, in download_and_prepare
>     self._download_and_prepare(
>   File "/Users/Timon/Documents/iEEG_deeplearning/wav2vec_pretrain/.venv/lib/python3.12/site-packages/datasets/builder.py", line 1647, in _download_and_prepare
>     super()._download_and_prepare(
>   File "/Users/Timon/Documents/iEEG_deeplearning/wav2vec_pretrain/.venv/lib/python3.12/site-packages/datasets/builder.py", line 977, in _download_and_prepare
>     split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
>                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/Users/Timon/.cache/huggingface/modules/datasets_modules/datasets/librispeech_asr/2712a8f82f0d20807a56faadcd08734f9bdd24c850bb118ba21ff33ebff0432f/librispeech_asr.py", line 115, in _split_generators
>     archive_path = dl_manager.download(_DL_URLS[self.config.name])
>                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/Users/Timon/Documents/iEEG_deeplearning/wav2vec_pretrain/.venv/lib/python3.12/site-packages/datasets/download/download_manager.py", line 159, in download
>     downloaded_path_or_paths = map_nested(
>                                ^^^^^^^^^^^
>   File "/Users/Timon/Documents/iEEG_deeplearning/wav2vec_pretrain/.venv/lib/python3.12/site-packages/datasets/utils/py_utils.py", line 512, in map_nested
>     _single_map_nested((function, obj, batched, batch_size, types, None, True, None))
>   File "/Users/Timon/Documents/iEEG_deeplearning/wav2vec_pretrain/.venv/lib/python3.12/site-packages/datasets/utils/py_utils.py", line 380, in _single_map_nested
>     return [mapped_item for batch in iter_batched(data_struct, batch_size) for mapped_item in function(batch)]
>                                                                                               ^^^^^^^^^^^^^^^
>   File "/Users/Timon/Documents/iEEG_deeplearning/wav2vec_pretrain/.venv/lib/python3.12/site-packages/datasets/download/download_manager.py", line 216, in _download_batched
>     self._download_single(url_or_filename, download_config=download_config)
>   File "/Users/Timon/Documents/iEEG_deeplearning/wav2vec_pretrain/.venv/lib/python3.12/site-packages/datasets/download/download_manager.py", line 225, in _download_single
>     out = cached_path(url_or_filename, download_config=download_config)
>           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/Users/Timon/Documents/iEEG_deeplearning/wav2vec_pretrain/.venv/lib/python3.12/site-packages/datasets/utils/file_utils.py", line 205, in cached_path
>     output_path = get_from_cache(
>                   ^^^^^^^^^^^^^^^
>   File "/Users/Timon/Documents/iEEG_deeplearning/wav2vec_pretrain/.venv/lib/python3.12/site-packages/datasets/utils/file_utils.py", line 415, in get_from_cache
>     fsspec_get(url, temp_file, storage_options=storage_options, desc=download_desc, disable_tqdm=disable_tqdm)
>   File "/Users/Timon/Documents/iEEG_deeplearning/wav2vec_pretrain/.venv/lib/python3.12/site-packages/datasets/utils/file_utils.py", line 334, in fsspec_get
>     fs.get_file(path, temp_file.name, callback=callback)
>   File "/Users/Timon/Documents/iEEG_deeplearning/wav2vec_pretrain/.venv/lib/python3.12/site-packages/fsspec/asyn.py", line 118, in wrapper
>     return sync(self.loop, func, *args, **kwargs)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/Users/Timon/Documents/iEEG_deeplearning/wav2vec_pretrain/.venv/lib/python3.12/site-packages/fsspec/asyn.py", line 101, in sync
>     raise FSTimeoutError from return_result
> fsspec.exceptions.FSTimeoutError
> Downloading data:  61%|██████████████▋         | 3.92G/6.39G [05:00<03:09, 13.0MB/s]

### Expected behavior

Complete the download

### Environment info

Python version 3.12.6

Dependencies:
> dependencies = [
>     "accelerate>=0.34.2",
>     "datasets[audio]>=3.0.0",
>     "ipython>=8.18.1",
>     "librosa>=0.10.2.post1",
>     "torch>=2.4.1",
>     "torchaudio>=2.4.1",
>     "transformers>=4.44.2",
> ]

MacOS 14.6.1 (23G93)