HuggingFace does not cache downloaded files aggressively/early enough
NONE
I also faced this. Any update?
2023-12-20T06:51:45Z
https://github.com/huggingface/datasets/issues/5896#issuecomment-1863942281
https://api.github.com/repos/huggingface/datasets/issues/5896
IC_kwDODunzps5vGYCJ
https://api.github.com/repos/huggingface/datasets/issues/comments/1863942281/reactions
2023-12-20T06:51:45Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1863942281
https://avatars.githubusercontent.com/u/45726499?v=4
https://api.github.com/users/photomz/events{/privacy}
https://api.github.com/users/photomz/followers
https://api.github.com/users/photomz/following{/other_user}
https://api.github.com/users/photomz/gists{/gist_id}
https://github.com/photomz
photomz
MDQ6VXNlcjQ1NzI2NDk5
https://api.github.com/users/photomz/orgs
https://api.github.com/users/photomz/received_events
https://api.github.com/users/photomz/repos
https://api.github.com/users/photomz/starred{/owner}{/repo}
https://api.github.com/users/photomz/subscriptions
User
https://api.github.com/users/photomz
public
COLLABORATOR
We've dropped the `apache-beam` dependency in https://huggingface.co/datasets/wikipedia/discussions/19, so you should no longer get this error.
2024-03-15T15:36:07Z
https://github.com/huggingface/datasets/issues/5896#issuecomment-1999924693
https://api.github.com/repos/huggingface/datasets/issues/5896
IC_kwDODunzps53NG3V
https://api.github.com/repos/huggingface/datasets/issues/comments/1999924693/reactions
2024-03-15T15:36:07Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1999924693
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
### Describe the bug

I wrote the following script:

```
import datasets

dataset = datasets.load.load_dataset("wikipedia", "20220301.en", split="train[:10000]")
```

I ran it and spent 90 minutes downloading a 20GB file. Then I saw:

```
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20.3G/20.3G [1:30:29<00:00, 3.73MB/s]
Traceback (most recent call last):
  File "/home/jack/Code/Projects/Transformers/Codebase/main.py", line 5, in <module>
    dataset = datasets.load.load_dataset("wikipedia", "20220301.en", split="train[:10000]")
  File "/home/jack/.local/lib/python3.10/site-packages/datasets/load.py", line 1782, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/jack/.local/lib/python3.10/site-packages/datasets/builder.py", line 883, in download_and_prepare
    self._save_info()
  File "/home/jack/.local/lib/python3.10/site-packages/datasets/builder.py", line 2037, in _save_info
    import apache_beam as beam
ModuleNotFoundError: No module named 'apache_beam'
```

And the 20GB of data was seemingly instantly gone forever, because when I ran the script again, it had to do the download again.

### Steps to reproduce the bug

See above

### Expected behavior

See above

### Environment info

datasets 2.10.1
Python 3.10