Cached dataset overflowing disk space
MEMBER
Hi! I'm transferring this issue over to `datasets`
2021-07-05T10:44:39Z
https://github.com/huggingface/datasets/issues/2591#issuecomment-874011203
https://api.github.com/repos/huggingface/datasets/issues/2591
MDEyOklzc3VlQ29tbWVudDg3NDAxMTIwMw==
https://api.github.com/repos/huggingface/datasets/issues/comments/874011203/reactions
2021-07-05T10:44:39Z
https://api.github.com/repos/huggingface/datasets/issues/comments/874011203
https://avatars.githubusercontent.com/u/30755778?v=4
https://api.github.com/users/LysandreJik/events{/privacy}
https://api.github.com/users/LysandreJik/followers
https://api.github.com/users/LysandreJik/following{/other_user}
https://api.github.com/users/LysandreJik/gists{/gist_id}
https://github.com/LysandreJik
LysandreJik
MDQ6VXNlcjMwNzU1Nzc4
https://api.github.com/users/LysandreJik/orgs
https://api.github.com/users/LysandreJik/received_events
https://api.github.com/users/LysandreJik/repos
https://api.github.com/users/LysandreJik/starred{/owner}{/repo}
https://api.github.com/users/LysandreJik/subscriptions
User
https://api.github.com/users/LysandreJik
public
CONTRIBUTOR
I'm using the datasets concatenate dataset to combine the datasets and then train.
train_dataset = concatenate_datasets([dataset1, dataset2, common_voice_train])


2021-07-05T10:47:14Z
https://github.com/huggingface/datasets/issues/2591#issuecomment-874012653
https://api.github.com/repos/huggingface/datasets/issues/2591
MDEyOklzc3VlQ29tbWVudDg3NDAxMjY1Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/874012653/reactions
2021-07-05T10:47:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/874012653
https://avatars.githubusercontent.com/u/1704131?v=4
https://api.github.com/users/BirgerMoell/events{/privacy}
https://api.github.com/users/BirgerMoell/followers
https://api.github.com/users/BirgerMoell/following{/other_user}
https://api.github.com/users/BirgerMoell/gists{/gist_id}
https://github.com/BirgerMoell
BirgerMoell
MDQ6VXNlcjE3MDQxMzE=
https://api.github.com/users/BirgerMoell/orgs
https://api.github.com/users/BirgerMoell/received_events
https://api.github.com/users/BirgerMoell/repos
https://api.github.com/users/BirgerMoell/starred{/owner}{/repo}
https://api.github.com/users/BirgerMoell/subscriptions
User
https://api.github.com/users/BirgerMoell
public
MEMBER
Hi @BirgerMoell.

You have several options:
- to set caching to be stored on a different path location, other than the default one (`~/.cache/huggingface/datasets`):
  - either setting the environment variable `HF_DATASETS_CACHE` with the path to the new cache location
  - or by passing it with the parameter `cache_dir` when loading each of the datasets: `dataset = load_dataset(..., cache_dir=your_new_location)`

  You can get all the information in the docs: https://huggingface.co/docs/datasets/loading_datasets.html#cache-directory
- I wouldn't recommend disabling caching, because current implementation generates cache files anyway, although in a temporary directory and they are deleted when the session closes. See details here: https://huggingface.co/docs/datasets/processing.html#enable-or-disable-caching
- You could alternatively load the datasets in streaming mode. This is a new feature which allows loading the datasets without downloading the entire files. More information here: https://huggingface.co/docs/datasets/dataset_streaming.html
2021-07-06T05:37:32Z
https://github.com/huggingface/datasets/issues/2591#issuecomment-874474624
https://api.github.com/repos/huggingface/datasets/issues/2591
MDEyOklzc3VlQ29tbWVudDg3NDQ3NDYyNA==
https://api.github.com/repos/huggingface/datasets/issues/comments/874474624/reactions
2021-07-06T05:37:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/874474624
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
Hi @BirgerMoell,

We are planning to add a new feature to datasets, which could be interesting in your case: Add the option to delete temporary files (decompressed files) from the cache directory (see: #2481, #2604).

We will ping you once this feature is implemented, so that the size of your cache directory will be considerably reduced.
2021-07-08T05:15:16Z
https://github.com/huggingface/datasets/issues/2591#issuecomment-876133704
https://api.github.com/repos/huggingface/datasets/issues/2591
MDEyOklzc3VlQ29tbWVudDg3NjEzMzcwNA==
https://api.github.com/repos/huggingface/datasets/issues/comments/876133704/reactions
2021-07-08T05:15:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/876133704
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
I'm training a Swedish Wav2vec2 model on a Linux GPU and having issues that the huggingface cached dataset folder is completely filling up my disk space (I'm training on a dataset of around 500 gb).

The cache folder is 500gb (and now my disk space is full).

Is there a way to toggle caching or set the caching to be stored on a different device (I have another drive with 4 tb that could hold the caching files).

This might not technically be a bug, but I was unsure and I felt that the bug was the closest one.

Traceback (most recent call last):
  File "/home/birger/miniconda3/envs/wav2vec2/lib/python3.7/site-packages/multiprocess/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/home/birger/miniconda3/envs/wav2vec2/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 186, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/home/birger/miniconda3/envs/wav2vec2/lib/python3.7/site-packages/datasets/fingerprint.py", line 397, in wrapper
    out = func(self, *args, **kwargs)
  File "/home/birger/miniconda3/envs/wav2vec2/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 1983, in _map_single
    writer.finalize()
  File "/home/birger/miniconda3/envs/wav2vec2/lib/python3.7/site-packages/datasets/arrow_writer.py", line 418, in finalize
    self.pa_writer.close()
  File "pyarrow/ipc.pxi", line 402, in pyarrow.lib._CRecordBatchWriter.close
  File "pyarrow/error.pxi", line 97, in pyarrow.lib.check_status
OSError: [Errno 28] Error writing bytes to file. Detail: [errno 28] No space left on device
"""

The above exception was the direct cause of the following exception:
