Not enough disk space (Needed: Unknown size) when caching on a cluster
MEMBER
Hi ! 

What do you mean by "disk_usage(".").free` can't compute on the cluster's shared disk" exactly ?
Does it return 0 ?
2021-01-28T13:32:50Z
https://github.com/huggingface/datasets/issues/1785#issuecomment-769054735
https://api.github.com/repos/huggingface/datasets/issues/1785
MDEyOklzc3VlQ29tbWVudDc2OTA1NDczNQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/769054735/reactions
2021-01-28T13:32:50Z
https://api.github.com/repos/huggingface/datasets/issues/comments/769054735
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Yes, that's right. It shows 0 free space even though there is. I suspect it might have to do with permissions on the shared disk.

```python
>>> disk_usage(".")
usage(total=999999, used=999999, free=0)
```
2021-01-28T15:03:54Z
https://github.com/huggingface/datasets/issues/1785#issuecomment-769145909
https://api.github.com/repos/huggingface/datasets/issues/1785
MDEyOklzc3VlQ29tbWVudDc2OTE0NTkwOQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/769145909/reactions
2021-01-28T15:03:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/769145909
https://avatars.githubusercontent.com/u/4341867?v=4
https://api.github.com/users/olinguyen/events{/privacy}
https://api.github.com/users/olinguyen/followers
https://api.github.com/users/olinguyen/following{/other_user}
https://api.github.com/users/olinguyen/gists{/gist_id}
https://github.com/olinguyen
olinguyen
MDQ6VXNlcjQzNDE4Njc=
https://api.github.com/users/olinguyen/orgs
https://api.github.com/users/olinguyen/received_events
https://api.github.com/users/olinguyen/repos
https://api.github.com/users/olinguyen/starred{/owner}{/repo}
https://api.github.com/users/olinguyen/subscriptions
User
https://api.github.com/users/olinguyen
public
MEMBER
That's an interesting behavior...
Do you know any other way to get the free space that works in your case ?
Also if it's a permission issue could you try fix the permissions and let mus know if that helped ?
2021-01-28T15:09:05Z
https://github.com/huggingface/datasets/issues/1785#issuecomment-769149626
https://api.github.com/repos/huggingface/datasets/issues/1785
MDEyOklzc3VlQ29tbWVudDc2OTE0OTYyNg==
https://api.github.com/repos/huggingface/datasets/issues/comments/769149626/reactions
2021-01-28T15:09:05Z
https://api.github.com/repos/huggingface/datasets/issues/comments/769149626
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
I think its an issue on the clusters end (unclear exactly why -- maybe something with docker containers?), will close the issue
2021-01-30T01:07:56Z
https://github.com/huggingface/datasets/issues/1785#issuecomment-770127633
https://api.github.com/repos/huggingface/datasets/issues/1785
MDEyOklzc3VlQ29tbWVudDc3MDEyNzYzMw==
https://api.github.com/repos/huggingface/datasets/issues/comments/770127633/reactions
2021-01-30T01:07:56Z
https://api.github.com/repos/huggingface/datasets/issues/comments/770127633
https://avatars.githubusercontent.com/u/4341867?v=4
https://api.github.com/users/olinguyen/events{/privacy}
https://api.github.com/users/olinguyen/followers
https://api.github.com/users/olinguyen/following{/other_user}
https://api.github.com/users/olinguyen/gists{/gist_id}
https://github.com/olinguyen
olinguyen
MDQ6VXNlcjQzNDE4Njc=
https://api.github.com/users/olinguyen/orgs
https://api.github.com/users/olinguyen/received_events
https://api.github.com/users/olinguyen/repos
https://api.github.com/users/olinguyen/starred{/owner}{/repo}
https://api.github.com/users/olinguyen/subscriptions
User
https://api.github.com/users/olinguyen
public
NONE
Were you able to figure it out?
2022-05-19T20:32:24Z
https://github.com/huggingface/datasets/issues/1785#issuecomment-1132177694
https://api.github.com/repos/huggingface/datasets/issues/1785
IC_kwDODunzps5De6ke
https://api.github.com/repos/huggingface/datasets/issues/comments/1132177694/reactions
2022-05-19T20:32:24Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1132177694
https://avatars.githubusercontent.com/u/20135561?v=4
https://api.github.com/users/philippnoah/events{/privacy}
https://api.github.com/users/philippnoah/followers
https://api.github.com/users/philippnoah/following{/other_user}
https://api.github.com/users/philippnoah/gists{/gist_id}
https://github.com/philippnoah
philippnoah
MDQ6VXNlcjIwMTM1NTYx
https://api.github.com/users/philippnoah/orgs
https://api.github.com/users/philippnoah/received_events
https://api.github.com/users/philippnoah/repos
https://api.github.com/users/philippnoah/starred{/owner}{/repo}
https://api.github.com/users/philippnoah/subscriptions
User
https://api.github.com/users/philippnoah
public
CONTRIBUTOR
@philippnoah I had fixed it with a small hack where I patched `has_sufficient_disk_space` to always return `True`. you can do that with an import without having to modify the `datasets` package
2022-05-19T20:57:22Z
https://github.com/huggingface/datasets/issues/1785#issuecomment-1132201732
https://api.github.com/repos/huggingface/datasets/issues/1785
IC_kwDODunzps5DfAcE
https://api.github.com/repos/huggingface/datasets/issues/comments/1132201732/reactions
2022-05-19T20:57:22Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1132201732
https://avatars.githubusercontent.com/u/4341867?v=4
https://api.github.com/users/olinguyen/events{/privacy}
https://api.github.com/users/olinguyen/followers
https://api.github.com/users/olinguyen/following{/other_user}
https://api.github.com/users/olinguyen/gists{/gist_id}
https://github.com/olinguyen
olinguyen
MDQ6VXNlcjQzNDE4Njc=
https://api.github.com/users/olinguyen/orgs
https://api.github.com/users/olinguyen/received_events
https://api.github.com/users/olinguyen/repos
https://api.github.com/users/olinguyen/starred{/owner}{/repo}
https://api.github.com/users/olinguyen/subscriptions
User
https://api.github.com/users/olinguyen
public
NONE
@olinguyen Thanks for the suggestion, it works but I had to to edit builder.py in the installed package. Can you please explain how were you able to do this using import?
2022-06-05T14:16:10Z
https://github.com/huggingface/datasets/issues/1785#issuecomment-1146812983
https://api.github.com/repos/huggingface/datasets/issues/1785
IC_kwDODunzps5EWvo3
https://api.github.com/repos/huggingface/datasets/issues/comments/1146812983/reactions
2022-06-05T14:16:10Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1146812983
https://avatars.githubusercontent.com/u/25888741?v=4
https://api.github.com/users/sahutkarsh/events{/privacy}
https://api.github.com/users/sahutkarsh/followers
https://api.github.com/users/sahutkarsh/following{/other_user}
https://api.github.com/users/sahutkarsh/gists{/gist_id}
https://github.com/sahutkarsh
sahutkarsh
MDQ6VXNlcjI1ODg4NzQx
https://api.github.com/users/sahutkarsh/orgs
https://api.github.com/users/sahutkarsh/received_events
https://api.github.com/users/sahutkarsh/repos
https://api.github.com/users/sahutkarsh/starred{/owner}{/repo}
https://api.github.com/users/sahutkarsh/subscriptions
User
https://api.github.com/users/sahutkarsh
public
NONE
I was able to patch the builder code in my notebook before the load data call and it works. 
```
import datasets
datasets.builder.has_sufficient_disk_space = lambda needed_bytes, directory='.': True
```
2022-11-07T16:33:03Z
https://github.com/huggingface/datasets/issues/1785#issuecomment-1305872400
https://api.github.com/repos/huggingface/datasets/issues/1785
IC_kwDODunzps5N1ggQ
https://api.github.com/repos/huggingface/datasets/issues/comments/1305872400/reactions
2022-11-07T16:33:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1305872400
https://avatars.githubusercontent.com/u/910450?v=4
https://api.github.com/users/nitsanluke/events{/privacy}
https://api.github.com/users/nitsanluke/followers
https://api.github.com/users/nitsanluke/following{/other_user}
https://api.github.com/users/nitsanluke/gists{/gist_id}
https://github.com/nitsanluke
nitsanluke
MDQ6VXNlcjkxMDQ1MA==
https://api.github.com/users/nitsanluke/orgs
https://api.github.com/users/nitsanluke/received_events
https://api.github.com/users/nitsanluke/repos
https://api.github.com/users/nitsanluke/starred{/owner}{/repo}
https://api.github.com/users/nitsanluke/subscriptions
User
https://api.github.com/users/nitsanluke
public
NONE
import datasets
    datasets.builder.has_sufficient_disk_space = lambda needed_bytes, directory='.': True
2024-12-04T02:56:59Z
https://github.com/huggingface/datasets/issues/1785#issuecomment-2516061330
https://api.github.com/repos/huggingface/datasets/issues/1785
IC_kwDODunzps6V-AyS
https://api.github.com/repos/huggingface/datasets/issues/comments/2516061330/reactions
2024-12-04T02:56:59Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2516061330
https://avatars.githubusercontent.com/u/33318353?v=4
https://api.github.com/users/sankexin/events{/privacy}
https://api.github.com/users/sankexin/followers
https://api.github.com/users/sankexin/following{/other_user}
https://api.github.com/users/sankexin/gists{/gist_id}
https://github.com/sankexin
sankexin
MDQ6VXNlcjMzMzE4MzUz
https://api.github.com/users/sankexin/orgs
https://api.github.com/users/sankexin/received_events
https://api.github.com/users/sankexin/repos
https://api.github.com/users/sankexin/starred{/owner}{/repo}
https://api.github.com/users/sankexin/subscriptions
User
https://api.github.com/users/sankexin
public
I'm running some experiments where I'm caching datasets on a cluster and accessing it through multiple compute nodes. However, I get an error when loading the cached dataset from the shared disk.

The exact error thrown:

```bash
>>> load_dataset(dataset, cache_dir="/path/to/cluster/shared/path")
OSError: Not enough disk space. Needed: Unknown size (download: Unknown size, generated: Unknown size, post-processed: Unknown size)
```


[`utils.has_sufficient_disk_space`](https://github.com/huggingface/datasets/blob/8a03ab7d123a76ee744304f21ce868c75f411214/src/datasets/utils/py_utils.py#L332) fails on each job because of how the cluster system is designed (`disk_usage(".").free` can't compute on the cluster's shared disk).


This is exactly where the error gets thrown:
https://github.com/huggingface/datasets/blob/master/src/datasets/builder.py#L502

```python
if not utils.has_sufficient_disk_space(self.info.size_in_bytes or 0, directory=self._cache_dir_root):
    raise IOError(
          "Not enough disk space. Needed: {} (download: {}, generated: {}, post-processed: {})".format(
          utils.size_str(self.info.size_in_bytes or 0),
          utils.size_str(self.info.download_size or 0),
          utils.size_str(self.info.dataset_size or 0),
          utils.size_str(self.info.post_processing_size or 0),
       )
    )

```

What would be a good way to circumvent this? my current fix is to manually comment out that part, but that is not ideal. 
Would it be possible to pass a flag to skip this check on disk space?