Dataset loads indefinitely after modifying default cache path (~/.cache/huggingface)
MEMBER
Hi ! I didn't managed to reproduce the issue. When you kill the process, is there any stacktrace that shows at what point in the code python is hanging ?
2022-03-28T13:23:26Z
https://github.com/huggingface/datasets/issues/3986#issuecomment-1080646047
https://api.github.com/repos/huggingface/datasets/issues/3986
IC_kwDODunzps5AaVmf
https://api.github.com/repos/huggingface/datasets/issues/comments/1080646047/reactions
2022-03-28T13:23:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1080646047
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Hi @lhoestq , I've traced the issue back to file locking. It's similar to this thread, using Lustre filesystem as well. https://github.com/huggingface/datasets/issues/329 . In this case the user was able to modify and add -o flock option while mounting and it solved the problem.  
However in other cases such as mine, we do not have the permissions to modify the commands while mounting. I'm still trying to figure out a workaround. Any ideas how can we use a mounted Lustre filesystem with no flock option?

2022-03-31T03:50:20Z
https://github.com/huggingface/datasets/issues/3986#issuecomment-1084044354
https://api.github.com/repos/huggingface/datasets/issues/3986
IC_kwDODunzps5AnTRC
https://api.github.com/repos/huggingface/datasets/issues/comments/1084044354/reactions
2022-03-31T03:51:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1084044354
https://avatars.githubusercontent.com/u/10686779?v=4
https://api.github.com/users/kelvinAI/events{/privacy}
https://api.github.com/users/kelvinAI/followers
https://api.github.com/users/kelvinAI/following{/other_user}
https://api.github.com/users/kelvinAI/gists{/gist_id}
https://github.com/kelvinAI
kelvinAI
MDQ6VXNlcjEwNjg2Nzc5
https://api.github.com/users/kelvinAI/orgs
https://api.github.com/users/kelvinAI/received_events
https://api.github.com/users/kelvinAI/repos
https://api.github.com/users/kelvinAI/starred{/owner}{/repo}
https://api.github.com/users/kelvinAI/subscriptions
User
https://api.github.com/users/kelvinAI
public
NONE
Hi @kelvinAI , I've had this issue on our institution's system which uses Lustre (in addition to our compute nodes being siloed off from external network access). The workaround I made for downloading/loading datasets was to set the `$HFHOME` environment variable to a location on the node's local storage (SSD), effectively a location that gets cleared regularly and sometimes gets used for temporary or cached files which is pretty common, e.g. "scratch" storage. Maybe your sysadmins, if you have them, could point you to subdirectories on a node that aren't linked to the Lustre filesystem. After downloading to scratch I found that the transformers, modules, and metrics cached folders were fine to move to my user drives on the Lustre filesystem but cached datasets that had fingerprints still had some issues with filelock, so it would help to use the function `my_dataset.save_to_disk('path/on/lustre_fs')` and static class function `Dataset.load_from_disk('path/on/lustre_fs')`. In rough steps:

1. Initially download to scratch storage with `ds = datasets.load_dataset(dataset_name)`
2. Call `ds.save_to_disk(my_path_on_lustre)` with a path in your user space on the Lustre filesystem
3. Load datasets with `from datasets import Dataset; new_ds = Dataset.load_from_disk(my_path_on_lustre)`

Obviously this hinges on there existing scratch storage on the nodes you're using. Fingers crossed.
2022-06-02T17:13:36Z
https://github.com/huggingface/datasets/issues/3986#issuecomment-1145106342
https://api.github.com/repos/huggingface/datasets/issues/3986
IC_kwDODunzps5EQO-m
https://api.github.com/repos/huggingface/datasets/issues/comments/1145106342/reactions
2022-06-02T17:13:36Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1145106342
https://avatars.githubusercontent.com/u/7060571?v=4
https://api.github.com/users/jpmcd/events{/privacy}
https://api.github.com/users/jpmcd/followers
https://api.github.com/users/jpmcd/following{/other_user}
https://api.github.com/users/jpmcd/gists{/gist_id}
https://github.com/jpmcd
jpmcd
MDQ6VXNlcjcwNjA1NzE=
https://api.github.com/users/jpmcd/orgs
https://api.github.com/users/jpmcd/received_events
https://api.github.com/users/jpmcd/repos
https://api.github.com/users/jpmcd/starred{/owner}{/repo}
https://api.github.com/users/jpmcd/subscriptions
User
https://api.github.com/users/jpmcd
public
NONE
Hi @jpmcd , thanks for sharing your experience. For my case, the Lustre filesystem (with more storage space) is the scratch storage like the one you've mentioned. We have a local storage for each user but unfortunately there's not enough space in it to 'cache' huge datasets, hence that is why I tried changing HF_HOME to point to the scratch disk with more space and encountered the flock issue. Unfortunately I'm not aware of any viable solution to this for now so I simply fall back to using torch dataset. 
2022-06-14T07:20:07Z
https://github.com/huggingface/datasets/issues/3986#issuecomment-1154811093
https://api.github.com/repos/huggingface/datasets/issues/3986
IC_kwDODunzps5E1QTV
https://api.github.com/repos/huggingface/datasets/issues/comments/1154811093/reactions
2022-06-14T07:20:07Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1154811093
https://avatars.githubusercontent.com/u/10686779?v=4
https://api.github.com/users/kelvinAI/events{/privacy}
https://api.github.com/users/kelvinAI/followers
https://api.github.com/users/kelvinAI/following{/other_user}
https://api.github.com/users/kelvinAI/gists{/gist_id}
https://github.com/kelvinAI
kelvinAI
MDQ6VXNlcjEwNjg2Nzc5
https://api.github.com/users/kelvinAI/orgs
https://api.github.com/users/kelvinAI/received_events
https://api.github.com/users/kelvinAI/repos
https://api.github.com/users/kelvinAI/starred{/owner}{/repo}
https://api.github.com/users/kelvinAI/subscriptions
User
https://api.github.com/users/kelvinAI
public
NONE
@jpmcd your comment saved me from pulling my hair out in frustration. Setting `HF_HOME` to a directory that's not on Lustre works like a charm. ✨ 
2023-03-06T16:55:04Z
https://github.com/huggingface/datasets/issues/3986#issuecomment-1456512901
https://api.github.com/repos/huggingface/datasets/issues/3986
IC_kwDODunzps5W0J-F
https://api.github.com/repos/huggingface/datasets/issues/comments/1456512901/reactions
2023-03-06T16:55:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1456512901
https://avatars.githubusercontent.com/u/37114774?v=4
https://api.github.com/users/minaremeli/events{/privacy}
https://api.github.com/users/minaremeli/followers
https://api.github.com/users/minaremeli/following{/other_user}
https://api.github.com/users/minaremeli/gists{/gist_id}
https://github.com/minaremeli
minaremeli
MDQ6VXNlcjM3MTE0Nzc0
https://api.github.com/users/minaremeli/orgs
https://api.github.com/users/minaremeli/received_events
https://api.github.com/users/minaremeli/repos
https://api.github.com/users/minaremeli/starred{/owner}{/repo}
https://api.github.com/users/minaremeli/subscriptions
User
https://api.github.com/users/minaremeli
public
## Describe the bug
Dataset loads indefinitely after modifying cache path (~/.cache/huggingface)
If none of the environment variables are set, this custom dataset loads fine ( json-based dataset with custom dataset load script)
** Update: Transformer modules faces the same issue as well during loading

## A clear and concise description of what the bug is.

Issue:
- Dataset loading stalls / freezes indefinitely when HF_HOME is changed to a custom directory
- No error code, had to terminate the process
- There are some files created in the cache directory:

```
custom_cache_dir  
  | -- modules  
            | -- __init__.py  
            | -- datasets_modules  
                      | -- __init__.py  
                      | -- datasets  
                             | -- __init__.py  
                             | -- script.py (Dataset loading script)  
                             | -- script.lock   

```

There's no error nor any logs thrown so I'm out of ideas of how to to debug this. The custom dataset works fine if the default ~/.cache dir is used, but unfortunately it's out of space and we do not have permissions to modify the disk.

## Steps to reproduce the bug

What I've tried:
- Modifying HF_HOME  (https://github.com/huggingface/transformers/issues/8703)
- Modifying HF_DATASETS_CACHE (https://huggingface.co/docs/datasets/v1.12.0/cache.html)
- Modifying cache_dir param during runtime
```python
>>> from datasets import load_dataset
>>> dataset = load_dataset('test_dataset', cache_dir='/path/to/new/cache')
```
- Disabling dataset cache
```python
>>> from datasets import set_caching_enabled
>>> set_caching_enabled(False)
```

## Expected results
Datasets should load / cache as usual with the only exception that cache directory is different

## Actual results

Any actions taken above to change the cache directory results in loading indefinitely without terminating.

## Environment info

- `transformers` version: 4.18.0.dev0
- Platform: Linux-4.15.0-54-generic-x86_64-with-glibc2.10
- Python version: 3.8.8
- Huggingface_hub version: 0.4.0
- PyTorch version (GPU?): 1.8.1+cu102 (True)
- Tensorflow version (GPU?): 2.4.1 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

