Infinitely attempting lock
MEMBER
Hi ! Thanks for reporting. We're using filelock` as our locking mechanism.

Can you try deleting the .lock file mentioned in the logs and try again ? Make sure that no other process is generating the `cnn_dailymail` dataset.

If it doesn't work, could you try to set up a lock using  the latest version of filelock` and see if it works ?

```
pip install filelock
```
here is a code example from the `filelock` documentation that you can try:

```python
from filelock import FileLock

lock = FileLock("test.txt.lock")
with lock:
    with open("test.txt", "a") as f:
        f.write("foo")
```
2022-03-28T13:12:57Z
https://github.com/huggingface/datasets/issues/3983#issuecomment-1080633226
https://api.github.com/repos/huggingface/datasets/issues/3983
IC_kwDODunzps5AaSeK
https://api.github.com/repos/huggingface/datasets/issues/comments/1080633226/reactions
2024-04-29T12:40:17Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1080633226
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
I ran into this problem on my school server as well? Any update on how we can solve it? Thanks! 
2024-04-21T19:12:36Z
https://github.com/huggingface/datasets/issues/3983#issuecomment-2068170027
https://api.github.com/repos/huggingface/datasets/issues/3983
IC_kwDODunzps57RcUr
https://api.github.com/repos/huggingface/datasets/issues/comments/2068170027/reactions
2024-04-21T19:12:36Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2068170027
https://avatars.githubusercontent.com/u/36837456?v=4
https://api.github.com/users/TonyJiang17/events{/privacy}
https://api.github.com/users/TonyJiang17/followers
https://api.github.com/users/TonyJiang17/following{/other_user}
https://api.github.com/users/TonyJiang17/gists{/gist_id}
https://github.com/TonyJiang17
TonyJiang17
MDQ6VXNlcjM2ODM3NDU2
https://api.github.com/users/TonyJiang17/orgs
https://api.github.com/users/TonyJiang17/received_events
https://api.github.com/users/TonyJiang17/repos
https://api.github.com/users/TonyJiang17/starred{/owner}{/repo}
https://api.github.com/users/TonyJiang17/subscriptions
User
https://api.github.com/users/TonyJiang17
public
MEMBER
Have you tried running the code above to check if FileLock works in your setup ? You may also be interested in checking the https://github.com/tox-dev/filelock repository for issues
2024-04-29T12:40:31Z
https://github.com/huggingface/datasets/issues/3983#issuecomment-2082626321
https://api.github.com/repos/huggingface/datasets/issues/3983
IC_kwDODunzps58IlsR
https://api.github.com/repos/huggingface/datasets/issues/comments/2082626321/reactions
2024-04-29T12:40:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2082626321
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Can you try using a different cache directory ? Maybe there are permissions issues with the default one.

You can do so by passing `cache_dir=...` to load_dataset()
2024-05-07T12:38:32Z
https://github.com/huggingface/datasets/issues/3983#issuecomment-2098310862
https://api.github.com/repos/huggingface/datasets/issues/3983
IC_kwDODunzps59Ea7O
https://api.github.com/repos/huggingface/datasets/issues/comments/2098310862/reactions
2024-05-07T12:38:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2098310862
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
I am trying to run one of the examples of the `transformers` repo, which makes use of `datasets`.
Important to note is that I am trying to run this via a Databricks notebook, and all the files reside in the Databricks Filesystem (DBFS).

```
%sh
python /dbfs/transformers/examples/pytorch/summarization/run_summarization.py \
    --model_name_or_path t5-small \
    --do_train \
    --do_eval \
    --dataset_name cnn_dailymail \
    --dataset_config "3.0.0" \
    --source_prefix "summarize: " \
    --output_dir /dbfs/transformers/tmp/tst-summarization \
    --per_device_train_batch_size=4 \
    --per_device_eval_batch_size=4 \
    --overwrite_output_dir \
    --predict_with_generate \
    --log_level debug \
    --cache_dir /dbfs/transformers/cache
```

All goes well until acquiring a lock --

```
03/21/2022 17:53:19 - DEBUG - datasets.utils.filelock - Attempting to acquire lock 140386484514192 on /dbfs/transformers/cache/_dbfs_transformers_cache_cnn_dailymail_3.0.0_3.0.0_3cb851bf7cf5826e45d49db2863f627cba583cbc32342df7349dfe6c38060234.lock
03/21/2022 17:53:19 - DEBUG - datasets.utils.filelock - Lock 140386484514192 not acquired on /dbfs/transformers/cache/_dbfs_transformers_cache_cnn_dailymail_3.0.0_3.0.0_3cb851bf7cf5826e45d49db2863f627cba583cbc32342df7349dfe6c38060234.lock, waiting 0.05 seconds ...
03/21/2022 17:53:20 - DEBUG - datasets.utils.filelock - Attempting to acquire lock 140386484514192 on /dbfs/transformers/cache/_dbfs_transformers_cache_cnn_dailymail_3.0.0_3.0.0_3cb851bf7cf5826e45d49db2863f627cba583cbc32342df7349dfe6c38060234.lock
03/21/2022 17:53:20 - DEBUG - datasets.utils.filelock - Lock 140386484514192 not acquired on /dbfs/transformers/cache/_dbfs_transformers_cache_cnn_dailymail_3.0.0_3.0.0_3cb851bf7cf5826e45d49db2863f627cba583cbc32342df7349dfe6c38060234.lock, waiting 0.05 seconds ...
03/21/2022 17:53:20 - DEBUG - datasets.utils.filelock - Attempting to acquire lock 140386484514192 on /dbfs/transformers/cache/_dbfs_transformers_cache_cnn_dailymail_3.0.0_3.0.0_3cb851bf7cf5826e45d49db2863f627cba583cbc32342df7349dfe6c38060234.lock
03/21/2022 17:53:20 - DEBUG - datasets.utils.filelock - Lock 140386484514192 not acquired on /dbfs/transformers/cache/_dbfs_transformers_cache_cnn_dailymail_3.0.0_3.0.0_3cb851bf7cf5826e45d49db2863f627cba583cbc32342df7349dfe6c38060234.lock, waiting 0.05 seconds ...
03/21/2022 17:53:20 - DEBUG - datasets.utils.filelock - Attempting to acquire lock 140386484514192 on /dbfs/transformers/cache/_dbfs_transformers_cache_cnn_dailymail_3.0.0_3.0.0_3cb851bf7cf5826e45d49db2863f627cba583cbc32342df7349dfe6c38060234.lock
03/21/2022 17:53:20 - DEBUG - datasets.utils.filelock - Lock 140386484514192 not acquired on /dbfs/transformers/cache/_dbfs_transformers_cache_cnn_dailymail_3.0.0_3.0.0_3cb851bf7cf5826e45d49db2863f627cba583cbc32342df7349dfe6c38060234.lock, waiting 0.05 seconds ...
03/21/2022 17:53:20 - DEBUG - datasets.utils.filelock - Attempting to acquire lock 140386484514192 on /dbfs/transformers/cache/_dbfs_transformers_cache_cnn_dailymail_3.0.0_3.0.0_3cb851bf7cf5826e45d49db2863f627cba583cbc32342df7349dfe6c38060234.lock
03/21/2022 17:53:20 - DEBUG - datasets.utils.filelock - Lock 140386484514192 not acquired on /dbfs/transformers/cache/_dbfs_transformers_cache_cnn_dailymail_3.0.0_3.0.0_3cb851bf7cf5826e45d49db2863f627cba583cbc32342df7349dfe6c38060234.lock, waiting 0.05 seconds ...
03/21/2022 17:53:20 - DEBUG - datasets.utils.filelock - Attempting to acquire lock 140386484514192 on /dbfs/transformers/cache/_dbfs_transformers_cache_cnn_dailymail_3.0.0_3.0.0_3cb851bf7cf5826e45d49db2863f627cba583cbc32342df7349dfe6c38060234.lock
03/21/2022 17:53:20 - DEBUG - datasets.utils.filelock - Lock 140386484514192 not acquired on /dbfs/transformers/cache/_dbfs_transformers_cache_cnn_dailymail_3.0.0_3.0.0_3cb851bf7cf5826e45d49db2863f627cba583cbc32342df7349dfe6c38060234.lock, waiting 0.05 seconds ...
```

and so on.

I imagine this has to do with DBFS -- is there a way to tackle this?