"File name too long" error for file locks
MEMBER
Hi, the filename here is less than 255
```python
>>> len("_home_garrett_.cache_huggingface_datasets_csv_test-7c856aea083a7043_0.0.0_9144e0a4e8435090117cea53e6c7537173ef2304525df4a077c435d8ee7828ff.incomplete.lock")
154
```
so not sure why it's considered too long for your filesystem.
(also note that the lock files we use always have smaller filenames than 255)

https://github.com/huggingface/datasets/blob/5d1a9f1e3c6c495dc0610b459e39d2eb8893f152/src/datasets/utils/filelock.py#L135-L135
2021-09-21T13:52:19Z
https://github.com/huggingface/datasets/issues/2924#issuecomment-924011309
https://api.github.com/repos/huggingface/datasets/issues/2924
IC_kwDODunzps43E0st
https://api.github.com/repos/huggingface/datasets/issues/comments/924011309/reactions
2021-09-21T13:52:19Z
https://api.github.com/repos/huggingface/datasets/issues/comments/924011309
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Yes, you're right! I need to get you more info here. Either there's something going with the name itself that the file system doesn't like (an encoding that blows up the name length??) or perhaps there's something with the path that's causing the entire string to  be used as a name. I haven't seen this on any system before and the Internet's not forthcoming with any info.
2021-09-21T14:03:13Z
https://github.com/huggingface/datasets/issues/2924#issuecomment-924021257
https://api.github.com/repos/huggingface/datasets/issues/2924
IC_kwDODunzps43E3IJ
https://api.github.com/repos/huggingface/datasets/issues/comments/924021257/reactions
2021-09-21T14:03:13Z
https://api.github.com/repos/huggingface/datasets/issues/comments/924021257
https://avatars.githubusercontent.com/u/184949?v=4
https://api.github.com/users/gar1t/events{/privacy}
https://api.github.com/users/gar1t/followers
https://api.github.com/users/gar1t/following{/other_user}
https://api.github.com/users/gar1t/gists{/gist_id}
https://github.com/gar1t
gar1t
MDQ6VXNlcjE4NDk0OQ==
https://api.github.com/users/gar1t/orgs
https://api.github.com/users/gar1t/received_events
https://api.github.com/users/gar1t/repos
https://api.github.com/users/gar1t/starred{/owner}{/repo}
https://api.github.com/users/gar1t/subscriptions
User
https://api.github.com/users/gar1t
public
NONE
Snap, encountered when trying to run [this example from PyTorch Lightning Flash](https://lightning-flash.readthedocs.io/en/latest/reference/speech_recognition.html):

```py
import torch

import flash
from flash.audio import SpeechRecognition, SpeechRecognitionData
from flash.core.data.utils import download_data

# 1. Create the DataModule
download_data("https://pl-flash-data.s3.amazonaws.com/timit_data.zip", "./data")

datamodule = SpeechRecognitionData.from_json(
    input_fields="file",
    target_fields="text",
    train_file="data/timit/train.json",
    test_file="data/timit/test.json",
)
```

Gave this traceback:

```py
Traceback (most recent call last):
  File "lf_ft.py", line 10, in <module>
    datamodule = SpeechRecognitionData.from_json(
  File "/home/louis/miniconda3/envs/w2vlf/lib/python3.8/site-packages/flash/core/data/data_module.py", line 1005, in from_json
    return cls.from_data_source(
  File "/home/louis/miniconda3/envs/w2vlf/lib/python3.8/site-packages/flash/core/data/data_module.py", line 571, in from_data_source
    train_dataset, val_dataset, test_dataset, predict_dataset = data_source.to_datasets(
  File "/home/louis/miniconda3/envs/w2vlf/lib/python3.8/site-packages/flash/core/data/data_source.py", line 307, in to_datasets
    train_dataset = self.generate_dataset(train_data, RunningStage.TRAINING)
  File "/home/louis/miniconda3/envs/w2vlf/lib/python3.8/site-packages/flash/core/data/data_source.py", line 344, in generate_dataset
    data = load_data(data, mock_dataset)
  File "/home/louis/miniconda3/envs/w2vlf/lib/python3.8/site-packages/flash/audio/speech_recognition/data.py", line 103, in load_data
    dataset_dict = load_dataset(self.filetype, data_files={stage: str(file)})
  File "/home/louis/miniconda3/envs/w2vlf/lib/python3.8/site-packages/datasets/load.py", line 1599, in load_dataset
    builder_instance = load_dataset_builder(
  File "/home/louis/miniconda3/envs/w2vlf/lib/python3.8/site-packages/datasets/load.py", line 1457, in load_dataset_builder
    builder_instance: DatasetBuilder = builder_cls(
  File "/home/louis/miniconda3/envs/w2vlf/lib/python3.8/site-packages/datasets/builder.py", line 285, in __init__
    with FileLock(lock_path):
  File "/home/louis/miniconda3/envs/w2vlf/lib/python3.8/site-packages/datasets/utils/filelock.py", line 323, in __enter__
    self.acquire()
  File "/home/louis/miniconda3/envs/w2vlf/lib/python3.8/site-packages/datasets/utils/filelock.py", line 272, in acquire
    self._acquire()
  File "/home/louis/miniconda3/envs/w2vlf/lib/python3.8/site-packages/datasets/utils/filelock.py", line 403, in _acquire
    fd = os.open(self._lock_file, open_mode)
OSError: [Errno 36] File name too long: '/home/louis/.cache/huggingface/datasets/_home_louis_.cache_huggingface_datasets_json_default-98e6813a547f72fa_0.0.0_c2d554c3377ea79c7664b93dc65d0803b45e3279000f993c7bfd18937fd7f426.lock'
```

My home directory is encrypted, therefore the maximum length is 143 ([source 1](https://github.com/ray-project/ray/issues/1463#issuecomment-425674521), [source 2](https://stackoverflow.com/a/6571568/2668831))

From what I've read I think the error is in reference to the file name (just the final part of the path) which is 145 chars long:

```py
>>> len("_home_louis_.cache_huggingface_datasets_json_default-98e6813a547f72fa_0.0.0_c2d554c3377ea79c7664b93dc65d0803b45e3279000f993c7bfd18937fd7f426.lock")
145
```

I also have a file in this directory (i.e. whose length is not a problem):

```py
>>> len("_home_louis_.cache_huggingface_datasets_librispeech_asr_clean_2.1.0_468ec03677f46a8714ac6b5b64dba02d246a228d92cbbad7f3dc190fa039eab1.lock")
137
```
2021-10-26T20:31:32Z
https://github.com/huggingface/datasets/issues/2924#issuecomment-952300743
https://api.github.com/repos/huggingface/datasets/issues/2924
IC_kwDODunzps44wvTH
https://api.github.com/repos/huggingface/datasets/issues/comments/952300743/reactions
2021-10-26T20:31:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/952300743
https://avatars.githubusercontent.com/u/2979452?v=4
https://api.github.com/users/lmmx/events{/privacy}
https://api.github.com/users/lmmx/followers
https://api.github.com/users/lmmx/following{/other_user}
https://api.github.com/users/lmmx/gists{/gist_id}
https://github.com/lmmx
lmmx
MDQ6VXNlcjI5Nzk0NTI=
https://api.github.com/users/lmmx/orgs
https://api.github.com/users/lmmx/received_events
https://api.github.com/users/lmmx/repos
https://api.github.com/users/lmmx/starred{/owner}{/repo}
https://api.github.com/users/lmmx/subscriptions
User
https://api.github.com/users/lmmx
public
NONE
Perhaps this could be exposed as a config setting so you could change it manually?

https://github.com/huggingface/datasets/blob/5d1a9f1e3c6c495dc0610b459e39d2eb8893f152/src/datasets/utils/filelock.py#L135-L135

Rather than hard-code 255, default it to 255, and allow it to be changed, the same way is done for `datasets.config.IN_MEMORY_MAX_SIZE`:

https://github.com/huggingface/datasets/blob/12b7e13bc568b9f92705f64b249e148f3bc9a9ea/src/datasets/config.py#L171-L173

In fact there already appears to be an existing variable to do so:

https://github.com/huggingface/datasets/blob/12b7e13bc568b9f92705f64b249e148f3bc9a9ea/src/datasets/config.py#L187

It's used here:

https://github.com/huggingface/datasets/blob/efe89edd36e4ffa562fc3eebaf07a5fec26e6dac/src/datasets/builder.py#L163-L165

Perhaps it could be set based on a test (trying to create a 255 char length named lock file and seeing if it fails)
2021-10-26T21:12:30Z
https://github.com/huggingface/datasets/issues/2924#issuecomment-952330956
https://api.github.com/repos/huggingface/datasets/issues/2924
IC_kwDODunzps44w2rM
https://api.github.com/repos/huggingface/datasets/issues/comments/952330956/reactions
2021-10-26T21:16:39Z
https://api.github.com/repos/huggingface/datasets/issues/comments/952330956
https://avatars.githubusercontent.com/u/2979452?v=4
https://api.github.com/users/lmmx/events{/privacy}
https://api.github.com/users/lmmx/followers
https://api.github.com/users/lmmx/following{/other_user}
https://api.github.com/users/lmmx/gists{/gist_id}
https://github.com/lmmx
lmmx
MDQ6VXNlcjI5Nzk0NTI=
https://api.github.com/users/lmmx/orgs
https://api.github.com/users/lmmx/received_events
https://api.github.com/users/lmmx/repos
https://api.github.com/users/lmmx/starred{/owner}{/repo}
https://api.github.com/users/lmmx/subscriptions
User
https://api.github.com/users/lmmx
public
NONE
Just fixed it, sending a PR :smile:
2021-10-26T21:35:51Z
https://github.com/huggingface/datasets/issues/2924#issuecomment-952346686
https://api.github.com/repos/huggingface/datasets/issues/2924
IC_kwDODunzps44w6g-
https://api.github.com/repos/huggingface/datasets/issues/comments/952346686/reactions
2021-10-26T21:35:51Z
https://api.github.com/repos/huggingface/datasets/issues/comments/952346686
https://avatars.githubusercontent.com/u/2979452?v=4
https://api.github.com/users/lmmx/events{/privacy}
https://api.github.com/users/lmmx/followers
https://api.github.com/users/lmmx/following{/other_user}
https://api.github.com/users/lmmx/gists{/gist_id}
https://github.com/lmmx
lmmx
MDQ6VXNlcjI5Nzk0NTI=
https://api.github.com/users/lmmx/orgs
https://api.github.com/users/lmmx/received_events
https://api.github.com/users/lmmx/repos
https://api.github.com/users/lmmx/starred{/owner}{/repo}
https://api.github.com/users/lmmx/subscriptions
User
https://api.github.com/users/lmmx
public
COLLABORATOR
Hi @lmmx @gar1t ,

it would be helpful if you could run the following code and copy-paste the output here:
```python
import datasets
import os
os.statvfs(datasets.config.HF_DATASETS_CACHE)
```
2021-10-27T15:31:32Z
https://github.com/huggingface/datasets/issues/2924#issuecomment-953046649
https://api.github.com/repos/huggingface/datasets/issues/2924
IC_kwDODunzps44zlZ5
https://api.github.com/repos/huggingface/datasets/issues/comments/953046649/reactions
2021-10-27T15:31:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/953046649
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
`os.statvfs_result(f_bsize=4096, f_frsize=4096, f_blocks=240046344, f_bfree=96427610, f_bavail=84216487, f_files=61038592, f_ffree=58216027, f_favail=58216027, f_flag=4102, f_namemax=143)`
2021-10-28T09:16:03Z
https://github.com/huggingface/datasets/issues/2924#issuecomment-953663250
https://api.github.com/repos/huggingface/datasets/issues/2924
IC_kwDODunzps44178S
https://api.github.com/repos/huggingface/datasets/issues/comments/953663250/reactions
2021-10-28T09:16:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/953663250
https://avatars.githubusercontent.com/u/2979452?v=4
https://api.github.com/users/lmmx/events{/privacy}
https://api.github.com/users/lmmx/followers
https://api.github.com/users/lmmx/following{/other_user}
https://api.github.com/users/lmmx/gists{/gist_id}
https://github.com/lmmx
lmmx
MDQ6VXNlcjI5Nzk0NTI=
https://api.github.com/users/lmmx/orgs
https://api.github.com/users/lmmx/received_events
https://api.github.com/users/lmmx/repos
https://api.github.com/users/lmmx/starred{/owner}{/repo}
https://api.github.com/users/lmmx/subscriptions
User
https://api.github.com/users/lmmx
public
COLLABORATOR
Hi @lmmx,

Thanks for providing the result of the command. I've opened a PR, and it would be great if you could verify that the fix works on your system. To install the version of the datasets with the fix, please run the following command:
```
pip install git+https://github.com/huggingface/datasets.git@fix-2924
```

Btw, I saw your PR, and I appreciate your effort. However, my approach is a bit simpler for the end-user, so that's why I decided to fix the issue myself.
2021-10-28T12:05:39Z
https://github.com/huggingface/datasets/issues/2924#issuecomment-953780768
https://api.github.com/repos/huggingface/datasets/issues/2924
IC_kwDODunzps442Yog
https://api.github.com/repos/huggingface/datasets/issues/comments/953780768/reactions
2021-10-28T12:05:39Z
https://api.github.com/repos/huggingface/datasets/issues/comments/953780768
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
No problem Mario I didn't know that was where that value was recorded so I learnt something :smiley: I just wanted to get a local version working, of course you should implement whatever fix is best for HF. Yes can confirm this fixes it too. Thanks!
2021-10-28T16:13:14Z
https://github.com/huggingface/datasets/issues/2924#issuecomment-953995421
https://api.github.com/repos/huggingface/datasets/issues/2924
IC_kwDODunzps443NCd
https://api.github.com/repos/huggingface/datasets/issues/comments/953995421/reactions
2021-10-28T16:13:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/953995421
https://avatars.githubusercontent.com/u/2979452?v=4
https://api.github.com/users/lmmx/events{/privacy}
https://api.github.com/users/lmmx/followers
https://api.github.com/users/lmmx/following{/other_user}
https://api.github.com/users/lmmx/gists{/gist_id}
https://github.com/lmmx
lmmx
MDQ6VXNlcjI5Nzk0NTI=
https://api.github.com/users/lmmx/orgs
https://api.github.com/users/lmmx/received_events
https://api.github.com/users/lmmx/repos
https://api.github.com/users/lmmx/starred{/owner}{/repo}
https://api.github.com/users/lmmx/subscriptions
User
https://api.github.com/users/lmmx
public
NONE
Hello @mariosasko 

Has this fix shown up in the 2.10.1 version of huggingface datasets?
2023-03-28T06:50:18Z
https://github.com/huggingface/datasets/issues/2924#issuecomment-1486309299
https://api.github.com/repos/huggingface/datasets/issues/2924
IC_kwDODunzps5Yl0ez
https://api.github.com/repos/huggingface/datasets/issues/comments/1486309299/reactions
2023-03-28T06:50:18Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1486309299
https://avatars.githubusercontent.com/u/33834636?v=4
https://api.github.com/users/shashwat1002/events{/privacy}
https://api.github.com/users/shashwat1002/followers
https://api.github.com/users/shashwat1002/following{/other_user}
https://api.github.com/users/shashwat1002/gists{/gist_id}
https://github.com/shashwat1002
shashwat1002
MDQ6VXNlcjMzODM0NjM2
https://api.github.com/users/shashwat1002/orgs
https://api.github.com/users/shashwat1002/received_events
https://api.github.com/users/shashwat1002/repos
https://api.github.com/users/shashwat1002/starred{/owner}{/repo}
https://api.github.com/users/shashwat1002/subscriptions
User
https://api.github.com/users/shashwat1002
public
NONE
It looks like the issue is back with #6445 since the `MAX_FILENAME_LENGTH` is explicitly set to be 255 there
https://github.com/huggingface/datasets/blob/dd9044cdaabc1f9abce02c1b71bdb48fd3525d4e/src/datasets/utils/_filelock.py#L28
@mariosasko Could you please take a look
2023-12-08T11:05:02Z
https://github.com/huggingface/datasets/issues/2924#issuecomment-1846971196
https://api.github.com/repos/huggingface/datasets/issues/2924
IC_kwDODunzps5uFos8
https://api.github.com/repos/huggingface/datasets/issues/comments/1846971196/reactions
2023-12-08T11:12:28Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1846971196
https://avatars.githubusercontent.com/u/4479137?v=4
https://api.github.com/users/grapefroot/events{/privacy}
https://api.github.com/users/grapefroot/followers
https://api.github.com/users/grapefroot/following{/other_user}
https://api.github.com/users/grapefroot/gists{/gist_id}
https://github.com/grapefroot
grapefroot
MDQ6VXNlcjQ0NzkxMzc=
https://api.github.com/users/grapefroot/orgs
https://api.github.com/users/grapefroot/received_events
https://api.github.com/users/grapefroot/repos
https://api.github.com/users/grapefroot/starred{/owner}{/repo}
https://api.github.com/users/grapefroot/subscriptions
User
https://api.github.com/users/grapefroot
public
MEMBER
Ah good catch, I opened https://github.com/huggingface/datasets/pull/6482 with a quick fix
2023-12-08T13:39:50Z
https://github.com/huggingface/datasets/issues/2924#issuecomment-1847180750
https://api.github.com/repos/huggingface/datasets/issues/2924
IC_kwDODunzps5uGb3O
https://api.github.com/repos/huggingface/datasets/issues/comments/1847180750/reactions
2023-12-08T13:39:50Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1847180750
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
## Describe the bug

Getting the following error when calling `load_dataset("gar1t/test")`:

```
OSError: [Errno 36] File name too long: '<user>/.cache/huggingface/datasets/_home_garrett_.cache_huggingface_datasets_csv_test-7c856aea083a7043_0.0.0_9144e0a4e8435090117cea53e6c7537173ef2304525df4a077c435d8ee7828ff.incomplete.lock'
```

## Steps to reproduce the bug

Where the user cache dir (e.g. `~/.cache`) is on a file system that limits filenames to 255 chars (e.g. ext4):

```python
from datasets import load_dataset
load_dataset("gar1t/test")
```

## Expected results

Expect the function to return without an error.

## Actual results

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<python_venv>/lib/python3.9/site-packages/datasets/load.py", line 1112, in load_dataset
    builder_instance.download_and_prepare(
  File "<python_venv>/lib/python3.9/site-packages/datasets/builder.py", line 644, in download_and_prepare
    self._save_info()
  File "<python_venv>/lib/python3.9/site-packages/datasets/builder.py", line 765, in _save_info
    with FileLock(lock_path):
  File "<python_venv>/lib/python3.9/site-packages/datasets/utils/filelock.py", line 323, in __enter__
    self.acquire()
  File "<python_venv>/lib/python3.9/site-packages/datasets/utils/filelock.py", line 272, in acquire
    self._acquire()
  File "<python_venv>/lib/python3.9/site-packages/datasets/utils/filelock.py", line 403, in _acquire
    fd = os.open(self._lock_file, open_mode)
OSError: [Errno 36] File name too long: '<user>/.cache/huggingface/datasets/_home_garrett_.cache_huggingface_datasets_csv_test-7c856aea083a7043_0.0.0_9144e0a4e8435090117cea53e6c7537173ef2304525df4a077c435d8ee7828ff.incomplete.lock'
```

## Environment info

- `datasets` version: 1.12.1
- Platform: Linux-5.11.0-27-generic-x86_64-with-glibc2.31
- Python version: 3.9.7
- PyArrow version: 5.0.0
