Skip hidden files/directories in data files resolution and `iter_files`
NONE
_The documentation is not available anymore as the PR was closed or merged._
2022-05-26T12:14:58Z
https://github.com/huggingface/datasets/pull/4412#issuecomment-1138506438
https://api.github.com/repos/huggingface/datasets/issues/4412
IC_kwDODunzps5D3DrG
https://api.github.com/repos/huggingface/datasets/issues/comments/1138506438/reactions
2022-06-01T13:12:55Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1138506438
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
CONTRIBUTOR
This PR (via new release) broke many transformers tests.

I will try to post a summary shortly.

cc: @ydshieh 
2022-06-15T16:49:32Z
https://github.com/huggingface/datasets/pull/4412#issuecomment-1156707495
https://api.github.com/repos/huggingface/datasets/issues/4412
IC_kwDODunzps5E8fSn
https://api.github.com/repos/huggingface/datasets/issues/comments/1156707495/reactions
2022-06-15T16:49:53Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1156707495
https://avatars.githubusercontent.com/u/10676103?v=4
https://api.github.com/users/stas00/events{/privacy}
https://api.github.com/users/stas00/followers
https://api.github.com/users/stas00/following{/other_user}
https://api.github.com/users/stas00/gists{/gist_id}
https://github.com/stas00
stas00
MDQ6VXNlcjEwNjc2MTAz
https://api.github.com/users/stas00/orgs
https://api.github.com/users/stas00/received_events
https://api.github.com/users/stas00/repos
https://api.github.com/users/stas00/starred{/owner}{/repo}
https://api.github.com/users/stas00/subscriptions
User
https://api.github.com/users/stas00
public
CONTRIBUTOR
So now it can't handle a local path via: `--train_file tests/deepspeed/../fixtures/tests_samples/wmt_en_ro/train.json` even though it's there. it works just fine if I change the path to not have `..`

You can reproduce the original problem with:

```
$ cd transformers 
$ python examples/pytorch/translation/run_translation.py --model_name_or_path t5-small --train_file tests/fixtures/tests_samples/wmt_en_ro/train.json --validation_file tests/deepspeed/../fixtures/tests_samples/wmt_en_ro/val.json --output_dir /tmp/tmp5o5to4k0 --overwrite_output_dir --max_source_length 32 --max_target_length 32 --val_max_target_length 32 --warmup_steps 8 --predict_with_generate --save_steps 0 --eval_steps 1 --group_by_length --label_smoothing_factor 0.1 --source_lang en --target_lang ro --report_to none --source_prefix "translate English to Romanian: " --fp16 --do_train --num_train_epochs 1 --max_train_samples 16 --per_device_train_batch_size 2 --learning_rate 3e-3
[...]
Traceback (most recent call last):
  File "examples/pytorch/translation/run_translation.py", line 656, in <module>
    main()
  File "examples/pytorch/translation/run_translation.py", line 346, in main
    raw_datasets = load_dataset(
  File "/home/stas/anaconda3/envs/py38-pt111/lib/python3.8/site-packages/datasets/load.py", line 1656, in load_dataset
    builder_instance = load_dataset_builder(
  File "/home/stas/anaconda3/envs/py38-pt111/lib/python3.8/site-packages/datasets/load.py", line 1439, in load_dataset_builder
    dataset_module = dataset_module_factory(
  File "/home/stas/anaconda3/envs/py38-pt111/lib/python3.8/site-packages/datasets/load.py", line 1097, in dataset_module_factory
    return PackagedDatasetModuleFactory(
  File "/home/stas/anaconda3/envs/py38-pt111/lib/python3.8/site-packages/datasets/load.py", line 743, in get_module
    data_files = DataFilesDict.from_local_or_remote(
  File "/home/stas/anaconda3/envs/py38-pt111/lib/python3.8/site-packages/datasets/data_files.py", line 588, in from_local_or_remote
    DataFilesList.from_local_or_remote(
  File "/home/stas/anaconda3/envs/py38-pt111/lib/python3.8/site-packages/datasets/data_files.py", line 556, in from_local_or_remote
    data_files = resolve_patterns_locally_or_by_urls(base_path, patterns, allowed_extensions)
  File "/home/stas/anaconda3/envs/py38-pt111/lib/python3.8/site-packages/datasets/data_files.py", line 194, in resolve_patterns_locally_or_by_urls
    for path in _resolve_single_pattern_locally(base_path, pattern, allowed_extensions):
  File "/home/stas/anaconda3/envs/py38-pt111/lib/python3.8/site-packages/datasets/data_files.py", line 144, in _resolve_single_pattern_locally
    raise FileNotFoundError(error_msg)
FileNotFoundError: Unable to find '/mnt/nvme0/code/huggingface/transformers-master/tests/deepspeed/../fixtures/tests_samples/wmt_en_ro/val.json' at /mnt/nvme0/code/huggingface/transformers-master
```
2022-06-15T16:56:32Z
https://github.com/huggingface/datasets/pull/4412#issuecomment-1156714121
https://api.github.com/repos/huggingface/datasets/issues/4412
IC_kwDODunzps5E8g6J
https://api.github.com/repos/huggingface/datasets/issues/comments/1156714121/reactions
2022-06-15T16:57:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1156714121
https://avatars.githubusercontent.com/u/10676103?v=4
https://api.github.com/users/stas00/events{/privacy}
https://api.github.com/users/stas00/followers
https://api.github.com/users/stas00/following{/other_user}
https://api.github.com/users/stas00/gists{/gist_id}
https://github.com/stas00
stas00
MDQ6VXNlcjEwNjc2MTAz
https://api.github.com/users/stas00/orgs
https://api.github.com/users/stas00/received_events
https://api.github.com/users/stas00/repos
https://api.github.com/users/stas00/starred{/owner}{/repo}
https://api.github.com/users/stas00/subscriptions
User
https://api.github.com/users/stas00
public
CONTRIBUTOR
will apply a workaround to `transformers` tests here https://github.com/huggingface/transformers/pull/17721

2022-06-15T17:05:18Z
https://github.com/huggingface/datasets/pull/4412#issuecomment-1156722313
https://api.github.com/repos/huggingface/datasets/issues/4412
IC_kwDODunzps5E8i6J
https://api.github.com/repos/huggingface/datasets/issues/comments/1156722313/reactions
2022-06-15T17:05:18Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1156722313
https://avatars.githubusercontent.com/u/10676103?v=4
https://api.github.com/users/stas00/events{/privacy}
https://api.github.com/users/stas00/followers
https://api.github.com/users/stas00/following{/other_user}
https://api.github.com/users/stas00/gists{/gist_id}
https://github.com/stas00
stas00
MDQ6VXNlcjEwNjc2MTAz
https://api.github.com/users/stas00/orgs
https://api.github.com/users/stas00/received_events
https://api.github.com/users/stas00/repos
https://api.github.com/users/stas00/starred{/owner}{/repo}
https://api.github.com/users/stas00/subscriptions
User
https://api.github.com/users/stas00
public
MEMBER
This has been fixed with https://github.com/huggingface/datasets/pull/4505, will do a patch release tomorrow for `datasets` ;)
2022-06-15T17:06:46Z
https://github.com/huggingface/datasets/pull/4412#issuecomment-1156724006
https://api.github.com/repos/huggingface/datasets/issues/4412
IC_kwDODunzps5E8jUm
https://api.github.com/repos/huggingface/datasets/issues/comments/1156724006/reactions
2022-06-15T17:06:46Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1156724006
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Thank you for the quick fix, @lhoestq 
2022-06-15T17:11:24Z
https://github.com/huggingface/datasets/pull/4412#issuecomment-1156728307
https://api.github.com/repos/huggingface/datasets/issues/4412
IC_kwDODunzps5E8kXz
https://api.github.com/repos/huggingface/datasets/issues/comments/1156728307/reactions
2022-06-15T17:11:24Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1156728307
https://avatars.githubusercontent.com/u/10676103?v=4
https://api.github.com/users/stas00/events{/privacy}
https://api.github.com/users/stas00/followers
https://api.github.com/users/stas00/following{/other_user}
https://api.github.com/users/stas00/gists{/gist_id}
https://github.com/stas00
stas00
MDQ6VXNlcjEwNjc2MTAz
https://api.github.com/users/stas00/orgs
https://api.github.com/users/stas00/received_events
https://api.github.com/users/stas00/repos
https://api.github.com/users/stas00/starred{/owner}{/repo}
https://api.github.com/users/stas00/subscriptions
User
https://api.github.com/users/stas00
public
Fix #4115 