version2.3.2   load_dataset()data_files can't include .xxxx in path
COLLABORATOR
Version 2.3.2 is over one year old, so please use the latest release (2.14.0) to get the expected behavior. Version 2.3.2 does not contain some fixes we made to fix resolving hidden files/directories (starting with a dot).
2023-07-26T12:34:44Z
https://github.com/huggingface/datasets/issues/6073#issuecomment-1651719245
https://api.github.com/repos/huggingface/datasets/issues/6073
IC_kwDODunzps5iczxN
https://api.github.com/repos/huggingface/datasets/issues/comments/1651719245/reactions
2023-07-26T12:34:44Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1651719245
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
### Describe the bug

First, I cd workdir.
Then, I just use load_dataset("json", data_file={"train":"/a/b/c/.d/train/train.json", "test":"/a/b/c/.d/train/test.json"})
that couldn't work and 
<FileNotFoundError: Unable to find 
'/a/b/c/.d/train/train.jsonl' at 
/a/b/c/.d/>

And I debug, it is fine in version2.1.2

So there maybe a bug in path join.

Here is the whole bug report:
/x/datasets/loa │
│ d.py:1656 in load_dataset                                                    │
│                                                                              │
│   1653 │   ignore_verifications = ignore_verifications or save_infos         │
│   1654 │                                                                     │
│   1655 │   # Create a dataset builder                                        │
│ ❱ 1656 │   builder_instance = load_dataset_builder(                          │
│   1657 │   │   path=path,                                                    │
│   1658 │   │   name=name,                                                    │
│   1659 │   │   data_dir=data_dir,                                            │
│                                                                              │
│ x/datasets/loa │
│ d.py:1439 in load_dataset_builder                                            │
│                                                                              │
│   1436 │   if use_auth_token is not None:                                    │
│   1437 │   │   download_config = download_config.copy() if download_config e │
│   1438 │   │   download_config.use_auth_token = use_auth_token               │
│ ❱ 1439 │   dataset_module = dataset_module_factory(                          │
│   1440 │   │   path,                                                         │
│   1441 │   │   revision=revision,                                            │
│   1442 │   │   download_config=download_config,                              │
│                                                                              │
│ x/datasets/loa │
│ d.py:1097 in dataset_module_factory                                          │
│                                                                              │
│   1094 │                                                                     │
│   1095 │   # Try packaged                                                    │
│   1096 │   if path in _PACKAGED_DATASETS_MODULES:                            │
│ ❱ 1097 │   │   return PackagedDatasetModuleFactory(                          │
│   1098 │   │   │   path,                                                     │
│   1099 │   │   │   data_dir=data_dir,                                        │
│   1100 │   │   │   data_files=data_files,                                    │
│                                                                              │
│x/datasets/loa │
│ d.py:743 in get_module                                                       │
│                                                                              │
│    740 │   │   │   if self.data_dir is not None                              │
│    741 │   │   │   else get_patterns_locally(str(Path().resolve()))          │
│    742 │   │   )                                                             │
│ ❱  743 │   │   data_files = DataFilesDict.from_local_or_remote(              │
│    744 │   │   │   patterns,                                                 │
│    745 │   │   │   use_auth_token=self.download_config.use_auth_token,       │
│    746 │   │   │   base_path=str(Path(self.data_dir).resolve()) if self.data │
│                                                                              │
│ x/datasets/dat │
│ a_files.py:590 in from_local_or_remote                                       │
│                                                                              │
│   587 │   │   out = cls()                                                    │
│   588 │   │   for key, patterns_for_key in patterns.items():                 │
│   589 │   │   │   out[key] = (                                               │
│ ❱ 590 │   │   │   │   DataFilesList.from_local_or_remote(                    │
│   591 │   │   │   │   │   patterns_for_key,                                  │
│   592 │   │   │   │   │   base_path=base_path,                               │
│   593 │   │   │   │   │   allowed_extensions=allowed_extensions,             │
│                                                                              │
│ /x/datasets/dat │
│ a_files.py:558 in from_local_or_remote                                       │
│                                                                              │
│   555 │   │   use_auth_token: Optional[Union[bool, str]] = None,             │
│   556 │   ) -> "DataFilesList":                                              │
│   557 │   │   base_path = base_path if base_path is not None else str(Path() │
│ ❱ 558 │   │   data_files = resolve_patterns_locally_or_by_urls(base_path, pa │
│   559 │   │   origin_metadata = _get_origin_metadata_locally_or_by_urls(data │
│   560 │   │   return cls(data_files, origin_metadata)                        │
│   561                                                                        │
│                                                                              │
│ /x/datasets/dat │
│ a_files.py:195 in resolve_patterns_locally_or_by_urls                        │
│                                                                              │
│   192 │   │   if is_remote_url(pattern):                                     │
│   193 │   │   │   data_files.append(Url(pattern))                            │
│   194 │   │   else:                                                          │
│ ❱ 195 │   │   │   for path in _resolve_single_pattern_locally(base_path, pat │
│   196 │   │   │   │   data_files.append(path)                                │
│   197 │                                                                      │
│   198 │   if not data_files:                                                 │
│                                                                              │
│ /x/datasets/dat │
│ a_files.py:145 in _resolve_single_pattern_locally                            │
│                                                                              │
│   142 │   │   error_msg = f"Unable to find '{pattern}' at {Path(base_path).r │
│   143 │   │   if allowed_extensions is not None:                             │
│   144 │   │   │   error_msg += f" with any supported extension {list(allowed │
│ ❱ 145 │   │   raise FileNotFoundError(error_msg)                             │
│   146 │   return sorted(out)                                                 │
│   147                         

### Steps to reproduce the bug

1. Version=2.3.2
2. In shell, cd workdir.(cd /a/b/c/.d/)
3. load_dataset("json", data_file={"train":"/a/b/c/.d/train/train.json", "test":"/a/b/c/.d/train/test.json"})

### Expected behavior

fix it please~

### Environment info

2.3.2