Unable to load wiki_auto_asset_turk from GEM
COLLABORATOR
Hi! I've opened a [PR](https://huggingface.co/datasets/GEM/wiki_auto_asset_turk/discussions/5) with a fix. While waiting for it to be merged, you can load the dataset from the PR branch with `datasets.load_dataset("GEM/wiki_auto_asset_turk", revision="refs/pr/5")`
2024-04-26T16:12:29Z
https://github.com/huggingface/datasets/issues/6841#issuecomment-2079690880
https://api.github.com/repos/huggingface/datasets/issues/6841
IC_kwDODunzps579ZCA
https://api.github.com/repos/huggingface/datasets/issues/comments/2079690880/reactions
2024-04-26T16:12:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2079690880
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
Thanks Mario. Still getting the same issue though with the suggested fix

#cat gem_sari.py
import datasets
print (datasets.__version__)
dataset =datasets.load_dataset("GEM/wiki_auto_asset_turk", revision="refs/pr/5")

End up with 
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/datasets/load.py", line 2582, in load_dataset
    builder_instance.download_and_prepare(
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/datasets/builder.py", line 1005, in download_and_prepare
    self._download_and_prepare(
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/datasets/builder.py", line 1767, in _download_and_prepare
    super()._download_and_prepare(
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/datasets/builder.py", line 1100, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/datasets/builder.py", line 1565, in _prepare_split
    split_info = self.info.splits[split_generator.name]
                 ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/datasets/splits.py", line 532, in __getitem__
    instructions = make_file_instructions(
                   ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/datasets/arrow_reader.py", line 121, in make_file_instructions
    info.name: filenames_for_dataset_split(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/datasets/naming.py", line 72, in filenames_for_dataset_split
    prefix = os.path.join(path, prefix)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen posixpath>", line 76, in join
TypeError: expected str, bytes or os.PathLike object, not NoneType
2024-04-26T16:34:06Z
https://github.com/huggingface/datasets/issues/6841#issuecomment-2079721587
https://api.github.com/repos/huggingface/datasets/issues/6841
IC_kwDODunzps579ghz
https://api.github.com/repos/huggingface/datasets/issues/comments/2079721587/reactions
2024-04-26T16:34:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2079721587
https://avatars.githubusercontent.com/u/23074600?v=4
https://api.github.com/users/abhinavsethy/events{/privacy}
https://api.github.com/users/abhinavsethy/followers
https://api.github.com/users/abhinavsethy/following{/other_user}
https://api.github.com/users/abhinavsethy/gists{/gist_id}
https://github.com/abhinavsethy
abhinavsethy
MDQ6VXNlcjIzMDc0NjAw
https://api.github.com/users/abhinavsethy/orgs
https://api.github.com/users/abhinavsethy/received_events
https://api.github.com/users/abhinavsethy/repos
https://api.github.com/users/abhinavsethy/starred{/owner}{/repo}
https://api.github.com/users/abhinavsethy/subscriptions
User
https://api.github.com/users/abhinavsethy
public
COLLABORATOR
Hmm, that's weird. Maybe try deleting the cache with `!rm -rf ~/.cache/huggingface/datasets` and then re-download.
2024-04-26T16:56:08Z
https://github.com/huggingface/datasets/issues/6841#issuecomment-2079753036
https://api.github.com/repos/huggingface/datasets/issues/6841
IC_kwDODunzps579oNM
https://api.github.com/repos/huggingface/datasets/issues/comments/2079753036/reactions
2024-04-26T16:56:08Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2079753036
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
Tried that a couple of time. It does download the data fresh but end up with same error. Is there a way to see if its using the right version ?
2024-04-26T17:04:07Z
https://github.com/huggingface/datasets/issues/6841#issuecomment-2079764087
https://api.github.com/repos/huggingface/datasets/issues/6841
IC_kwDODunzps579q53
https://api.github.com/repos/huggingface/datasets/issues/comments/2079764087/reactions
2024-04-26T17:04:07Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2079764087
https://avatars.githubusercontent.com/u/23074600?v=4
https://api.github.com/users/abhinavsethy/events{/privacy}
https://api.github.com/users/abhinavsethy/followers
https://api.github.com/users/abhinavsethy/following{/other_user}
https://api.github.com/users/abhinavsethy/gists{/gist_id}
https://github.com/abhinavsethy
abhinavsethy
MDQ6VXNlcjIzMDc0NjAw
https://api.github.com/users/abhinavsethy/orgs
https://api.github.com/users/abhinavsethy/received_events
https://api.github.com/users/abhinavsethy/repos
https://api.github.com/users/abhinavsethy/starred{/owner}{/repo}
https://api.github.com/users/abhinavsethy/subscriptions
User
https://api.github.com/users/abhinavsethy
public
COLLABORATOR
You can check the version with `python -c "import datasets; print(datasets.__version__)"`
2024-04-26T17:19:25Z
https://github.com/huggingface/datasets/issues/6841#issuecomment-2079785908
https://api.github.com/repos/huggingface/datasets/issues/6841
IC_kwDODunzps579wO0
https://api.github.com/repos/huggingface/datasets/issues/comments/2079785908/reactions
2024-04-26T17:19:25Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2079785908
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
the datasets version is 2.18. 

I wanted to see if the command datasets.load_dataset("GEM/wiki_auto_asset_turk", revision="refs/pr/5") is using the right revision (refs/pr/5).  





 
2024-04-26T17:22:57Z
https://github.com/huggingface/datasets/issues/6841#issuecomment-2079790457
https://api.github.com/repos/huggingface/datasets/issues/6841
IC_kwDODunzps579xV5
https://api.github.com/repos/huggingface/datasets/issues/comments/2079790457/reactions
2024-04-26T17:22:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2079790457
https://avatars.githubusercontent.com/u/23074600?v=4
https://api.github.com/users/abhinavsethy/events{/privacy}
https://api.github.com/users/abhinavsethy/followers
https://api.github.com/users/abhinavsethy/following{/other_user}
https://api.github.com/users/abhinavsethy/gists{/gist_id}
https://github.com/abhinavsethy
abhinavsethy
MDQ6VXNlcjIzMDc0NjAw
https://api.github.com/users/abhinavsethy/orgs
https://api.github.com/users/abhinavsethy/received_events
https://api.github.com/users/abhinavsethy/repos
https://api.github.com/users/abhinavsethy/starred{/owner}{/repo}
https://api.github.com/users/abhinavsethy/subscriptions
User
https://api.github.com/users/abhinavsethy
public
NONE
Still have this problem
2024-05-29T10:56:57Z
https://github.com/huggingface/datasets/issues/6841#issuecomment-2137130989
https://api.github.com/repos/huggingface/datasets/issues/6841
IC_kwDODunzps5_Ygft
https://api.github.com/repos/huggingface/datasets/issues/comments/2137130989/reactions
2024-05-29T10:56:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2137130989
https://avatars.githubusercontent.com/u/47926125?v=4
https://api.github.com/users/MeshchaninovViacheslav/events{/privacy}
https://api.github.com/users/MeshchaninovViacheslav/followers
https://api.github.com/users/MeshchaninovViacheslav/following{/other_user}
https://api.github.com/users/MeshchaninovViacheslav/gists{/gist_id}
https://github.com/MeshchaninovViacheslav
MeshchaninovViacheslav
MDQ6VXNlcjQ3OTI2MTI1
https://api.github.com/users/MeshchaninovViacheslav/orgs
https://api.github.com/users/MeshchaninovViacheslav/received_events
https://api.github.com/users/MeshchaninovViacheslav/repos
https://api.github.com/users/MeshchaninovViacheslav/starred{/owner}{/repo}
https://api.github.com/users/MeshchaninovViacheslav/subscriptions
User
https://api.github.com/users/MeshchaninovViacheslav
public
MEMBER
The issue is fixed once the fixing PR has been merged and the dataset has been converted to Parquet.

If the problem persists on your side, you should update your `datasets` library:
```shell
pip install -U datasets
```
And if you have already the latest version of `datasets`, then you need to delete the old version of this dataset in your cache:
```shell
rm -fr ~/.cache/huggingface/datasets/GEM___wiki_auto_asset_turk
rm -fr ~/.cache/huggingface/modules/datasets_modules/datasets/GEM--wiki_auto_asset_turk
```
2024-05-29T13:54:02Z
https://github.com/huggingface/datasets/issues/6841#issuecomment-2137476073
https://api.github.com/repos/huggingface/datasets/issues/6841
IC_kwDODunzps5_Z0vp
https://api.github.com/repos/huggingface/datasets/issues/comments/2137476073/reactions
2024-05-29T13:54:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2137476073
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
### Describe the bug

I am unable to load the wiki_auto_asset_turk dataset. I get a fatal error while trying to access wiki_auto_asset_turk and load it with datasets.load_dataset. The error (TypeError: expected str, bytes or os.PathLike object, not NoneType) is from filenames_for_dataset_split in a os.path.join call 


>>import datasets
>>print (datasets.__version__)
>>dataset = datasets.load_dataset("GEM/wiki_auto_asset_turk")

System output:

Generating train split: 100%|█| 483801/483801 [00:03<00:00, 127164.26 examples/s
Generating validation split: 100%|█| 20000/20000 [00:00<00:00, 116052.94 example
Generating test_asset split: 100%|██| 359/359 [00:00<00:00, 76155.93 examples/s]
Generating test_turk split: 100%|███| 359/359 [00:00<00:00, 87691.76 examples/s]
Traceback (most recent call last):
  File "/Users/abhinav.sethy/Code/openai_evals/evals/evals/grammarly_tasks/gem_sari.py", line 3, in <module>
    dataset = datasets.load_dataset("GEM/wiki_auto_asset_turk")
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/datasets/load.py", line 2582, in load_dataset
    builder_instance.download_and_prepare(
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/datasets/builder.py", line 1005, in download_and_prepare
    self._download_and_prepare(
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/datasets/builder.py", line 1767, in _download_and_prepare
    super()._download_and_prepare(
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/datasets/builder.py", line 1100, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/datasets/builder.py", line 1565, in _prepare_split
    split_info = self.info.splits[split_generator.name]
                 ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/datasets/splits.py", line 532, in __getitem__
    instructions = make_file_instructions(
                   ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/datasets/arrow_reader.py", line 121, in make_file_instructions
    info.name: filenames_for_dataset_split(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/datasets/naming.py", line 72, in filenames_for_dataset_split
    prefix = os.path.join(path, prefix)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen posixpath>", line 76, in join
TypeError: expected str, bytes or os.PathLike object, not NoneType

### Steps to reproduce the bug

import datasets
print (datasets.__version__)
dataset = datasets.load_dataset("GEM/wiki_auto_asset_turk")


### Expected behavior

Should be able to load the dataset without any issues

### Environment info

datasets version 2.18.0 (was able to reproduce bug with older versions 2.16 and 2.14 also)
Python 3.12.0