FileNotFoundError when trying to load the downloaded dataset with `load_dataset(..., streaming=True)`
COLLABORATOR
You can clone the `togethercomputer/RedPajama-Data-1T-Sample` repo and load the dataset with `load_dataset("path/to/cloned_repo")` to use it offline.
2023-10-23T18:35:36Z
https://github.com/huggingface/datasets/issues/6327#issuecomment-1775787689
https://api.github.com/repos/huggingface/datasets/issues/6327
IC_kwDODunzps5p2F6p
https://api.github.com/repos/huggingface/datasets/issues/comments/1775787689/reactions
2023-10-23T18:48:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1775787689
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
CONTRIBUTOR
@mariosasko Thank you for your kind reply! I'll try it as a workaround.
Does that mean that currently it's not supported to simply load with a short name?
2023-10-23T18:38:44Z
https://github.com/huggingface/datasets/issues/6327#issuecomment-1775793463
https://api.github.com/repos/huggingface/datasets/issues/6327
IC_kwDODunzps5p2HU3
https://api.github.com/repos/huggingface/datasets/issues/comments/1775793463/reactions
2023-10-23T18:38:44Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1775793463
https://avatars.githubusercontent.com/u/18402347?v=4
https://api.github.com/users/yzhangcs/events{/privacy}
https://api.github.com/users/yzhangcs/followers
https://api.github.com/users/yzhangcs/following{/other_user}
https://api.github.com/users/yzhangcs/gists{/gist_id}
https://github.com/yzhangcs
yzhangcs
MDQ6VXNlcjE4NDAyMzQ3
https://api.github.com/users/yzhangcs/orgs
https://api.github.com/users/yzhangcs/received_events
https://api.github.com/users/yzhangcs/repos
https://api.github.com/users/yzhangcs/starred{/owner}{/repo}
https://api.github.com/users/yzhangcs/subscriptions
User
https://api.github.com/users/yzhangcs
public
COLLABORATOR
It is, but manually downloading repo files to the cache can easily lead to failure (the HF cache is not meant to be modified by a user besides deleting the files 🙂), as in your case. Hence, the clone + `load_dataset("path/to/cloned_repo")` workflow should be used instead.
2023-10-23T18:48:59Z
https://github.com/huggingface/datasets/issues/6327#issuecomment-1775816629
https://api.github.com/repos/huggingface/datasets/issues/6327
IC_kwDODunzps5p2M-1
https://api.github.com/repos/huggingface/datasets/issues/comments/1775816629/reactions
2023-10-23T18:49:21Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1775816629
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
### Describe the bug

Hi, I'm trying to load the dataset `togethercomputer/RedPajama-Data-1T-Sample` with `load_dataset` in streaming mode, i.e., `streaming=True`, but `FileNotFoundError` occurs.

### Steps to reproduce the bug

I've downloaded the dataset and save it to the cache dir in advance. My hope is loading the files in offline environment and without taking too much hours to prepross the entire data before running into the training process.
So I try the following code to load the files streamingly
```py
dataset = load_dataset('togethercomputer/RedPajama-Data-1T-Sample', streaming=True)
print(next(iter(dataset['train'])))
```
Sadly, it raises the following:
```
FileNotFoundError: [Errno 2] No such file or directory: 'CURRENT_CODE_PATH/arxiv_sample.jsonl'
```
I've noticed that the dataset can be properly found in the begining
```
Using the latest cached version of the module from /root/.cache/huggingface/modules/datasets_modules/datasets/togethercomputer--RedPajama-Data-1T-Sample/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 (last modified on Sat Oct 21 20:12:57 2023) since it couldn't be found locally at togethercomputer/RedPajama-Data-1T-Sample., or remotely on the Hugging Face Hub.
```
But it seems that the paths couldn't be properly parsed when loading iteratively.
 
How should I fix this error. I've tried specifying `data_files` or `data_dir` as `.../arxiv_sample.jsonl` but none of them works.

Thanks.

### Expected behavior

Properly load the dataset.

### Environment info

`datasets==2.14.5`