Discussion using datasets in offline mode
NONE
No comments ?
2020-11-30T14:40:22Z
https://github.com/huggingface/datasets/issues/824#issuecomment-735825910
https://api.github.com/repos/huggingface/datasets/issues/824
MDEyOklzc3VlQ29tbWVudDczNTgyNTkxMA==
https://api.github.com/repos/huggingface/datasets/issues/comments/735825910/reactions
2020-11-30T14:40:22Z
https://api.github.com/repos/huggingface/datasets/issues/comments/735825910
https://avatars.githubusercontent.com/u/77193?v=4
https://api.github.com/users/mandubian/events{/privacy}
https://api.github.com/users/mandubian/followers
https://api.github.com/users/mandubian/following{/other_user}
https://api.github.com/users/mandubian/gists{/gist_id}
https://github.com/mandubian
mandubian
MDQ6VXNlcjc3MTkz
https://api.github.com/users/mandubian/orgs
https://api.github.com/users/mandubian/received_events
https://api.github.com/users/mandubian/repos
https://api.github.com/users/mandubian/starred{/owner}{/repo}
https://api.github.com/users/mandubian/subscriptions
User
https://api.github.com/users/mandubian
public
NONE
I think it would be very cool. I'm currently working on a cluster from Compute Canada, and I have internet access only when I'm not in the nodes where I run the scripts. So I was expecting to be able to use the wmt14 dataset until I realized I needed internet connection even if I downloaded the data already. I'm going to try option 2 you mention for now though! Thanks ;)
2020-12-15T21:46:48Z
https://github.com/huggingface/datasets/issues/824#issuecomment-745587777
https://api.github.com/repos/huggingface/datasets/issues/824
MDEyOklzc3VlQ29tbWVudDc0NTU4Nzc3Nw==
https://api.github.com/repos/huggingface/datasets/issues/comments/745587777/reactions
2020-12-15T21:46:48Z
https://api.github.com/repos/huggingface/datasets/issues/comments/745587777
https://avatars.githubusercontent.com/u/9610770?v=4
https://api.github.com/users/LuCeHe/events{/privacy}
https://api.github.com/users/LuCeHe/followers
https://api.github.com/users/LuCeHe/following{/other_user}
https://api.github.com/users/LuCeHe/gists{/gist_id}
https://github.com/LuCeHe
LuCeHe
MDQ6VXNlcjk2MTA3NzA=
https://api.github.com/users/LuCeHe/orgs
https://api.github.com/users/LuCeHe/received_events
https://api.github.com/users/LuCeHe/repos
https://api.github.com/users/LuCeHe/starred{/owner}{/repo}
https://api.github.com/users/LuCeHe/subscriptions
User
https://api.github.com/users/LuCeHe
public
NONE
Requiring online connection is a deal breaker in some cases unfortunately so it'd be great if offline mode is added similar to how `transformers` loads models offline fine.

@mandubian's second bullet point suggests that there's a workaround allowing you to use your offline (custom?) dataset with `datasets`. Could you please elaborate on how that should look like?
2020-12-27T11:56:26Z
https://github.com/huggingface/datasets/issues/824#issuecomment-751458844
https://api.github.com/repos/huggingface/datasets/issues/824
MDEyOklzc3VlQ29tbWVudDc1MTQ1ODg0NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/751458844/reactions
2020-12-27T11:56:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/751458844
https://avatars.githubusercontent.com/u/13801078?v=4
https://api.github.com/users/geniki/events{/privacy}
https://api.github.com/users/geniki/followers
https://api.github.com/users/geniki/following{/other_user}
https://api.github.com/users/geniki/gists{/gist_id}
https://github.com/geniki
geniki
MDQ6VXNlcjEzODAxMDc4
https://api.github.com/users/geniki/orgs
https://api.github.com/users/geniki/received_events
https://api.github.com/users/geniki/repos
https://api.github.com/users/geniki/starred{/owner}{/repo}
https://api.github.com/users/geniki/subscriptions
User
https://api.github.com/users/geniki
public
NONE
here is my way to load a dataset offline, but it **requires** an online machine
1. (online machine)
```
import datasets
data = datasets.load_dataset(...)
data.save_to_disk(/YOUR/DATASET/DIR)
```
2. copy the dir from online to the offline machine
3. (offline machine)
```
import datasets
data = datasets.load_from_disk(/SAVED/DATA/DIR)
```

HTH.
2021-01-12T02:46:09Z
https://github.com/huggingface/datasets/issues/824#issuecomment-758358089
https://api.github.com/repos/huggingface/datasets/issues/824
MDEyOklzc3VlQ29tbWVudDc1ODM1ODA4OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/758358089/reactions
2021-01-12T02:46:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/758358089
https://avatars.githubusercontent.com/u/8414897?v=4
https://api.github.com/users/ZizhenWang/events{/privacy}
https://api.github.com/users/ZizhenWang/followers
https://api.github.com/users/ZizhenWang/following{/other_user}
https://api.github.com/users/ZizhenWang/gists{/gist_id}
https://github.com/ZizhenWang
ZizhenWang
MDQ6VXNlcjg0MTQ4OTc=
https://api.github.com/users/ZizhenWang/orgs
https://api.github.com/users/ZizhenWang/received_events
https://api.github.com/users/ZizhenWang/repos
https://api.github.com/users/ZizhenWang/starred{/owner}{/repo}
https://api.github.com/users/ZizhenWang/subscriptions
User
https://api.github.com/users/ZizhenWang
public
NONE
> here is my way to load a dataset offline, but it **requires** an online machine
> 
> 1. (online machine)
> 
> ```
> 
> import datasets
> 
> data = datasets.load_dataset(...)
> 
> data.save_to_disk(/YOUR/DATASET/DIR)
> 
> ```
> 
> 2. copy the dir from online to the offline machine
> 
> 3. (offline machine)
> 
> ```
> 
> import datasets
> 
> data = datasets.load_from_disk(/SAVED/DATA/DIR)
> 
> ```
> 
> 
> 
> HTH.


2021-01-12T02:47:42Z
https://github.com/huggingface/datasets/issues/824#issuecomment-758358601
https://api.github.com/repos/huggingface/datasets/issues/824
MDEyOklzc3VlQ29tbWVudDc1ODM1ODYwMQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/758358601/reactions
2021-01-12T02:47:42Z
https://api.github.com/repos/huggingface/datasets/issues/comments/758358601
https://avatars.githubusercontent.com/u/77094423?v=4
https://api.github.com/users/aballigier/events{/privacy}
https://api.github.com/users/aballigier/followers
https://api.github.com/users/aballigier/following{/other_user}
https://api.github.com/users/aballigier/gists{/gist_id}
https://github.com/aballigier
aballigier
MDQ6VXNlcjc3MDk0NDIz
https://api.github.com/users/aballigier/orgs
https://api.github.com/users/aballigier/received_events
https://api.github.com/users/aballigier/repos
https://api.github.com/users/aballigier/starred{/owner}{/repo}
https://api.github.com/users/aballigier/subscriptions
User
https://api.github.com/users/aballigier
public
MEMBER
I opened a PR that allows to reload modules that have already been loaded once even if there's no internet.

Let me know if you know other ways that can make the offline mode experience better. I'd be happy to add them :) 

I already note the "freeze" modules option, to prevent local modules updates. It would be a cool feature.

----------

> @mandubian's second bullet point suggests that there's a workaround allowing you to use your offline (custom?) dataset with `datasets`. Could you please elaborate on how that should look like?

Indeed `load_dataset` allows to load remote dataset script (squad, glue, etc.) but also you own local ones.
For example if you have a dataset script at `./my_dataset/my_dataset.py` then you can do
```python
load_dataset("./my_dataset")
```
and the dataset script will generate your dataset once and for all.

----------

About I'm looking into having `csv`, `json`, `text`, `pandas` dataset builders already included in the `datasets` package, so that they are available offline by default, as opposed to the other datasets that require the script to be downloaded.
cf #1724 
2021-01-12T15:53:47Z
https://github.com/huggingface/datasets/issues/824#issuecomment-758751773
https://api.github.com/repos/huggingface/datasets/issues/824
MDEyOklzc3VlQ29tbWVudDc1ODc1MTc3Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/758751773/reactions
2021-01-12T16:41:27Z
https://api.github.com/repos/huggingface/datasets/issues/comments/758751773
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
The local dataset builders (csv, text , json and pandas) are now part of the `datasets` package since #1726 :)
You can now use them offline
```python
datasets = load_dataset('text', data_files=data_files)
```

We'll do a new release soon
2021-01-20T14:05:04Z
https://github.com/huggingface/datasets/issues/824#issuecomment-763626973
https://api.github.com/repos/huggingface/datasets/issues/824
MDEyOklzc3VlQ29tbWVudDc2MzYyNjk3Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/763626973/reactions
2021-01-20T14:05:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/763626973
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Already fixed by:
- #1726
2022-02-15T10:32:36Z
https://github.com/huggingface/datasets/issues/824#issuecomment-1040110215
https://api.github.com/repos/huggingface/datasets/issues/824
IC_kwDODunzps49_tKH
https://api.github.com/repos/huggingface/datasets/issues/comments/1040110215/reactions
2022-02-15T10:32:36Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1040110215
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
NONE
> 

really helps
2023-09-03T14:27:03Z
https://github.com/huggingface/datasets/issues/824#issuecomment-1704320232
https://api.github.com/repos/huggingface/datasets/issues/824
IC_kwDODunzps5lldzo
https://api.github.com/repos/huggingface/datasets/issues/comments/1704320232/reactions
2023-09-03T14:27:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1704320232
https://avatars.githubusercontent.com/u/50269851?v=4
https://api.github.com/users/sunyuhan19981208/events{/privacy}
https://api.github.com/users/sunyuhan19981208/followers
https://api.github.com/users/sunyuhan19981208/following{/other_user}
https://api.github.com/users/sunyuhan19981208/gists{/gist_id}
https://github.com/sunyuhan19981208
sunyuhan19981208
MDQ6VXNlcjUwMjY5ODUx
https://api.github.com/users/sunyuhan19981208/orgs
https://api.github.com/users/sunyuhan19981208/received_events
https://api.github.com/users/sunyuhan19981208/repos
https://api.github.com/users/sunyuhan19981208/starred{/owner}{/repo}
https://api.github.com/users/sunyuhan19981208/subscriptions
User
https://api.github.com/users/sunyuhan19981208
public
NONE
@albertvillanova 
datasets version： 2.10.1
I load_dataset and save_to_disk sucessfully on windows10, and I copy the dataset dir
into a ubuntu system, and when I load_from_disk(dir),  something weird happens:


```
load_from_disk('/LLM/data/wiki')
  File "/usr/local/miniconda3/lib/python3.8/site-packages/datasets/load.py", line 1874, in load_from_disk
    return DatasetDict.load_from_disk(dataset_path, keep_in_memory=keep_in_memory, storage_options=storage_options)
  File "/usr/local/miniconda3/lib/python3.8/site-packages/datasets/dataset_dict.py", line 1309, in load_from_disk
    dataset_dict[k] = Dataset.load_from_disk(
  File "/usr/local/miniconda3/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1543, in load_from_disk
    fs_token_paths = fsspec.get_fs_token_paths(dataset_path, storage_options=storage_options)
  File "/usr/local/miniconda3/lib/python3.8/site-packages/fsspec/core.py", line 610, in get_fs_token_paths
    chain = _un_chain(urlpath0, storage_options or {})
  File "/usr/local/miniconda3/lib/python3.8/site-packages/fsspec/core.py", line 325, in _un_chain
    cls = get_filesystem_class(protocol)
  File "/usr/local/miniconda3/lib/python3.8/site-packages/fsspec/registry.py", line 232, in get_filesystem_class
    raise ValueError(f"Protocol not known: {protocol}")
ValueError: Protocol not known: /LLM/data/wiki
```
It seems that something went wrong on the arrow file?
How can I solve this , since currently I can not save_to_disk on ubuntu system
2023-10-26T03:44:11Z
https://github.com/huggingface/datasets/issues/824#issuecomment-1780368062
https://api.github.com/repos/huggingface/datasets/issues/824
IC_kwDODunzps5qHkK-
https://api.github.com/repos/huggingface/datasets/issues/comments/1780368062/reactions
2023-10-26T03:44:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1780368062
https://avatars.githubusercontent.com/u/13804492?v=4
https://api.github.com/users/brisker/events{/privacy}
https://api.github.com/users/brisker/followers
https://api.github.com/users/brisker/following{/other_user}
https://api.github.com/users/brisker/gists{/gist_id}
https://github.com/brisker
brisker
MDQ6VXNlcjEzODA0NDky
https://api.github.com/users/brisker/orgs
https://api.github.com/users/brisker/received_events
https://api.github.com/users/brisker/repos
https://api.github.com/users/brisker/starred{/owner}{/repo}
https://api.github.com/users/brisker/subscriptions
User
https://api.github.com/users/brisker
public
MEMBER
It looks like a bug in `fsspec`, can you try updating `fsspec` (and maybe `datasets` as well) ?
2023-10-26T09:26:26Z
https://github.com/huggingface/datasets/issues/824#issuecomment-1780744610
https://api.github.com/repos/huggingface/datasets/issues/824
IC_kwDODunzps5qJAGi
https://api.github.com/repos/huggingface/datasets/issues/comments/1780744610/reactions
2023-10-26T09:26:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1780744610
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
`datasets.load_dataset("csv", ...)` breaks if you have no connection (There is already this issue https://github.com/huggingface/datasets/issues/761 about it). It seems to be the same for metrics too.

I create this ticket to discuss a bit and gather what you have in mind or other propositions.

Here are some points to open discussion:
- if you want to prepare your code/datasets on your machine (having internet connexion) but run it on another offline machine (not having internet connexion), it won't work as is, even if you have all files locally on this machine.
- AFAIK, you can make it work if you manually put the python files (csv.py for example) on this offline machine and change your code to `datasets.load_dataset("MY_PATH/csv.py", ...)`. But it would be much better if you could run ths same code without modification if files are available locally.
- I've also been considering the requirement of downloading Python code and execute on your machine to use datasets. This can be an issue in a professional context. Downloading a CSV/H5 file is acceptable, downloading an executable script can open many security issues. We certainly need a mechanism to at least "freeze" the dataset code you retrieved once so that you can review it if you want and then be sure you use this one everywhere and not a version dowloaded from internet.
 
WDYT? (thks)

