[Question] Using/adding a local dataset
MEMBER
Hi @zphang,

So you can just give the local path to a dataset script file and it should work.

Here is an example:
- you can download one of the scripts in the `datasets` folder of the present repo (or clone the repo)
- then you can load it with `load_dataset('PATH/TO/YOUR/LOCAL/SCRIPT.py')`

Does it make sense?
2020-05-17T12:54:28Z
https://github.com/huggingface/datasets/issues/133#issuecomment-629792620
https://api.github.com/repos/huggingface/datasets/issues/133
MDEyOklzc3VlQ29tbWVudDYyOTc5MjYyMA==
https://api.github.com/repos/huggingface/datasets/issues/comments/629792620/reactions
2020-05-17T12:54:28Z
https://api.github.com/repos/huggingface/datasets/issues/comments/629792620
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
NONE
Could you give a more concrete example, please? 

I looked up wikitext dataset script from the repo. Should I just overwrite the `data_file` on line 98 to point to the local dataset directory? Would it work for different configurations of wikitext (wikitext2, wikitext103 etc.)?

Or maybe we can use DownloadManager to specify local dataset location? In that case, where do we use DownloadManager instance?

Thanks
2020-05-21T19:40:31Z
https://github.com/huggingface/datasets/issues/133#issuecomment-632304626
https://api.github.com/repos/huggingface/datasets/issues/133
MDEyOklzc3VlQ29tbWVudDYzMjMwNDYyNg==
https://api.github.com/repos/huggingface/datasets/issues/comments/632304626/reactions
2020-05-21T19:40:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/632304626
https://avatars.githubusercontent.com/u/1519077?v=4
https://api.github.com/users/MaveriQ/events{/privacy}
https://api.github.com/users/MaveriQ/followers
https://api.github.com/users/MaveriQ/following{/other_user}
https://api.github.com/users/MaveriQ/gists{/gist_id}
https://github.com/MaveriQ
MaveriQ
MDQ6VXNlcjE1MTkwNzc=
https://api.github.com/users/MaveriQ/orgs
https://api.github.com/users/MaveriQ/received_events
https://api.github.com/users/MaveriQ/repos
https://api.github.com/users/MaveriQ/starred{/owner}{/repo}
https://api.github.com/users/MaveriQ/subscriptions
User
https://api.github.com/users/MaveriQ
public
CONTRIBUTOR
Hi @MaveriQ , although what I am doing is to commit a new dataset, but I think looking at imdb script might help.
You may want to use `dl_manager.download_custom`, give it a url(arbitrary string), a custom_download(arbitrary function) and return a path, and finally use _get sample to fetch a sample.
2020-06-05T06:05:06Z
https://github.com/huggingface/datasets/issues/133#issuecomment-639278102
https://api.github.com/repos/huggingface/datasets/issues/133
MDEyOklzc3VlQ29tbWVudDYzOTI3ODEwMg==
https://api.github.com/repos/huggingface/datasets/issues/comments/639278102/reactions
2020-06-05T06:05:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/639278102
https://avatars.githubusercontent.com/u/17963619?v=4
https://api.github.com/users/richarddwang/events{/privacy}
https://api.github.com/users/richarddwang/followers
https://api.github.com/users/richarddwang/following{/other_user}
https://api.github.com/users/richarddwang/gists{/gist_id}
https://github.com/richarddwang
richarddwang
MDQ6VXNlcjE3OTYzNjE5
https://api.github.com/users/richarddwang/orgs
https://api.github.com/users/richarddwang/received_events
https://api.github.com/users/richarddwang/repos
https://api.github.com/users/richarddwang/starred{/owner}{/repo}
https://api.github.com/users/richarddwang/subscriptions
User
https://api.github.com/users/richarddwang
public
MEMBER
The download manager supports local directories. You can specify a local directory instead of a url and it should work.
2020-06-08T09:34:27Z
https://github.com/huggingface/datasets/issues/133#issuecomment-640489740
https://api.github.com/repos/huggingface/datasets/issues/133
MDEyOklzc3VlQ29tbWVudDY0MDQ4OTc0MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/640489740/reactions
2020-06-08T09:34:27Z
https://api.github.com/repos/huggingface/datasets/issues/comments/640489740
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Closing this one.
Feel free to re-open if you have other questions :)
2020-07-23T16:44:09Z
https://github.com/huggingface/datasets/issues/133#issuecomment-663112433
https://api.github.com/repos/huggingface/datasets/issues/133
MDEyOklzc3VlQ29tbWVudDY2MzExMjQzMw==
https://api.github.com/repos/huggingface/datasets/issues/comments/663112433/reactions
2020-07-23T16:44:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/663112433
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
Users may want to either create/modify a local copy of a dataset, or use a custom-built dataset with the same `Dataset` API as externally downloaded datasets.

It appears to be possible to point to a local dataset path rather than downloading the external ones, but I'm not exactly sure how to go about doing this.

A notebook/example script demonstrating this would be very helpful.