Huggingface NLP, Uploading custom dataset
CONTRIBUTOR
What do you mean 'custom' ? You may want to elaborate on it when ask a question.

Anyway, there are two things you may interested
`nlp.Dataset.from_file` and `load_dataset(..., cache_dir=)`
2020-06-05T03:51:44Z
https://github.com/huggingface/datasets/issues/234#issuecomment-639242711
https://api.github.com/repos/huggingface/datasets/issues/234
MDEyOklzc3VlQ29tbWVudDYzOTI0MjcxMQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/639242711/reactions
2020-06-05T03:51:44Z
https://api.github.com/repos/huggingface/datasets/issues/comments/639242711
https://avatars.githubusercontent.com/u/17963619?v=4
https://api.github.com/users/richarddwang/events{/privacy}
https://api.github.com/users/richarddwang/followers
https://api.github.com/users/richarddwang/following{/other_user}
https://api.github.com/users/richarddwang/gists{/gist_id}
https://github.com/richarddwang
richarddwang
MDQ6VXNlcjE3OTYzNjE5
https://api.github.com/users/richarddwang/orgs
https://api.github.com/users/richarddwang/received_events
https://api.github.com/users/richarddwang/repos
https://api.github.com/users/richarddwang/starred{/owner}{/repo}
https://api.github.com/users/richarddwang/subscriptions
User
https://api.github.com/users/richarddwang
public
MEMBER
To load a dataset you need to have a script that defines the format of the examples, the splits and the way to generate examples. As your dataset has the same format of squad, you can just copy the squad script (see the [datasets](https://github.com/huggingface/nlp/tree/master/datasets) forlder) and just replace the url to load the data to your local or remote path.

Then what you can do is `load_dataset(<path/to/your/script>)`
2020-06-08T08:49:57Z
https://github.com/huggingface/datasets/issues/234#issuecomment-640463864
https://api.github.com/repos/huggingface/datasets/issues/234
MDEyOklzc3VlQ29tbWVudDY0MDQ2Mzg2NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/640463864/reactions
2020-06-08T08:50:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/640463864
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Also if you want to upload your script, you should be able to use the `nlp-cli`.

Unfortunately the upload feature was not shipped in the latest version 0.2.0. so right now you can either clone the repo to use it or wait for the next release. We will add some docs to explain how to upload datasets.

2020-06-08T08:56:55Z
https://github.com/huggingface/datasets/issues/234#issuecomment-640467868
https://api.github.com/repos/huggingface/datasets/issues/234
MDEyOklzc3VlQ29tbWVudDY0MDQ2Nzg2OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/640467868/reactions
2020-06-08T08:56:55Z
https://api.github.com/repos/huggingface/datasets/issues/comments/640467868
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Since the latest release 0.2.1 you can use 
```bash
nlp-cli upload_dataset <path/to/dataset>
```
where `<path/to/dataset>` is a path to a folder containing your script (ex: `squad.py`).
This will upload the script under your namespace on our S3.

Optionally the folder can also contain `dataset_infos.json` generated using
```bash
nlp-cli test <path/to/dataset> --all_configs --save_infos
```

Then you should be able to do
```python
nlp.load_dataset("my_namespace/dataset_name")
```
2020-06-15T16:53:58Z
https://github.com/huggingface/datasets/issues/234#issuecomment-644251043
https://api.github.com/repos/huggingface/datasets/issues/234
MDEyOklzc3VlQ29tbWVudDY0NDI1MTA0Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/644251043/reactions
2020-06-15T16:53:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/644251043
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
Hello,

Does anyone know how we can call our custom dataset using the nlp.load command? Let's say that I have a dataset based on the same format as that of squad-v1.1, how am I supposed to load it using huggingface nlp.

Thank you!