❓ Dataset loading script from Hugging Face Hub
MEMBER
Hi ! In the next version of `datasets`, your train and test splits will be correctly separated (changes from #3027) if you create a dataset repository with only your CSV files.

Also it seems that you overwrite the `data_files` and `data_dir` arguments in your code, when you instantiate the AGNewsConfig objects. Those parameters are not necessary since you already know which files you want to load.

You can find an example on how to specify which file the dataset has to download in this [example script](https://huggingface.co/datasets/lhoestq/custom_squad/blob/main/custom_squad.py#L101-L107):
```python
_URLS = {
    "train": "train-v1.1.json",  # you can use a URL or a relative path from the python script to your file in the repository
    "dev": "dev-v1.1.json",
}
```
```python
    def _split_generators(self, dl_manager):
        downloaded_files = dl_manager.download_and_extract(_URLS)

        return [
            datasets.SplitGenerator(name=datasets.Split.TRAIN, gen_kwargs={"filepath": downloaded_files["train"]}),
            datasets.SplitGenerator(name=datasets.Split.VALIDATION, gen_kwargs={"filepath": downloaded_files["dev"]}),
        ]
```
2021-11-19T15:30:42Z
https://github.com/huggingface/datasets/issues/3300#issuecomment-974170862
https://api.github.com/repos/huggingface/datasets/issues/3300
IC_kwDODunzps46EKru
https://api.github.com/repos/huggingface/datasets/issues/comments/974170862/reactions
2021-11-19T15:30:42Z
https://api.github.com/repos/huggingface/datasets/issues/comments/974170862
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Also I think the viewer will be updated when you fix the dataset script, let me know if it doesn't
2021-11-19T15:31:48Z
https://github.com/huggingface/datasets/issues/3300#issuecomment-974171773
https://api.github.com/repos/huggingface/datasets/issues/3300
IC_kwDODunzps46EK59
https://api.github.com/repos/huggingface/datasets/issues/comments/974171773/reactions
2021-11-19T15:31:48Z
https://api.github.com/repos/huggingface/datasets/issues/comments/974171773
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Hi @lhoestq,

Thanks a lot for the super quick answer!

Your suggestion solves my issue. I am now able to load the dataset properly 🚀 
However, the dataviewer is not working yet.

Really, thanks a lot for your help and consideration!

Best,
Pietro
2021-11-19T15:36:10Z
https://github.com/huggingface/datasets/issues/3300#issuecomment-974175195
https://api.github.com/repos/huggingface/datasets/issues/3300
IC_kwDODunzps46ELvb
https://api.github.com/repos/huggingface/datasets/issues/comments/974175195/reactions
2021-11-19T15:36:10Z
https://api.github.com/repos/huggingface/datasets/issues/comments/974175195
https://avatars.githubusercontent.com/u/61748653?v=4
https://api.github.com/users/pietrolesci/events{/privacy}
https://api.github.com/users/pietrolesci/followers
https://api.github.com/users/pietrolesci/following{/other_user}
https://api.github.com/users/pietrolesci/gists{/gist_id}
https://github.com/pietrolesci
pietrolesci
MDQ6VXNlcjYxNzQ4NjUz
https://api.github.com/users/pietrolesci/orgs
https://api.github.com/users/pietrolesci/received_events
https://api.github.com/users/pietrolesci/repos
https://api.github.com/users/pietrolesci/starred{/owner}{/repo}
https://api.github.com/users/pietrolesci/subscriptions
User
https://api.github.com/users/pietrolesci
public
MEMBER
Great ! We'll take a look at the viewer to fix it
2021-11-19T16:49:11Z
https://github.com/huggingface/datasets/issues/3300#issuecomment-974235485
https://api.github.com/repos/huggingface/datasets/issues/3300
IC_kwDODunzps46Eadd
https://api.github.com/repos/huggingface/datasets/issues/comments/974235485/reactions
2021-11-19T16:49:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/974235485
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
@lhoestq  I think I am having a related problem.
My call to load_dataset() looks like this:

```
    datasets = load_dataset(
        os.path.abspath(layoutlmft.data.datasets.xfun.__file__),
        f"xfun.{data_args.lang}",
        additional_langs=data_args.additional_langs,
        keep_in_memory=True,
    )

```

My _split_generation code is:

```
    def _split_generators(self, dl_manager):
        """Returns SplitGenerators."""

        downloaded_file = dl_manager.download_and_extract("https://guillaumejaume.github.io/FUNSD/dataset.zip")
        return [
            datasets.SplitGenerator(
                name=datasets.Split.TRAIN, gen_kwargs={"filepath": f"{downloaded_file}/dataset/training_data/"}
            ),
            datasets.SplitGenerator(
                name=datasets.Split.TEST, gen_kwargs={"filepath": f"{downloaded_file}/dataset/testing_data/"}
            ),
        ]

```
However I get the error "TypeError: _generate_examples() got an unexpected keyword argument 'filepath'"
The path looks right and I see the data in the path so I think the only problem I have is that it doesn't like the key "filepath".  However, the documentation (example [here](https://huggingface.co/datasets/lhoestq/custom_squad/blob/main/custom_squad.py#L101-L107)) seems to show that this is the correct parameter. 

Here is the full stack trace:

```
Downloading and preparing dataset xfun/xfun.en (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /Users/caseygre/.cache/huggingface/datasets/xfun/xfun.en/0.0.0/96b8cb7c57f6f822f0ab37ae3be7b82d84ac57062e774c9361ccf0a4b9ef61cc...
Traceback (most recent call last):
  File "/Users/caseygre/PycharmProjects/aegis-ml-new/unilm/venv-LayoutLM/lib/python3.9/site-packages/datasets/builder.py", line 574, in download_and_prepare
    self._download_and_prepare(
  File "/Users/caseygre/PycharmProjects/aegis-ml-new/unilm/venv-LayoutLM/lib/python3.9/site-packages/datasets/builder.py", line 652, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/Users/caseygre/PycharmProjects/aegis-ml-new/unilm/venv-LayoutLM/lib/python3.9/site-packages/datasets/builder.py", line 975, in _prepare_split
    generator = self._generate_examples(**split_generator.gen_kwargs)
TypeError: _generate_examples() got an unexpected keyword argument 'filepath'
python-BaseException
```
2021-11-19T22:45:47Z
https://github.com/huggingface/datasets/issues/3300#issuecomment-974521379
https://api.github.com/repos/huggingface/datasets/issues/3300
IC_kwDODunzps46FgQj
https://api.github.com/repos/huggingface/datasets/issues/comments/974521379/reactions
2021-11-19T22:46:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/974521379
https://avatars.githubusercontent.com/u/3164663?v=4
https://api.github.com/users/gwc4github/events{/privacy}
https://api.github.com/users/gwc4github/followers
https://api.github.com/users/gwc4github/following{/other_user}
https://api.github.com/users/gwc4github/gists{/gist_id}
https://github.com/gwc4github
gwc4github
MDQ6VXNlcjMxNjQ2NjM=
https://api.github.com/users/gwc4github/orgs
https://api.github.com/users/gwc4github/received_events
https://api.github.com/users/gwc4github/repos
https://api.github.com/users/gwc4github/starred{/owner}{/repo}
https://api.github.com/users/gwc4github/subscriptions
User
https://api.github.com/users/gwc4github
public
MEMBER
Hi ! The `gen_kwargs` dictionary is passed to `_generate_examples`, so in your case it must be defined this way:
```python
def _generate_examples(self, filepath):
    ...
```

And here is an additional tip: you can use `os.path.join(downloaded_file, "dataset/testing_data")` instead of `f"downloaded_file}/dataset/testing_data/"` to get compatibility with Windows and streaming.

Indeed Windows uses a backslash separator, not a slash, and streaming uses chained URLs (like `zip://dataset/testing_data::https://https://guillaumejaume.github.io/FUNSD/dataset.zip` for example)
2021-11-24T16:45:01Z
https://github.com/huggingface/datasets/issues/3300#issuecomment-978049047
https://api.github.com/repos/huggingface/datasets/issues/3300
IC_kwDODunzps46S9gX
https://api.github.com/repos/huggingface/datasets/issues/comments/978049047/reactions
2021-11-24T16:45:01Z
https://api.github.com/repos/huggingface/datasets/issues/comments/978049047
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Thanks for you quick reply @lhoestq and so sorry for my very delayed response.
We have gotten around the error another way but I will try to duplicate this when I can.  We may have had "filepaths" instead of "filepath" in our def of _generate_examples() and not noticed the difference.  If I find a more useful answer for others I will add to this ticket so they know what the issue was.
Note: we do have our own _generate_examples() defined with the same def as Quentin has.  (But one version does have "filepaths".)

2021-12-02T14:34:33Z
https://github.com/huggingface/datasets/issues/3300#issuecomment-984683437
https://api.github.com/repos/huggingface/datasets/issues/3300
IC_kwDODunzps46sROt
https://api.github.com/repos/huggingface/datasets/issues/comments/984683437/reactions
2021-12-02T14:34:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/984683437
https://avatars.githubusercontent.com/u/88689949?v=4
https://api.github.com/users/gregg-ADP/events{/privacy}
https://api.github.com/users/gregg-ADP/followers
https://api.github.com/users/gregg-ADP/following{/other_user}
https://api.github.com/users/gregg-ADP/gists{/gist_id}
https://github.com/gregg-ADP
gregg-ADP
MDQ6VXNlcjg4Njg5OTQ5
https://api.github.com/users/gregg-ADP/orgs
https://api.github.com/users/gregg-ADP/received_events
https://api.github.com/users/gregg-ADP/repos
https://api.github.com/users/gregg-ADP/starred{/owner}{/repo}
https://api.github.com/users/gregg-ADP/subscriptions
User
https://api.github.com/users/gregg-ADP
public
COLLABORATOR
Fixed in the viewer: https://huggingface.co/datasets/pietrolesci/ag_news
2021-12-22T10:57:56Z
https://github.com/huggingface/datasets/issues/3300#issuecomment-999482019
https://api.github.com/repos/huggingface/datasets/issues/3300
IC_kwDODunzps47kuKj
https://api.github.com/repos/huggingface/datasets/issues/comments/999482019/reactions
2021-12-22T10:57:56Z
https://api.github.com/repos/huggingface/datasets/issues/comments/999482019
https://avatars.githubusercontent.com/u/1676121?v=4
https://api.github.com/users/severo/events{/privacy}
https://api.github.com/users/severo/followers
https://api.github.com/users/severo/following{/other_user}
https://api.github.com/users/severo/gists{/gist_id}
https://github.com/severo
severo
MDQ6VXNlcjE2NzYxMjE=
https://api.github.com/users/severo/orgs
https://api.github.com/users/severo/received_events
https://api.github.com/users/severo/repos
https://api.github.com/users/severo/starred{/owner}{/repo}
https://api.github.com/users/severo/subscriptions
User
https://api.github.com/users/severo
public
Hi there,

I am trying to add my custom `ag_news` with its own loading script on the Hugging Face datasets hub. In particular, I would like to test the addition of a second configuration to the existing `ag_news` dataset. Once it works in my hub, I plan to make a PR to the original dataset. However, in trying to do so I have encountered certain problems as detailed below.

Issues I have encountered:
- Without a loading script, the train and test files are loaded together into a unique `dataset.Dataset` -> so I wrote a loading script. Also, I need a loading script otherwise I cannot specify multiple configurations
- Once my loading script is working locally, I do not manage to make it work on the hub. In particular, I would like to be able to load the dataset like this
```python
load_dataset("pietrolesci/ag_news", name="my_configuration")
```

Apparently, the `load_dataset` is able to pick up the loading script from the hub and run it. However, it errors because it is unable to find the files. The structure of my hub repo is the following
```
ag_news.py
train.csv
test.csv
```
and the loading script I specify  `data_dir=Path(__file__).parent` and `data_files=DataFilesDict({"train": "train.csv", "test": "test.csv"})`. In the documentation I could not find info regarding loading a dataset from the hub using a loading script present on the hub.

Any suggestion is very much appreciated.

Best,
Pietro

Link to the hub repo: https://huggingface.co/datasets/pietrolesci/ag_news

BONUS: how can I make the data viewer work in this specific case? :)