Updated OPUS Open Subtitles Dataset with metadata information
MEMBER
Hi !
About the problems you mentioned:
- Saving the infos is only done for the configurations inside the BUILDER_CONFIGS. Otherwise you would need to run the scripts on ALL language pairs, which is not what we want.
- Moreover when you're on your branch, please specify the path to your local version of the dataset script, like "./datasets/open_subtitles". Otherwise the dataset is loaded from the master branch on github.
Hope that clarifies things a bit

And of course feel free to add methods or classmethods to your builder.

2021-02-12T16:42:17Z
https://github.com/huggingface/datasets/pull/1865#issuecomment-778305251
https://api.github.com/repos/huggingface/datasets/issues/1865
MDEyOklzc3VlQ29tbWVudDc3ODMwNTI1MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/778305251/reactions
2021-02-12T16:42:17Z
https://api.github.com/repos/huggingface/datasets/issues/comments/778305251
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Great! Thank you :)
I'll close the issue as well.
2021-02-12T17:38:24Z
https://github.com/huggingface/datasets/pull/1865#issuecomment-778338172
https://api.github.com/repos/huggingface/datasets/issues/1865
MDEyOklzc3VlQ29tbWVudDc3ODMzODE3Mg==
https://api.github.com/repos/huggingface/datasets/issues/comments/778338172/reactions
2021-02-12T17:38:24Z
https://api.github.com/repos/huggingface/datasets/issues/comments/778338172
https://avatars.githubusercontent.com/u/19476123?v=4
https://api.github.com/users/Valahaar/events{/privacy}
https://api.github.com/users/Valahaar/followers
https://api.github.com/users/Valahaar/following{/other_user}
https://api.github.com/users/Valahaar/gists{/gist_id}
https://github.com/Valahaar
Valahaar
MDQ6VXNlcjE5NDc2MTIz
https://api.github.com/users/Valahaar/orgs
https://api.github.com/users/Valahaar/received_events
https://api.github.com/users/Valahaar/repos
https://api.github.com/users/Valahaar/starred{/owner}{/repo}
https://api.github.com/users/Valahaar/subscriptions
User
https://api.github.com/users/Valahaar
public
Close #1844 

Problems:
- I ran `python datasets-cli test datasets/open_subtitles --save_infos --all_configs`, hence the change in `dataset_infos.json`, but it appears that the metadata features have not been added for all pairs. Any idea why that might be?
- Possibly related to the above, I tried doing `pip uninstall datasets && pip install -e ".[dev]"` after the changes, and loading the dataset via `load_dataset("open_subtitles", lang1='hi', lang2='it')` to check if the update worked, but the loaded dataset did not contain the metadata fields (neither in the features nor doing `next(iter(dataset['train']))`). What step(s) did I miss?

Questions:
- Is it ok to have a `classmethod` in there? I have not seen any in the few other datasets I have checked. I could make it a local method of the `_generate_examples` method, but I'd rather not duplicate the logic...