Add CCAligned Multilingual Dataset
MEMBER
Hi !

We already have some datasets that can have many many configurations possible.
To be able to support that, we allow to subclass BuilderConfig to add as many additional parameters as you may need.
This way users can load any language they want. For example the [bible_para](https://github.com/huggingface/datasets/blob/master/datasets/bible_para/bible_para.py) dataset is a dataset for translation and therefore users should be able to provide any language pair. You can check how the subclass of BuilderConfig is defined [here](https://github.com/huggingface/datasets/blob/master/datasets/bible_para/bible_para.py#L49).

For testing, only the configurations defined in the `BUILDER_CONFIGS` class attribute are used.
All the other configs combinations are not tested, but they can be used by users. If a config doesn't already exist in `BUILDER_CONFIGS`, then it is created on the fly.
For example in [bible_para](https://github.com/huggingface/datasets/blob/master/datasets/bible_para/bible_para.py#L61), only 6 configs are defined in `BUILDER_CONFIGS`.

So what I would do in your case is have something like
```python

class CCAlignedConfig(datasets.BuilderConfig):
    def __init__(self, *args, documents_or_sentences=None, language_code=None, **kwargs):
        super().__init__(
            *args,
            name=f"{documents_or_sentences}-{language_code}",
            **kwargs,
        )
        self.documents_or_sentences = documents_or_sentences
        self.language_code = language_code
```
And of course, feel free to change/rename things if you want to. In particular I think we can improve the name of the parameter `documents_or_sentences`
2021-02-04T16:44:19Z
https://github.com/huggingface/datasets/pull/1815#issuecomment-773447238
https://api.github.com/repos/huggingface/datasets/issues/1815
MDEyOklzc3VlQ29tbWVudDc3MzQ0NzIzOA==
https://api.github.com/repos/huggingface/datasets/issues/comments/773447238/reactions
2021-02-04T16:44:19Z
https://api.github.com/repos/huggingface/datasets/issues/comments/773447238
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Hi @lhoestq,

Thanks a lot! I don't know why I didn't think about that. :P 
I'll make these changes and update.
2021-02-04T17:47:27Z
https://github.com/huggingface/datasets/pull/1815#issuecomment-773490224
https://api.github.com/repos/huggingface/datasets/issues/1815
MDEyOklzc3VlQ29tbWVudDc3MzQ5MDIyNA==
https://api.github.com/repos/huggingface/datasets/issues/comments/773490224/reactions
2021-02-04T17:47:27Z
https://api.github.com/repos/huggingface/datasets/issues/comments/773490224
https://avatars.githubusercontent.com/u/29076344?v=4
https://api.github.com/users/gchhablani/events{/privacy}
https://api.github.com/users/gchhablani/followers
https://api.github.com/users/gchhablani/following{/other_user}
https://api.github.com/users/gchhablani/gists{/gist_id}
https://github.com/gchhablani
gchhablani
MDQ6VXNlcjI5MDc2MzQ0
https://api.github.com/users/gchhablani/orgs
https://api.github.com/users/gchhablani/received_events
https://api.github.com/users/gchhablani/repos
https://api.github.com/users/gchhablani/starred{/owner}{/repo}
https://api.github.com/users/gchhablani/subscriptions
User
https://api.github.com/users/gchhablani
public
CONTRIBUTOR
Hi @lhoestq,

I have tested and added dummy files. Request you to review.

Also, does this mean BUILDER_CONFIGS is only needed while testing?
2021-02-04T19:32:30Z
https://github.com/huggingface/datasets/pull/1815#issuecomment-773552655
https://api.github.com/repos/huggingface/datasets/issues/1815
MDEyOklzc3VlQ29tbWVudDc3MzU1MjY1NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/773552655/reactions
2021-02-04T19:32:30Z
https://api.github.com/repos/huggingface/datasets/issues/comments/773552655
https://avatars.githubusercontent.com/u/29076344?v=4
https://api.github.com/users/gchhablani/events{/privacy}
https://api.github.com/users/gchhablani/followers
https://api.github.com/users/gchhablani/following{/other_user}
https://api.github.com/users/gchhablani/gists{/gist_id}
https://github.com/gchhablani
gchhablani
MDQ6VXNlcjI5MDc2MzQ0
https://api.github.com/users/gchhablani/orgs
https://api.github.com/users/gchhablani/received_events
https://api.github.com/users/gchhablani/repos
https://api.github.com/users/gchhablani/starred{/owner}{/repo}
https://api.github.com/users/gchhablani/subscriptions
User
https://api.github.com/users/gchhablani
public
CONTRIBUTOR
Hi @lhoestq,

Any changes required on this one?

Thanks,
Gunjan
2021-02-18T14:17:48Z
https://github.com/huggingface/datasets/pull/1815#issuecomment-781374675
https://api.github.com/repos/huggingface/datasets/issues/1815
MDEyOklzc3VlQ29tbWVudDc4MTM3NDY3NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/781374675/reactions
2021-02-18T14:17:48Z
https://api.github.com/repos/huggingface/datasets/issues/comments/781374675
https://avatars.githubusercontent.com/u/29076344?v=4
https://api.github.com/users/gchhablani/events{/privacy}
https://api.github.com/users/gchhablani/followers
https://api.github.com/users/gchhablani/following{/other_user}
https://api.github.com/users/gchhablani/gists{/gist_id}
https://github.com/gchhablani
gchhablani
MDQ6VXNlcjI5MDc2MzQ0
https://api.github.com/users/gchhablani/orgs
https://api.github.com/users/gchhablani/received_events
https://api.github.com/users/gchhablani/repos
https://api.github.com/users/gchhablani/starred{/owner}{/repo}
https://api.github.com/users/gchhablani/subscriptions
User
https://api.github.com/users/gchhablani
public
CONTRIBUTOR
Hi @lhoestq,

Sorry for the delay, I have added the changes from the review. For the ISO format language codes, I just selected the first two characters from the names, hoping those are correct. Let me know if you want me to verify :P

Thanks for taking the time to add such a detailed review. I'll keep all these changes in mind the next time I'm adding a dataset.

Thanks,
Gunjan
2021-02-26T02:10:15Z
https://github.com/huggingface/datasets/pull/1815#issuecomment-786360269
https://api.github.com/repos/huggingface/datasets/issues/1815
MDEyOklzc3VlQ29tbWVudDc4NjM2MDI2OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/786360269/reactions
2021-02-26T02:10:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/786360269
https://avatars.githubusercontent.com/u/29076344?v=4
https://api.github.com/users/gchhablani/events{/privacy}
https://api.github.com/users/gchhablani/followers
https://api.github.com/users/gchhablani/following{/other_user}
https://api.github.com/users/gchhablani/gists{/gist_id}
https://github.com/gchhablani
gchhablani
MDQ6VXNlcjI5MDc2MzQ0
https://api.github.com/users/gchhablani/orgs
https://api.github.com/users/gchhablani/received_events
https://api.github.com/users/gchhablani/repos
https://api.github.com/users/gchhablani/starred{/owner}{/repo}
https://api.github.com/users/gchhablani/subscriptions
User
https://api.github.com/users/gchhablani
public
CONTRIBUTOR
Hi @lhoestq,

I have changed the README, and added a single example per config. Even one example is long enough to make the files heavy. Hope that isn't an issue.

Thanks,
Gunjan
2021-02-26T21:41:07Z
https://github.com/huggingface/datasets/pull/1815#issuecomment-786908769
https://api.github.com/repos/huggingface/datasets/issues/1815
MDEyOklzc3VlQ29tbWVudDc4NjkwODc2OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/786908769/reactions
2021-02-26T21:41:07Z
https://api.github.com/repos/huggingface/datasets/issues/comments/786908769
https://avatars.githubusercontent.com/u/29076344?v=4
https://api.github.com/users/gchhablani/events{/privacy}
https://api.github.com/users/gchhablani/followers
https://api.github.com/users/gchhablani/following{/other_user}
https://api.github.com/users/gchhablani/gists{/gist_id}
https://github.com/gchhablani
gchhablani
MDQ6VXNlcjI5MDc2MzQ0
https://api.github.com/users/gchhablani/orgs
https://api.github.com/users/gchhablani/received_events
https://api.github.com/users/gchhablani/repos
https://api.github.com/users/gchhablani/starred{/owner}{/repo}
https://api.github.com/users/gchhablani/subscriptions
User
https://api.github.com/users/gchhablani
public
CONTRIBUTOR
Hi @lhoestq,

Thanks for approving.
2021-03-01T11:58:32Z
https://github.com/huggingface/datasets/pull/1815#issuecomment-787893642
https://api.github.com/repos/huggingface/datasets/issues/1815
MDEyOklzc3VlQ29tbWVudDc4Nzg5MzY0Mg==
https://api.github.com/repos/huggingface/datasets/issues/comments/787893642/reactions
2021-03-01T12:33:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/787893642
https://avatars.githubusercontent.com/u/29076344?v=4
https://api.github.com/users/gchhablani/events{/privacy}
https://api.github.com/users/gchhablani/followers
https://api.github.com/users/gchhablani/following{/other_user}
https://api.github.com/users/gchhablani/gists{/gist_id}
https://github.com/gchhablani
gchhablani
MDQ6VXNlcjI5MDc2MzQ0
https://api.github.com/users/gchhablani/orgs
https://api.github.com/users/gchhablani/received_events
https://api.github.com/users/gchhablani/repos
https://api.github.com/users/gchhablani/starred{/owner}{/repo}
https://api.github.com/users/gchhablani/subscriptions
User
https://api.github.com/users/gchhablani
public
Hello,

I'm trying to add [CCAligned Multilingual Dataset](http://www.statmt.org/cc-aligned/). This has the potential to close #1756.

This dataset has two types - Document-Pairs, and Sentence-Pairs.

The datasets are huge, so I won't be able to test all of them. At the same time, a user might only want to download one particular language and not all. To provide this feature, `load_dataset`'s `**config_kwargs` should allow some random keyword args, in this case -`language_code`. This will be needed before the dataset is downloaded and extracted.

I'm expecting the usage to be something like - 
`load_dataset('ccaligned_multilingual','documents',language_code='en_XX-af_ZA')`. Ofcourse, at a later stage we can provide just two character language codes. This also has an issue where one language has multiple files (`my_MM` and `my_MM_zaw` on the link), but before that the required functionality must be added to `load_dataset`.

It would be great if someone could either tell me an alternative way to do this, or point me to where changes need to be made, if any, apart from the `BuilderConfig` definition. 

Additionally, I believe the tests will also have to be modified if this change is made, since it would not be possible to test for any random keyword arguments. 

A decent way to go about this would be to provide all the options in a list/dictionary for `language_code` and use that to test the arguments. In essence, this is similar to the pre-trained checkpoint dictionary as `transformers`. That means writing dataset specific tests, or adding something new to dataset generation script to make it easier for everyone to add keyword arguments without having to worry about the tests.

Thanks,
Gunjan

Requesting @lhoestq / @yjernite to review.