[XGLUE] Adding new dataset
CONTRIBUTOR
Amazing! 
2020-10-21T12:26:39Z
https://github.com/huggingface/datasets/issues/749#issuecomment-713532234
https://api.github.com/repos/huggingface/datasets/issues/749
MDEyOklzc3VlQ29tbWVudDcxMzUzMjIzNA==
https://api.github.com/repos/huggingface/datasets/issues/comments/713532234/reactions
2020-10-21T12:26:39Z
https://api.github.com/repos/huggingface/datasets/issues/comments/713532234
https://avatars.githubusercontent.com/u/22514219?v=4
https://api.github.com/users/JetRunner/events{/privacy}
https://api.github.com/users/JetRunner/followers
https://api.github.com/users/JetRunner/following{/other_user}
https://api.github.com/users/JetRunner/gists{/gist_id}
https://github.com/JetRunner
JetRunner
MDQ6VXNlcjIyNTE0MjE5
https://api.github.com/users/JetRunner/orgs
https://api.github.com/users/JetRunner/received_events
https://api.github.com/users/JetRunner/repos
https://api.github.com/users/JetRunner/starred{/owner}{/repo}
https://api.github.com/users/JetRunner/subscriptions
User
https://api.github.com/users/JetRunner
public
CONTRIBUTOR
Small poll  @thomwolf @yjernite @lhoestq @JetRunner @qiweizhen .

As stated in the XGLUE paper: https://arxiv.org/pdf/2004.01401.pdf , for each of the 11 down-stream tasks training data is only available in English, whereas development and test data is available in multiple different language *cf.* here: 

![Screenshot from 2020-11-04 15-02-17](https://user-images.githubusercontent.com/23423619/98120893-d7499a80-1eae-11eb-9d0b-57dfe5d4ee68.png)

So, I'd suggest to have exactly 11 "language-independent" configs: "ner", "pos", ... and give the sample in each dataset in the config a "language" label being one of "ar", "bg", .... => To me this makes more sense than making languaga specific config, *e.g.* "ner-de", ...especially because training data is only available in English. Do you guys agree? 
2020-11-04T14:06:02Z
https://github.com/huggingface/datasets/issues/749#issuecomment-721751313
https://api.github.com/repos/huggingface/datasets/issues/749
MDEyOklzc3VlQ29tbWVudDcyMTc1MTMxMw==
https://api.github.com/repos/huggingface/datasets/issues/comments/721751313/reactions
2020-11-04T14:06:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/721751313
https://avatars.githubusercontent.com/u/23423619?v=4
https://api.github.com/users/patrickvonplaten/events{/privacy}
https://api.github.com/users/patrickvonplaten/followers
https://api.github.com/users/patrickvonplaten/following{/other_user}
https://api.github.com/users/patrickvonplaten/gists{/gist_id}
https://github.com/patrickvonplaten
patrickvonplaten
MDQ6VXNlcjIzNDIzNjE5
https://api.github.com/users/patrickvonplaten/orgs
https://api.github.com/users/patrickvonplaten/received_events
https://api.github.com/users/patrickvonplaten/repos
https://api.github.com/users/patrickvonplaten/starred{/owner}{/repo}
https://api.github.com/users/patrickvonplaten/subscriptions
User
https://api.github.com/users/patrickvonplaten
public
MEMBER
In this case we should have named splits, so config `ner` has splits `train`, `validation`, `test-en`, `test-ar`, `test-bg`, etc...

This is more in the spirit of the task afaiu, and will avoid making users do the filtering step themselves when testing different models or different configurations of the same model.
2020-11-04T14:50:21Z
https://github.com/huggingface/datasets/issues/749#issuecomment-721777079
https://api.github.com/repos/huggingface/datasets/issues/749
MDEyOklzc3VlQ29tbWVudDcyMTc3NzA3OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/721777079/reactions
2020-11-04T14:53:38Z
https://api.github.com/repos/huggingface/datasets/issues/comments/721777079
https://avatars.githubusercontent.com/u/10469459?v=4
https://api.github.com/users/yjernite/events{/privacy}
https://api.github.com/users/yjernite/followers
https://api.github.com/users/yjernite/following{/other_user}
https://api.github.com/users/yjernite/gists{/gist_id}
https://github.com/yjernite
yjernite
MDQ6VXNlcjEwNDY5NDU5
https://api.github.com/users/yjernite/orgs
https://api.github.com/users/yjernite/received_events
https://api.github.com/users/yjernite/repos
https://api.github.com/users/yjernite/starred{/owner}{/repo}
https://api.github.com/users/yjernite/subscriptions
User
https://api.github.com/users/yjernite
public
CONTRIBUTOR
I see your point! 

I think this would be quite feasible to do and makes sense to me as well! In the paper results are reported per language, so it seems more natural to do it this way. 

Good for me @yjernite ! What do the others think? @lhoestq 

2020-11-04T15:08:34Z
https://github.com/huggingface/datasets/issues/749#issuecomment-721787737
https://api.github.com/repos/huggingface/datasets/issues/749
MDEyOklzc3VlQ29tbWVudDcyMTc4NzczNw==
https://api.github.com/repos/huggingface/datasets/issues/comments/721787737/reactions
2020-11-04T15:08:44Z
https://api.github.com/repos/huggingface/datasets/issues/comments/721787737
https://avatars.githubusercontent.com/u/23423619?v=4
https://api.github.com/users/patrickvonplaten/events{/privacy}
https://api.github.com/users/patrickvonplaten/followers
https://api.github.com/users/patrickvonplaten/following{/other_user}
https://api.github.com/users/patrickvonplaten/gists{/gist_id}
https://github.com/patrickvonplaten
patrickvonplaten
MDQ6VXNlcjIzNDIzNjE5
https://api.github.com/users/patrickvonplaten/orgs
https://api.github.com/users/patrickvonplaten/received_events
https://api.github.com/users/patrickvonplaten/repos
https://api.github.com/users/patrickvonplaten/starred{/owner}{/repo}
https://api.github.com/users/patrickvonplaten/subscriptions
User
https://api.github.com/users/patrickvonplaten
public
CONTRIBUTOR
I agree with Yacine on this!
2020-11-04T16:08:26Z
https://github.com/huggingface/datasets/issues/749#issuecomment-721823123
https://api.github.com/repos/huggingface/datasets/issues/749
MDEyOklzc3VlQ29tbWVudDcyMTgyMzEyMw==
https://api.github.com/repos/huggingface/datasets/issues/comments/721823123/reactions
2020-11-04T16:08:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/721823123
https://avatars.githubusercontent.com/u/22514219?v=4
https://api.github.com/users/JetRunner/events{/privacy}
https://api.github.com/users/JetRunner/followers
https://api.github.com/users/JetRunner/following{/other_user}
https://api.github.com/users/JetRunner/gists{/gist_id}
https://github.com/JetRunner
JetRunner
MDQ6VXNlcjIyNTE0MjE5
https://api.github.com/users/JetRunner/orgs
https://api.github.com/users/JetRunner/received_events
https://api.github.com/users/JetRunner/repos
https://api.github.com/users/JetRunner/starred{/owner}{/repo}
https://api.github.com/users/JetRunner/subscriptions
User
https://api.github.com/users/JetRunner
public
CONTRIBUTOR
Okey actually not that easy to add things like `test-de` to `datasets` => this would be the first dataset to have this.
See: https://github.com/huggingface/datasets/pull/802
2020-11-04T17:33:10Z
https://github.com/huggingface/datasets/issues/749#issuecomment-721871727
https://api.github.com/repos/huggingface/datasets/issues/749
MDEyOklzc3VlQ29tbWVudDcyMTg3MTcyNw==
https://api.github.com/repos/huggingface/datasets/issues/comments/721871727/reactions
2020-11-04T17:33:23Z
https://api.github.com/repos/huggingface/datasets/issues/comments/721871727
https://avatars.githubusercontent.com/u/23423619?v=4
https://api.github.com/users/patrickvonplaten/events{/privacy}
https://api.github.com/users/patrickvonplaten/followers
https://api.github.com/users/patrickvonplaten/following{/other_user}
https://api.github.com/users/patrickvonplaten/gists{/gist_id}
https://github.com/patrickvonplaten
patrickvonplaten
MDQ6VXNlcjIzNDIzNjE5
https://api.github.com/users/patrickvonplaten/orgs
https://api.github.com/users/patrickvonplaten/received_events
https://api.github.com/users/patrickvonplaten/repos
https://api.github.com/users/patrickvonplaten/starred{/owner}{/repo}
https://api.github.com/users/patrickvonplaten/subscriptions
User
https://api.github.com/users/patrickvonplaten
public
MEMBER
IMO we should have one config per language. That's what we're doing for xnli, xtreme etc.
Having split names that depend on the language seems wrong. We should try to avoid split names that are not train/val/test.
Sorry for late response on this one
2020-11-18T10:39:53Z
https://github.com/huggingface/datasets/issues/749#issuecomment-729592845
https://api.github.com/repos/huggingface/datasets/issues/749
MDEyOklzc3VlQ29tbWVudDcyOTU5Mjg0NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/729592845/reactions
2020-11-18T10:39:53Z
https://api.github.com/repos/huggingface/datasets/issues/comments/729592845
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
@lhoestq agreed on having one config per language, but we also need to be able to have different split names and people are going to want to use hyphens, so we should at the very least warn them why it's failing :) E.g. for ANLI with different stages of data (currently using underscores) or https://www.tau-nlp.org/commonsenseqa with their train-sanity or dev-sanity splits
2020-11-19T16:35:21Z
https://github.com/huggingface/datasets/issues/749#issuecomment-730492872
https://api.github.com/repos/huggingface/datasets/issues/749
MDEyOklzc3VlQ29tbWVudDczMDQ5Mjg3Mg==
https://api.github.com/repos/huggingface/datasets/issues/comments/730492872/reactions
2020-11-19T16:35:21Z
https://api.github.com/repos/huggingface/datasets/issues/comments/730492872
https://avatars.githubusercontent.com/u/10469459?v=4
https://api.github.com/users/yjernite/events{/privacy}
https://api.github.com/users/yjernite/followers
https://api.github.com/users/yjernite/following{/other_user}
https://api.github.com/users/yjernite/gists{/gist_id}
https://github.com/yjernite
yjernite
MDQ6VXNlcjEwNDY5NDU5
https://api.github.com/users/yjernite/orgs
https://api.github.com/users/yjernite/received_events
https://api.github.com/users/yjernite/repos
https://api.github.com/users/yjernite/starred{/owner}{/repo}
https://api.github.com/users/yjernite/subscriptions
User
https://api.github.com/users/yjernite
public
MEMBER
Yes sure ! Could you open a separate issue for that ?
2020-11-19T16:36:57Z
https://github.com/huggingface/datasets/issues/749#issuecomment-730493804
https://api.github.com/repos/huggingface/datasets/issues/749
MDEyOklzc3VlQ29tbWVudDczMDQ5MzgwNA==
https://api.github.com/repos/huggingface/datasets/issues/comments/730493804/reactions
2020-11-19T16:36:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/730493804
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Really cool dataset 👍 btw. does Transformers support all 11 tasks 🤔 would be awesome to have a xglue script (like the "normal" glue one)
2020-11-19T19:02:47Z
https://github.com/huggingface/datasets/issues/749#issuecomment-730574871
https://api.github.com/repos/huggingface/datasets/issues/749
MDEyOklzc3VlQ29tbWVudDczMDU3NDg3MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/730574871/reactions
2020-11-19T19:02:47Z
https://api.github.com/repos/huggingface/datasets/issues/comments/730574871
https://avatars.githubusercontent.com/u/20651387?v=4
https://api.github.com/users/stefan-it/events{/privacy}
https://api.github.com/users/stefan-it/followers
https://api.github.com/users/stefan-it/following{/other_user}
https://api.github.com/users/stefan-it/gists{/gist_id}
https://github.com/stefan-it
stefan-it
MDQ6VXNlcjIwNjUxMzg3
https://api.github.com/users/stefan-it/orgs
https://api.github.com/users/stefan-it/received_events
https://api.github.com/users/stefan-it/repos
https://api.github.com/users/stefan-it/starred{/owner}{/repo}
https://api.github.com/users/stefan-it/subscriptions
User
https://api.github.com/users/stefan-it
public
CONTRIBUTOR
Just to make sure this is what we want here. If we add one config per language, 

this means that this dataset ends up with well over 100 different configs most of which will have the same `train` split. The train split is always in English. Also, I'm not sure whether it's better for the user to be honest. 

I think it could be quite confusing for the user to have

```python
train_dataset = load_dataset("xglue", "ner-de", split="train")
```

in English even though it's `ner-de`.

To be honest, I'd prefer:

```python
train_dataset = load_dataset("xglue", "ner", split="train")
test_dataset_de = load_dataset("xglue", "ner", split="test-de")
test_dataset_fr = load_dataset("xglue", "ner", split="test-fr")
```

here
2020-11-27T09:43:31Z
https://github.com/huggingface/datasets/issues/749#issuecomment-734744336
https://api.github.com/repos/huggingface/datasets/issues/749
MDEyOklzc3VlQ29tbWVudDczNDc0NDMzNg==
https://api.github.com/repos/huggingface/datasets/issues/comments/734744336/reactions
2020-11-27T09:47:25Z
https://api.github.com/repos/huggingface/datasets/issues/comments/734744336
https://avatars.githubusercontent.com/u/23423619?v=4
https://api.github.com/users/patrickvonplaten/events{/privacy}
https://api.github.com/users/patrickvonplaten/followers
https://api.github.com/users/patrickvonplaten/following{/other_user}
https://api.github.com/users/patrickvonplaten/gists{/gist_id}
https://github.com/patrickvonplaten
patrickvonplaten
MDQ6VXNlcjIzNDIzNjE5
https://api.github.com/users/patrickvonplaten/orgs
https://api.github.com/users/patrickvonplaten/received_events
https://api.github.com/users/patrickvonplaten/repos
https://api.github.com/users/patrickvonplaten/starred{/owner}{/repo}
https://api.github.com/users/patrickvonplaten/subscriptions
User
https://api.github.com/users/patrickvonplaten
public
MEMBER
Oh yes right I didn't notice the train set was always in english sorry.
Moreover it seems that the way this dataset is used is to pick a pretrained multilingual model, fine-tune it on the english train set and then evaluate on each test set (one per language).
So to better fit the usual usage of this dataset, I agree that it's better to have one test split per language. 

Something like your latest example patrick is fine imo :
```python
train_dataset = load_dataset("xglue", "ner", split="train")
test_dataset_de = load_dataset("xglue", "ner", split="test.de")
```

I just replace test-de with test.de since `-` is not allowed for split names (it has to follow the `\w+` regex), and usually we specify the language after a point. 
2020-11-27T10:38:29Z
https://github.com/huggingface/datasets/issues/749#issuecomment-734768538
https://api.github.com/repos/huggingface/datasets/issues/749
MDEyOklzc3VlQ29tbWVudDczNDc2ODUzOA==
https://api.github.com/repos/huggingface/datasets/issues/comments/734768538/reactions
2020-11-27T10:38:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/734768538
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Closing since XGLUE has been added in #802 , thanks patrick :) 
2021-01-06T10:02:55Z
https://github.com/huggingface/datasets/issues/749#issuecomment-755205802
https://api.github.com/repos/huggingface/datasets/issues/749
MDEyOklzc3VlQ29tbWVudDc1NTIwNTgwMg==
https://api.github.com/repos/huggingface/datasets/issues/comments/755205802/reactions
2021-01-06T10:02:55Z
https://api.github.com/repos/huggingface/datasets/issues/comments/755205802
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
I need xglue Urdu summarization dataset so how can i get it?
2022-09-29T13:00:59Z
https://github.com/huggingface/datasets/issues/749#issuecomment-1262244943
https://api.github.com/repos/huggingface/datasets/issues/749
IC_kwDODunzps5LPFRP
https://api.github.com/repos/huggingface/datasets/issues/comments/1262244943/reactions
2022-09-29T13:00:59Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1262244943
https://avatars.githubusercontent.com/u/113641466?v=4
https://api.github.com/users/mwz772/events{/privacy}
https://api.github.com/users/mwz772/followers
https://api.github.com/users/mwz772/following{/other_user}
https://api.github.com/users/mwz772/gists{/gist_id}
https://github.com/mwz772
mwz772
U_kgDOBsYH-g
https://api.github.com/users/mwz772/orgs
https://api.github.com/users/mwz772/received_events
https://api.github.com/users/mwz772/repos
https://api.github.com/users/mwz772/starred{/owner}{/repo}
https://api.github.com/users/mwz772/subscriptions
User
https://api.github.com/users/mwz772
public
MEMBER
According to the table in https://huggingface.co/datasets/xglue, Urdu only exists for POS and XNLI in XGLUE - not for summarization
2022-09-29T14:43:27Z
https://github.com/huggingface/datasets/issues/749#issuecomment-1262385876
https://api.github.com/repos/huggingface/datasets/issues/749
IC_kwDODunzps5LPnrU
https://api.github.com/repos/huggingface/datasets/issues/comments/1262385876/reactions
2022-09-29T14:43:27Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1262385876
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
XGLUE is a multilingual GLUE like dataset propesed in this [paper](https://arxiv.org/pdf/2004.01401.pdf).

I'm planning on adding the dataset to the library myself in a couple of weeks.
Also tagging @JetRunner @qiweizhen in case I need some guidance 