add flue dataset
CONTRIBUTOR
The dummy data file was wrong. I only fixed it for the book config. Even though the tests are all green here, this should also be fixed for all other configs. Could you take a look there @mariamabarham ? 
2020-06-08T18:00:18Z
https://github.com/huggingface/datasets/pull/253#issuecomment-640783164
https://api.github.com/repos/huggingface/datasets/issues/253
MDEyOklzc3VlQ29tbWVudDY0MDc4MzE2NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/640783164/reactions
2020-06-08T18:00:18Z
https://api.github.com/repos/huggingface/datasets/issues/comments/640783164
https://avatars.githubusercontent.com/u/23423619?v=4
https://api.github.com/users/patrickvonplaten/events{/privacy}
https://api.github.com/users/patrickvonplaten/followers
https://api.github.com/users/patrickvonplaten/following{/other_user}
https://api.github.com/users/patrickvonplaten/gists{/gist_id}
https://github.com/patrickvonplaten
patrickvonplaten
MDQ6VXNlcjIzNDIzNjE5
https://api.github.com/users/patrickvonplaten/orgs
https://api.github.com/users/patrickvonplaten/received_events
https://api.github.com/users/patrickvonplaten/repos
https://api.github.com/users/patrickvonplaten/starred{/owner}{/repo}
https://api.github.com/users/patrickvonplaten/subscriptions
User
https://api.github.com/users/patrickvonplaten
public
CONTRIBUTOR
Hi @mariamabarham 

FLUE can indeed become a very interesting benchmark for french NLP !
Unfortunately, it seems that we've both been working on adding it to the repo...
I was going to open a pull request before I came across yours.
I didn't want to open a duplicate, that's why I'm commenting here (I hope it's not rude).

When I look at your code there is one issue that jump out at me: for both `vsd` and `nsd`, the labels are missing. I believe this is more a data issue, as they were not kept in the cleaned dataframes of #223. I think the *word sense disambiguation* task was a bit misunderstood. 

Maybe you should directly use the data provided by FLUE for these ?
2020-06-08T19:16:08Z
https://github.com/huggingface/datasets/pull/253#issuecomment-640820069
https://api.github.com/repos/huggingface/datasets/issues/253
MDEyOklzc3VlQ29tbWVudDY0MDgyMDA2OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/640820069/reactions
2020-06-08T19:16:36Z
https://api.github.com/repos/huggingface/datasets/issues/comments/640820069
https://avatars.githubusercontent.com/u/37028092?v=4
https://api.github.com/users/TheophileBlard/events{/privacy}
https://api.github.com/users/TheophileBlard/followers
https://api.github.com/users/TheophileBlard/following{/other_user}
https://api.github.com/users/TheophileBlard/gists{/gist_id}
https://github.com/TheophileBlard
TheophileBlard
MDQ6VXNlcjM3MDI4MDky
https://api.github.com/users/TheophileBlard/orgs
https://api.github.com/users/TheophileBlard/received_events
https://api.github.com/users/TheophileBlard/repos
https://api.github.com/users/TheophileBlard/starred{/owner}{/repo}
https://api.github.com/users/TheophileBlard/subscriptions
User
https://api.github.com/users/TheophileBlard
public
CONTRIBUTOR
Hi @TheophileBlard thanks for pointing this out. I will give a look at it or maybe if you already done it you can update this PR. Also I haven't added yet the parsing datasets, I submited a request to get access to them. If you already have them, you can also add them.
2020-06-09T07:03:47Z
https://github.com/huggingface/datasets/pull/253#issuecomment-641077416
https://api.github.com/repos/huggingface/datasets/issues/253
MDEyOklzc3VlQ29tbWVudDY0MTA3NzQxNg==
https://api.github.com/repos/huggingface/datasets/issues/comments/641077416/reactions
2020-06-09T07:03:47Z
https://api.github.com/repos/huggingface/datasets/issues/comments/641077416
https://avatars.githubusercontent.com/u/38249783?v=4
https://api.github.com/users/mariamabarham/events{/privacy}
https://api.github.com/users/mariamabarham/followers
https://api.github.com/users/mariamabarham/following{/other_user}
https://api.github.com/users/mariamabarham/gists{/gist_id}
https://github.com/mariamabarham
mariamabarham
MDQ6VXNlcjM4MjQ5Nzgz
https://api.github.com/users/mariamabarham/orgs
https://api.github.com/users/mariamabarham/received_events
https://api.github.com/users/mariamabarham/repos
https://api.github.com/users/mariamabarham/starred{/owner}{/repo}
https://api.github.com/users/mariamabarham/subscriptions
User
https://api.github.com/users/mariamabarham
public
NONE
Hi,

As @TheophileBlard pointed out, the labels for the vsd and nsd stains are missing.

For the wsd, it is my mistake, I added the files containing the labels on the drive.
There is still the join to do between the files that I didn't have time to do. It can be done after importing the two files, however if you wish to have a single dataframe already containing all the information, I could do it but only when I have free time because I have a lot of work at the moment at INSERM with the covid.

For the nsd, I've downloaded the files at https://zenodo.org/record/3549806, and if you do the same you'll see that they don't contain any labels.
In the files, you can see that some words have a WN code. I don't know what it corresponds to. On the FLUE github, they say to use the disambiguate tool (https://github.com/getalp/disambiguate) but I don't understand what he's doing.

@mariamabarham  for the parsing datasets, I have them in my possession. What it does that I haven't shared them is that they are licensed and you have to make a request to their creators. They give them away very easily for research purposes. For another use, you have to ask a commercial licence. All this means that if the data is freely available on your librairy, their licence and their application form are no longer of interest, which is why I did not add them.
Afterwards, maybe the authors will change their policies and decide to make the data freely available through your librairy
2020-06-09T08:05:22Z
https://github.com/huggingface/datasets/pull/253#issuecomment-641109411
https://api.github.com/repos/huggingface/datasets/issues/253
MDEyOklzc3VlQ29tbWVudDY0MTEwOTQxMQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/641109411/reactions
2020-06-09T08:10:21Z
https://api.github.com/repos/huggingface/datasets/issues/comments/641109411
https://avatars.githubusercontent.com/u/58078086?v=4
https://api.github.com/users/lbourdois/events{/privacy}
https://api.github.com/users/lbourdois/followers
https://api.github.com/users/lbourdois/following{/other_user}
https://api.github.com/users/lbourdois/gists{/gist_id}
https://github.com/lbourdois
lbourdois
MDQ6VXNlcjU4MDc4MDg2
https://api.github.com/users/lbourdois/orgs
https://api.github.com/users/lbourdois/received_events
https://api.github.com/users/lbourdois/repos
https://api.github.com/users/lbourdois/starred{/owner}{/repo}
https://api.github.com/users/lbourdois/subscriptions
User
https://api.github.com/users/lbourdois
public
CONTRIBUTOR
@mariamabarham  @lbourdois, Yea I don't think we can had the parsing datasets without asking the authors permission first. I also hope they'll change their policy.

Regarding `vsd` and `nsd`, if I understand well the task, the labels are "word senses" and the goal is to find the correct word sense for each ambiguous word. For `vsd` there is one ambiguous verb per sentence, and the labels we manually annotated with "wiktionary senses". For `nsd`, there are multiple ambiguous word per sentence, and the labels are WordNet Princeton Identifiers (hence the WN tag). This dataset was translated in french & automatically aligned.

Imo, for these 2 datasets, each example should be made of:
-  a list of string tokens (the words of the sentence)
- a list of string labels (the word senses or 'O' when the word is not ambiguous.

In fact, for `vsd` it could be even simpler, with a single string label (as there is only one ambiguous verb), + some "idx" feature to indicate the location of the ambiguous verb.

Unfortunately, I cannot update your PR as I'm not a maintainer of the project. Maybe we could work together on a fork ? Here's [mine](https://github.com/TheophileBlard/nlp/commits/flue-benchmark).

2020-06-09T09:46:49Z
https://github.com/huggingface/datasets/pull/253#issuecomment-641173310
https://api.github.com/repos/huggingface/datasets/issues/253
MDEyOklzc3VlQ29tbWVudDY0MTE3MzMxMA==
https://api.github.com/repos/huggingface/datasets/issues/comments/641173310/reactions
2020-06-09T09:49:34Z
https://api.github.com/repos/huggingface/datasets/issues/comments/641173310
https://avatars.githubusercontent.com/u/37028092?v=4
https://api.github.com/users/TheophileBlard/events{/privacy}
https://api.github.com/users/TheophileBlard/followers
https://api.github.com/users/TheophileBlard/following{/other_user}
https://api.github.com/users/TheophileBlard/gists{/gist_id}
https://github.com/TheophileBlard
TheophileBlard
MDQ6VXNlcjM3MDI4MDky
https://api.github.com/users/TheophileBlard/orgs
https://api.github.com/users/TheophileBlard/received_events
https://api.github.com/users/TheophileBlard/repos
https://api.github.com/users/TheophileBlard/starred{/owner}{/repo}
https://api.github.com/users/TheophileBlard/subscriptions
User
https://api.github.com/users/TheophileBlard
public
NONE
Hi

Any news about this PR ?
Because thinking back FLUE basically offers only two new datasets : those for the Word Sense Disambiguation task (vsd and nsd).

Wouldn't it be more clever to make separate PRs to add the datasets of the other tasks which are multi-lingual (and therefore can be used for other languages) ?

XNLI being already present on your library, there would only be PAWS-X (datasets and bibtex available here : https://github.com/google-research-datasets/paws/tree/master/pawsx) and the Webis-CLS-10 dataset (dataset : https://zenodo.org/record/3251672#.XvCXN-d8taQ and bibtex : https://zenodo.org/record/3251672/export/hx#.XvCXZ-d8taQ) to do.

And next for the FLUE benchmark, all you would have to do would be to use your own library by making an nlp.load_dataset() (for example nlp.load_dataset('xnli') which is already present in your library) for each of the datasets of the benchmark tasks and to keep only the 'fr' data.



Also @mariamabarham , did you get any feedback for the parsing task dataset request?
In case of refusal from the authors, there are other datasets in French to perform this task and in this case, I would open a new topic

2020-06-22T11:39:24Z
https://github.com/huggingface/datasets/pull/253#issuecomment-647462583
https://api.github.com/repos/huggingface/datasets/issues/253
MDEyOklzc3VlQ29tbWVudDY0NzQ2MjU4Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/647462583/reactions
2020-06-22T17:12:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/647462583
https://avatars.githubusercontent.com/u/58078086?v=4
https://api.github.com/users/lbourdois/events{/privacy}
https://api.github.com/users/lbourdois/followers
https://api.github.com/users/lbourdois/following{/other_user}
https://api.github.com/users/lbourdois/gists{/gist_id}
https://github.com/lbourdois
lbourdois
MDQ6VXNlcjU4MDc4MDg2
https://api.github.com/users/lbourdois/orgs
https://api.github.com/users/lbourdois/received_events
https://api.github.com/users/lbourdois/repos
https://api.github.com/users/lbourdois/starred{/owner}{/repo}
https://api.github.com/users/lbourdois/subscriptions
User
https://api.github.com/users/lbourdois
public
CONTRIBUTOR
Hi @lbourdois ,
PAWS-X is also present in the lib, it's part of `xtreme` dataset, so it can be loaded by `nlp.load_dataset('xtreme', 'PAWS-X.fr')` for the french version.
I think the parsing and the Word Sense Disambiguation task datasets are the only missing in the lib now. 
I did not get a feedback yet for the parsing dataset.

2020-06-22T15:35:09Z
https://github.com/huggingface/datasets/pull/253#issuecomment-647599891
https://api.github.com/repos/huggingface/datasets/issues/253
MDEyOklzc3VlQ29tbWVudDY0NzU5OTg5MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/647599891/reactions
2020-06-22T15:35:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/647599891
https://avatars.githubusercontent.com/u/38249783?v=4
https://api.github.com/users/mariamabarham/events{/privacy}
https://api.github.com/users/mariamabarham/followers
https://api.github.com/users/mariamabarham/following{/other_user}
https://api.github.com/users/mariamabarham/gists{/gist_id}
https://github.com/mariamabarham
mariamabarham
MDQ6VXNlcjM4MjQ5Nzgz
https://api.github.com/users/mariamabarham/orgs
https://api.github.com/users/mariamabarham/received_events
https://api.github.com/users/mariamabarham/repos
https://api.github.com/users/mariamabarham/starred{/owner}{/repo}
https://api.github.com/users/mariamabarham/subscriptions
User
https://api.github.com/users/mariamabarham
public
CONTRIBUTOR
By the way, @TheophileBlard  I commented some days ago in your fork. It would be great if you can maybe open a new PR with your code or if you have a better way  to make it available to others for review.
2020-06-22T15:57:11Z
https://github.com/huggingface/datasets/pull/253#issuecomment-647613419
https://api.github.com/repos/huggingface/datasets/issues/253
MDEyOklzc3VlQ29tbWVudDY0NzYxMzQxOQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/647613419/reactions
2020-06-22T15:57:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/647613419
https://avatars.githubusercontent.com/u/38249783?v=4
https://api.github.com/users/mariamabarham/events{/privacy}
https://api.github.com/users/mariamabarham/followers
https://api.github.com/users/mariamabarham/following{/other_user}
https://api.github.com/users/mariamabarham/gists{/gist_id}
https://github.com/mariamabarham
mariamabarham
MDQ6VXNlcjM4MjQ5Nzgz
https://api.github.com/users/mariamabarham/orgs
https://api.github.com/users/mariamabarham/received_events
https://api.github.com/users/mariamabarham/repos
https://api.github.com/users/mariamabarham/starred{/owner}{/repo}
https://api.github.com/users/mariamabarham/subscriptions
User
https://api.github.com/users/mariamabarham
public
CONTRIBUTOR
> By the way, @TheophileBlard I commented some days ago in your fork. It would be great if you can maybe open a new PR with your code or if you have a better way to make it available to others for review.

Yea sorry, missed that! I think @lbourdois has a point, it helps no one to have the same dataset in multiple places. I will try to find some time to adapt the code of my fork and open PRs for `Webis-CLS-10` and `nsd`/`vsd`. Maybe we should group `nsd`/`vsd` together ?
2020-06-23T07:54:55Z
https://github.com/huggingface/datasets/pull/253#issuecomment-647975890
https://api.github.com/repos/huggingface/datasets/issues/253
MDEyOklzc3VlQ29tbWVudDY0Nzk3NTg5MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/647975890/reactions
2020-06-23T07:54:55Z
https://api.github.com/repos/huggingface/datasets/issues/comments/647975890
https://avatars.githubusercontent.com/u/37028092?v=4
https://api.github.com/users/TheophileBlard/events{/privacy}
https://api.github.com/users/TheophileBlard/followers
https://api.github.com/users/TheophileBlard/following{/other_user}
https://api.github.com/users/TheophileBlard/gists{/gist_id}
https://github.com/TheophileBlard
TheophileBlard
MDQ6VXNlcjM3MDI4MDky
https://api.github.com/users/TheophileBlard/orgs
https://api.github.com/users/TheophileBlard/received_events
https://api.github.com/users/TheophileBlard/repos
https://api.github.com/users/TheophileBlard/starred{/owner}{/repo}
https://api.github.com/users/TheophileBlard/subscriptions
User
https://api.github.com/users/TheophileBlard
public
MEMBER
Shall we close this PR then ? @mariamabarham @TheophileBlard @lbourdois 
2020-07-10T14:16:28Z
https://github.com/huggingface/datasets/pull/253#issuecomment-656699656
https://api.github.com/repos/huggingface/datasets/issues/253
MDEyOklzc3VlQ29tbWVudDY1NjY5OTY1Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/656699656/reactions
2020-07-10T14:16:28Z
https://api.github.com/repos/huggingface/datasets/issues/comments/656699656
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
This PR add the Flue dataset as requested in this issue #223  . @lbourdois made  a detailed description in that issue.
