Added PragmEval benchmark
CONTRIBUTOR
> Really cool ! Thanks for adding this one :)
> Good job at adding all those citations for each task
> 
> Looks like the dummy data test doesn't pass. Maybe some files are missing in the dummy_data.zip files ?
> The error reports `pragmeval/verifiability/train.tsv` to be missing
> 
> Also could you add the tags part of the dataset card (the rest is optional) ?
> See more info here : https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md#tag-the-dataset-and-write-the-dataset-card

In the prior commits I generated dataset_infos and the dummy files myself
Now they are generated with the cli, and the tests now seem to be passing better
I will look into the tag

2020-12-01T18:20:31Z
https://github.com/huggingface/datasets/pull/955#issuecomment-736732369
https://api.github.com/repos/huggingface/datasets/issues/955
MDEyOklzc3VlQ29tbWVudDczNjczMjM2OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/736732369/reactions
2020-12-01T18:20:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/736732369
https://avatars.githubusercontent.com/u/9168444?v=4
https://api.github.com/users/sileod/events{/privacy}
https://api.github.com/users/sileod/followers
https://api.github.com/users/sileod/following{/other_user}
https://api.github.com/users/sileod/gists{/gist_id}
https://github.com/sileod
sileod
MDQ6VXNlcjkxNjg0NDQ=
https://api.github.com/users/sileod/orgs
https://api.github.com/users/sileod/received_events
https://api.github.com/users/sileod/repos
https://api.github.com/users/sileod/starred{/owner}{/repo}
https://api.github.com/users/sileod/subscriptions
User
https://api.github.com/users/sileod
public
MEMBER
Looks like you did a good job with dummy data in the first place !
The downside of automatically generated dummy data is that the files are heavier (here 40KB per file).
If you could replace the generated dummy files with the one you created yourself it would be awesome, since the one you did yourself are way lighter (around 1KB per file). Using small files make `git clone` run faster so we encourage to use small dummy_data files.
2020-12-01T23:32:15Z
https://github.com/huggingface/datasets/pull/955#issuecomment-736887935
https://api.github.com/repos/huggingface/datasets/issues/955
MDEyOklzc3VlQ29tbWVudDczNjg4NzkzNQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/736887935/reactions
2020-12-01T23:32:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/736887935
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
could you rebase from master ? it should fix the CI
2020-12-02T09:12:12Z
https://github.com/huggingface/datasets/pull/955#issuecomment-737097706
https://api.github.com/repos/huggingface/datasets/issues/955
MDEyOklzc3VlQ29tbWVudDczNzA5NzcwNg==
https://api.github.com/repos/huggingface/datasets/issues/comments/737097706/reactions
2020-12-02T09:12:12Z
https://api.github.com/repos/huggingface/datasets/issues/comments/737097706
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
> could you rebase from master ? it should fix the CI

I think it is due to the file structure of the dummy data that causes test failure. The automatically generated dummy data pass the tests
2020-12-02T10:01:57Z
https://github.com/huggingface/datasets/pull/955#issuecomment-737125568
https://api.github.com/repos/huggingface/datasets/issues/955
MDEyOklzc3VlQ29tbWVudDczNzEyNTU2OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/737125568/reactions
2020-12-02T10:01:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/737125568
https://avatars.githubusercontent.com/u/9168444?v=4
https://api.github.com/users/sileod/events{/privacy}
https://api.github.com/users/sileod/followers
https://api.github.com/users/sileod/following{/other_user}
https://api.github.com/users/sileod/gists{/gist_id}
https://github.com/sileod
sileod
MDQ6VXNlcjkxNjg0NDQ=
https://api.github.com/users/sileod/orgs
https://api.github.com/users/sileod/received_events
https://api.github.com/users/sileod/repos
https://api.github.com/users/sileod/starred{/owner}{/repo}
https://api.github.com/users/sileod/subscriptions
User
https://api.github.com/users/sileod
public
MEMBER
Indeed the error reports that `pragmeval/verifiability/train.tsv` is missing for the verifiability dummy_data.zip file.
To fix that you should add the missing data files in each dummy_data.zip file.
To test that your dummy data work you can run
```
RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_all_configs_
```
if some file is missing it should tell you which one
2020-12-02T10:10:54Z
https://github.com/huggingface/datasets/pull/955#issuecomment-737130395
https://api.github.com/repos/huggingface/datasets/issues/955
MDEyOklzc3VlQ29tbWVudDczNzEzMDM5NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/737130395/reactions
2020-12-02T10:10:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/737130395
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Also it looks like you haven't rebased from master yet, even though you did a `rebase` commit. 

rebasing should fix the other CI fails
2020-12-02T10:12:06Z
https://github.com/huggingface/datasets/pull/955#issuecomment-737131009
https://api.github.com/repos/huggingface/datasets/issues/955
MDEyOklzc3VlQ29tbWVudDczNzEzMTAwOQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/737131009/reactions
2020-12-02T10:12:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/737131009
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
It's ok if we have `RemoteDatasetTest ` errors, they're fixed on master
2020-12-03T09:27:39Z
https://github.com/huggingface/datasets/pull/955#issuecomment-737778505
https://api.github.com/repos/huggingface/datasets/issues/955
MDEyOklzc3VlQ29tbWVudDczNzc3ODUwNQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/737778505/reactions
2020-12-03T09:27:39Z
https://api.github.com/repos/huggingface/datasets/issues/comments/737778505
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
merging since the CI is fixed on master
2020-12-03T09:36:40Z
https://github.com/huggingface/datasets/pull/955#issuecomment-737786396
https://api.github.com/repos/huggingface/datasets/issues/955
MDEyOklzc3VlQ29tbWVudDczNzc4NjM5Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/737786396/reactions
2020-12-03T09:36:40Z
https://api.github.com/repos/huggingface/datasets/issues/comments/737786396
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Hey @sileod! Super nice to see you participating ;)

Did you officially joined the sprint by posting on [the forum thread](https://discuss.huggingface.co/t/open-to-the-community-one-week-team-effort-to-reach-v2-0-of-hf-datasets-library/2176) and joining our slack?

I can't seem to find you there! Should I add you directly with your gmail address?
2020-12-04T08:55:10Z
https://github.com/huggingface/datasets/pull/955#issuecomment-738656755
https://api.github.com/repos/huggingface/datasets/issues/955
MDEyOklzc3VlQ29tbWVudDczODY1Njc1NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/738656755/reactions
2020-12-04T08:55:10Z
https://api.github.com/repos/huggingface/datasets/issues/comments/738656755
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
MEMBER
Hi @sileod 👋 
2020-12-04T10:43:32Z
https://github.com/huggingface/datasets/pull/955#issuecomment-738711800
https://api.github.com/repos/huggingface/datasets/issues/955
MDEyOklzc3VlQ29tbWVudDczODcxMTgwMA==
https://api.github.com/repos/huggingface/datasets/issues/comments/738711800/reactions
2020-12-04T10:43:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/738711800
https://avatars.githubusercontent.com/u/326577?v=4
https://api.github.com/users/julien-c/events{/privacy}
https://api.github.com/users/julien-c/followers
https://api.github.com/users/julien-c/following{/other_user}
https://api.github.com/users/julien-c/gists{/gist_id}
https://github.com/julien-c
julien-c
MDQ6VXNlcjMyNjU3Nw==
https://api.github.com/users/julien-c/orgs
https://api.github.com/users/julien-c/received_events
https://api.github.com/users/julien-c/repos
https://api.github.com/users/julien-c/starred{/owner}{/repo}
https://api.github.com/users/julien-c/subscriptions
User
https://api.github.com/users/julien-c
public