WMT21 & WMT22
MEMBER
Hi ! That would be awesome to have them indeed, thanks for opening this issue

I just added you to the WMT org on the HF Hub if you're interested in adding those datasets.

Feel free to create a dataset repository for each dataset and upload the data files there :) preferably in ZIP archives instead of TAR archives (the current WMT scripts don't support streaming TAR archives, so it would break the dataset preview). We've also had issues with the `statmt.org` host (data unavailable, slow download speed), that's why I think it's better if we re-host the files on the Hub.

`wmt21` (and wmt22) can be added <s>in this GitHub repository I think</s> on the HF Hub under the `WMT` org (we'll move the previous ones to this org soon as well).
To add it, you can copy paste the code of the previous one (e.g. wmt19), and add the new data:
- in wmt_utils.py, add the new data subsets. You need to provide the download URLs, as well as the target and source languages
- in wmt21.py (renamed from wmt19.py), you can specify the subsets that WMT21 uses (i.e. the one you just added)
- in wmt_utils.py, define the python function that must be used to parse the subsets you added. To do so, you must go in `_generate_examples` and chose the proper `sub_generator` based on the subset name. For example, the `paracrawl_v3` subset uses the `_parse_tmx` function:

https://github.com/huggingface/datasets/blob/ede72d3f9796339701ec59899c7c31d2427046fb/datasets/wmt19/wmt_utils.py#L834-L835

Hopefully the data is in a format that is already supported and there's no need to write a new `_parse_*` function for the new subsets. Let me know if you have questions or if I can help :)
2022-07-19T10:07:44Z
https://github.com/huggingface/datasets/issues/4709#issuecomment-1188861028
https://api.github.com/repos/huggingface/datasets/issues/4709
IC_kwDODunzps5G3JRk
https://api.github.com/repos/huggingface/datasets/issues/comments/1188861028/reactions
2022-10-05T13:43:36Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1188861028
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
@Muennighoff , @lhoestq  let me know if you want me to look into this. Happy to help bring WMT21 & WMT22 datasets into 🤗 ! 
2022-10-05T12:56:17Z
https://github.com/huggingface/datasets/issues/4709#issuecomment-1268399783
https://api.github.com/repos/huggingface/datasets/issues/4709
IC_kwDODunzps5Lmj6n
https://api.github.com/repos/huggingface/datasets/issues/comments/1268399783/reactions
2022-10-05T12:56:17Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1268399783
https://avatars.githubusercontent.com/u/2330069?v=4
https://api.github.com/users/srhrshr/events{/privacy}
https://api.github.com/users/srhrshr/followers
https://api.github.com/users/srhrshr/following{/other_user}
https://api.github.com/users/srhrshr/gists{/gist_id}
https://github.com/srhrshr
srhrshr
MDQ6VXNlcjIzMzAwNjk=
https://api.github.com/users/srhrshr/orgs
https://api.github.com/users/srhrshr/received_events
https://api.github.com/users/srhrshr/repos
https://api.github.com/users/srhrshr/starred{/owner}{/repo}
https://api.github.com/users/srhrshr/subscriptions
User
https://api.github.com/users/srhrshr
public
MEMBER
Hi @srhrshr :) Sure, feel free to create a dataset repository on the Hub and start from the implementation of WMT19 if you want. Then we can move the dataset under the WMT org (we'll move the other ones there as well).

Let me know if you have questions or if I can help
2022-10-05T13:45:26Z
https://github.com/huggingface/datasets/issues/4709#issuecomment-1268461474
https://api.github.com/repos/huggingface/datasets/issues/4709
IC_kwDODunzps5Lmy-i
https://api.github.com/repos/huggingface/datasets/issues/comments/1268461474/reactions
2022-10-05T13:45:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1268461474
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
#self-assign
2022-11-23T12:22:28Z
https://github.com/huggingface/datasets/issues/4709#issuecomment-1324980344
https://api.github.com/repos/huggingface/datasets/issues/4709
IC_kwDODunzps5O-Zh4
https://api.github.com/repos/huggingface/datasets/issues/comments/1324980344/reactions
2022-11-23T12:22:28Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1324980344
https://avatars.githubusercontent.com/u/67466295?v=4
https://api.github.com/users/Aniketsingh12/events{/privacy}
https://api.github.com/users/Aniketsingh12/followers
https://api.github.com/users/Aniketsingh12/following{/other_user}
https://api.github.com/users/Aniketsingh12/gists{/gist_id}
https://github.com/Aniketsingh12
Aniketsingh12
MDQ6VXNlcjY3NDY2Mjk1
https://api.github.com/users/Aniketsingh12/orgs
https://api.github.com/users/Aniketsingh12/received_events
https://api.github.com/users/Aniketsingh12/repos
https://api.github.com/users/Aniketsingh12/starred{/owner}{/repo}
https://api.github.com/users/Aniketsingh12/subscriptions
User
https://api.github.com/users/Aniketsingh12
public
NONE
#self-assign
2023-06-19T21:28:25Z
https://github.com/huggingface/datasets/issues/4709#issuecomment-1597785441
https://api.github.com/repos/huggingface/datasets/issues/4709
IC_kwDODunzps5fPEVh
https://api.github.com/repos/huggingface/datasets/issues/comments/1597785441/reactions
2023-06-19T21:29:39Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1597785441
https://avatars.githubusercontent.com/u/92247226?v=4
https://api.github.com/users/Etelis/events{/privacy}
https://api.github.com/users/Etelis/followers
https://api.github.com/users/Etelis/following{/other_user}
https://api.github.com/users/Etelis/gists{/gist_id}
https://github.com/Etelis
Etelis
U_kgDOBX-Uug
https://api.github.com/users/Etelis/orgs
https://api.github.com/users/Etelis/received_events
https://api.github.com/users/Etelis/repos
https://api.github.com/users/Etelis/starred{/owner}{/repo}
https://api.github.com/users/Etelis/subscriptions
User
https://api.github.com/users/Etelis
public
NONE
Hello @lhoestq ,

Would it be possible for me to be granted in the WMT organization (on hf ofc) in order to facilitate dataset uploads? I've already initiated the joining process at this link: https://huggingface.co/wmt

I appreciate your help with this. Thank you!
2023-06-20T06:21:18Z
https://github.com/huggingface/datasets/issues/4709#issuecomment-1598181382
https://api.github.com/repos/huggingface/datasets/issues/4709
IC_kwDODunzps5fQlAG
https://api.github.com/repos/huggingface/datasets/issues/comments/1598181382/reactions
2023-06-20T06:21:18Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1598181382
https://avatars.githubusercontent.com/u/92247226?v=4
https://api.github.com/users/Etelis/events{/privacy}
https://api.github.com/users/Etelis/followers
https://api.github.com/users/Etelis/following{/other_user}
https://api.github.com/users/Etelis/gists{/gist_id}
https://github.com/Etelis
Etelis
U_kgDOBX-Uug
https://api.github.com/users/Etelis/orgs
https://api.github.com/users/Etelis/received_events
https://api.github.com/users/Etelis/repos
https://api.github.com/users/Etelis/starred{/owner}{/repo}
https://api.github.com/users/Etelis/subscriptions
User
https://api.github.com/users/Etelis
public
MEMBER
Hi ! Cool I just added you
2023-06-20T09:02:11Z
https://github.com/huggingface/datasets/issues/4709#issuecomment-1598392180
https://api.github.com/repos/huggingface/datasets/issues/4709
IC_kwDODunzps5fRYd0
https://api.github.com/repos/huggingface/datasets/issues/comments/1598392180/reactions
2023-06-20T09:02:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1598392180
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
## Adding a Dataset
- **Name:** WMT21 & WMT22
- **Description:** We are going to have three tracks: two small tasks and a large task.
The small tracks evaluate translation between fairly related languages and English (all pairs). The large track uses 101 languages.
- **Paper:** /
- **Data:** https://statmt.org/wmt21/large-scale-multilingual-translation-task.html https://statmt.org/wmt22/large-scale-multilingual-translation-task.html
- **Motivation:** Many more languages than previous WMT versions - Could be very high impact

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/main/ADD_NEW_DATASET.md).


I could also tackle this. I saw the existing logic for WMT models is a bit complex (datasets are stored on the wmt account & retrieved in separate wmt datasets afaict). How long do you think it would take me? @lhoestq 

