Add Polyglot-NER Dataset
CONTRIBUTOR
Hi @joeddav thanks for adding this! (I did a long webarchive.org session to actually find that dataset a while ago).

One question: should we manually correct the labeling scheme to (at least) IOB1?

That means "LOC" will be converted to "I-LOC". IOB1 is not explict. mentioned in the paper, but it is used in the documentation:

https://polyglot.readthedocs.io/en/latest/NamedEntityRecognition.html
2020-09-18T14:13:26Z
https://github.com/huggingface/datasets/pull/641#issuecomment-694893223
https://api.github.com/repos/huggingface/datasets/issues/641
MDEyOklzc3VlQ29tbWVudDY5NDg5MzIyMw==
https://api.github.com/repos/huggingface/datasets/issues/comments/694893223/reactions
2020-09-18T14:13:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/694893223
https://avatars.githubusercontent.com/u/20651387?v=4
https://api.github.com/users/stefan-it/events{/privacy}
https://api.github.com/users/stefan-it/followers
https://api.github.com/users/stefan-it/following{/other_user}
https://api.github.com/users/stefan-it/gists{/gist_id}
https://github.com/stefan-it
stefan-it
MDQ6VXNlcjIwNjUxMzg3
https://api.github.com/users/stefan-it/orgs
https://api.github.com/users/stefan-it/received_events
https://api.github.com/users/stefan-it/repos
https://api.github.com/users/stefan-it/starred{/owner}{/repo}
https://api.github.com/users/stefan-it/subscriptions
User
https://api.github.com/users/stefan-it
public
CONTRIBUTOR
@stefan-it I went back and forth on this. My biggest problem with it is that once you are in IOB, there is the expectation that the beginning of new entities are marked with a `B-` (at least in the case of two back-to-back entities):
```
Today    O
Alice    I-PER
Bob      B-PER
and      O
I        O 
ate      O
lasagna  O
```
If we just prepend `I-` to everything, `Bob` would be incorrectly tagged `I-PER`, meaning `Bob Alice` is a single entity. The current format is bad but is at least clear that it does not contain that information.

But I could go either way if someone has a strong opinion.
2020-09-18T15:04:57Z
https://github.com/huggingface/datasets/pull/641#issuecomment-694921964
https://api.github.com/repos/huggingface/datasets/issues/641
MDEyOklzc3VlQ29tbWVudDY5NDkyMTk2NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/694921964/reactions
2020-09-18T15:04:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/694921964
https://avatars.githubusercontent.com/u/9353833?v=4
https://api.github.com/users/joeddav/events{/privacy}
https://api.github.com/users/joeddav/followers
https://api.github.com/users/joeddav/following{/other_user}
https://api.github.com/users/joeddav/gists{/gist_id}
https://github.com/joeddav
joeddav
MDQ6VXNlcjkzNTM4MzM=
https://api.github.com/users/joeddav/orgs
https://api.github.com/users/joeddav/received_events
https://api.github.com/users/joeddav/repos
https://api.github.com/users/joeddav/starred{/owner}{/repo}
https://api.github.com/users/joeddav/subscriptions
User
https://api.github.com/users/joeddav
public
MEMBER
Indeed I'm not sure we can convert them to IOB because of this issue. I'm fine with keeping it like that
2020-09-18T17:33:35Z
https://github.com/huggingface/datasets/pull/641#issuecomment-694994611
https://api.github.com/repos/huggingface/datasets/issues/641
MDEyOklzc3VlQ29tbWVudDY5NDk5NDYxMQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/694994611/reactions
2020-09-18T17:33:35Z
https://api.github.com/repos/huggingface/datasets/issues/comments/694994611
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
I'll do a release later today, hopefully we can include this dataset in the release :)

Let me know if you need help with the dummy data
2020-09-18T17:38:00Z
https://github.com/huggingface/datasets/pull/641#issuecomment-694996643
https://api.github.com/repos/huggingface/datasets/issues/641
MDEyOklzc3VlQ29tbWVudDY5NDk5NjY0Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/694996643/reactions
2020-09-18T17:38:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/694996643
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
@lhoestq cool thanks, I think I've got it right now – just zipped them wrong. I'm running tests locally now and then will push.
2020-09-18T17:59:48Z
https://github.com/huggingface/datasets/pull/641#issuecomment-695006853
https://api.github.com/repos/huggingface/datasets/issues/641
MDEyOklzc3VlQ29tbWVudDY5NTAwNjg1Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/695006853/reactions
2020-09-18T17:59:48Z
https://api.github.com/repos/huggingface/datasets/issues/comments/695006853
https://avatars.githubusercontent.com/u/9353833?v=4
https://api.github.com/users/joeddav/events{/privacy}
https://api.github.com/users/joeddav/followers
https://api.github.com/users/joeddav/following{/other_user}
https://api.github.com/users/joeddav/gists{/gist_id}
https://github.com/joeddav
joeddav
MDQ6VXNlcjkzNTM4MzM=
https://api.github.com/users/joeddav/orgs
https://api.github.com/users/joeddav/received_events
https://api.github.com/users/joeddav/repos
https://api.github.com/users/joeddav/starred{/owner}{/repo}
https://api.github.com/users/joeddav/subscriptions
User
https://api.github.com/users/joeddav
public
CONTRIBUTOR
@lhoestq set to merge?
2020-09-19T17:58:13Z
https://github.com/huggingface/datasets/pull/641#issuecomment-695338320
https://api.github.com/repos/huggingface/datasets/issues/641
MDEyOklzc3VlQ29tbWVudDY5NTMzODMyMA==
https://api.github.com/repos/huggingface/datasets/issues/comments/695338320/reactions
2020-09-19T17:58:13Z
https://api.github.com/repos/huggingface/datasets/issues/comments/695338320
https://avatars.githubusercontent.com/u/9353833?v=4
https://api.github.com/users/joeddav/events{/privacy}
https://api.github.com/users/joeddav/followers
https://api.github.com/users/joeddav/following{/other_user}
https://api.github.com/users/joeddav/gists{/gist_id}
https://github.com/joeddav
joeddav
MDQ6VXNlcjkzNTM4MzM=
https://api.github.com/users/joeddav/orgs
https://api.github.com/users/joeddav/received_events
https://api.github.com/users/joeddav/repos
https://api.github.com/users/joeddav/starred{/owner}{/repo}
https://api.github.com/users/joeddav/subscriptions
User
https://api.github.com/users/joeddav
public
CONTRIBUTOR
@joeddav I'm fine with keeping the original labeling scheme :) 
2020-09-19T21:14:01Z
https://github.com/huggingface/datasets/pull/641#issuecomment-695356668
https://api.github.com/repos/huggingface/datasets/issues/641
MDEyOklzc3VlQ29tbWVudDY5NTM1NjY2OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/695356668/reactions
2020-09-19T21:14:01Z
https://api.github.com/repos/huggingface/datasets/issues/comments/695356668
https://avatars.githubusercontent.com/u/20651387?v=4
https://api.github.com/users/stefan-it/events{/privacy}
https://api.github.com/users/stefan-it/followers
https://api.github.com/users/stefan-it/following{/other_user}
https://api.github.com/users/stefan-it/gists{/gist_id}
https://github.com/stefan-it
stefan-it
MDQ6VXNlcjIwNjUxMzg3
https://api.github.com/users/stefan-it/orgs
https://api.github.com/users/stefan-it/received_events
https://api.github.com/users/stefan-it/repos
https://api.github.com/users/stefan-it/starred{/owner}{/repo}
https://api.github.com/users/stefan-it/subscriptions
User
https://api.github.com/users/stefan-it
public
Adds the [Polyglot-NER dataset](https://sites.google.com/site/rmyeid/projects/polylgot-ner) with named entity tags for 40 languages. I include separate configs for each language as well as a `combined` config which lumps them all together.