Adding the NorNE dataset for NER
CONTRIBUTOR
Quick question, @lhoestq. In this specific dataset, two special types `GPE_LOC` and `GPE_ORG` can easily be altered depending on the task, choosing either the more general `GPE` tag or the more specific `LOC`/`ORG` tags, conflating them with the other annotations of the same type. However, I have not found an easy way to implement that. Using splits or configs does not seem appropriate.

2021-01-18T10:47:02Z
https://github.com/huggingface/datasets/pull/1720#issuecomment-762162756
https://api.github.com/repos/huggingface/datasets/issues/1720
MDEyOklzc3VlQ29tbWVudDc2MjE2Mjc1Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/762162756/reactions
2021-01-18T10:47:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/762162756
https://avatars.githubusercontent.com/u/173537?v=4
https://api.github.com/users/versae/events{/privacy}
https://api.github.com/users/versae/followers
https://api.github.com/users/versae/following{/other_user}
https://api.github.com/users/versae/gists{/gist_id}
https://github.com/versae
versae
MDQ6VXNlcjE3MzUzNw==
https://api.github.com/users/versae/orgs
https://api.github.com/users/versae/received_events
https://api.github.com/users/versae/repos
https://api.github.com/users/versae/starred{/owner}{/repo}
https://api.github.com/users/versae/subscriptions
User
https://api.github.com/users/versae
public
CONTRIBUTOR
About the `GPE_LOC` and `GPE_ORG`. The original NorNE paper in which they published the dataset, does an evaluation on three different NER tag sets, one considering `GPE_LOC` and `GPE_ORG` as they are, another changing them to be just `GPE`, and another one by changing it to become `LOC` and `ORG`. The called these sets, `norne-full`, `norne-7`, and `norne-9`. What I would like is to provide a way for the user of this dataset to get `norne-7` and `norne-9` without having to duplicate the code.
2021-01-19T15:38:14Z
https://github.com/huggingface/datasets/pull/1720#issuecomment-762923815
https://api.github.com/repos/huggingface/datasets/issues/1720
MDEyOklzc3VlQ29tbWVudDc2MjkyMzgxNQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/762923815/reactions
2021-01-19T15:38:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/762923815
https://avatars.githubusercontent.com/u/173537?v=4
https://api.github.com/users/versae/events{/privacy}
https://api.github.com/users/versae/followers
https://api.github.com/users/versae/following{/other_user}
https://api.github.com/users/versae/gists{/gist_id}
https://github.com/versae
versae
MDQ6VXNlcjE3MzUzNw==
https://api.github.com/users/versae/orgs
https://api.github.com/users/versae/received_events
https://api.github.com/users/versae/repos
https://api.github.com/users/versae/starred{/owner}{/repo}
https://api.github.com/users/versae/subscriptions
User
https://api.github.com/users/versae
public
MEMBER
Ok I see !
I guess you can have three configurations  `norne-full`, `norne-7` and `norne-9`.
Each config can have different feature types. You can simply check for the `self.config.name` in the `_info(self)` method and pick the right ClassLabel names accordingly. And then in `_generate_examples` as well you can check for `self.config.name` to know how to process the labels to yield either GPE_LOC/GPE_ORG, GPE or LOC/ORG
2021-01-19T15:56:15Z
https://github.com/huggingface/datasets/pull/1720#issuecomment-762936612
https://api.github.com/repos/huggingface/datasets/issues/1720
MDEyOklzc3VlQ29tbWVudDc2MjkzNjYxMg==
https://api.github.com/repos/huggingface/datasets/issues/comments/762936612/reactions
2021-01-19T15:56:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/762936612
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
But I'm already using the configurations for the different language
varieties. So you propose having something like `bokmaal`, `bokmaal-7`,
etc? Would there be a different way? If not, I'd be fine the corpus as it
is until we come up with a solution. Thanks in any case.

--
Sent using a cell-phone, so sorry for the typos and wrong auto-corrections.

On Tue, Jan 19, 2021, 4:56 PM Quentin Lhoest <notifications@github.com>
wrote:

> Ok I see !
> I guess you can have three configurations norne-full, norne-7 and norne-9.
> Each config can have different feature types. You can simply check for the
> self.config.name in the _info(self) method and pick the right ClassLabel
> names accordingly. And then in _generate_examples as well you can check
> for self.config.name to know how to process the labels to yield either
> GPE_LOC/GPE_ORG, GPE or LOC/ORG
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/pull/1720#issuecomment-762936612>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AABKLYOWNDBD76WZPJHFCWLS2WTTHANCNFSM4V6GSUQA>
> .
>

2021-01-19T18:14:42Z
https://github.com/huggingface/datasets/pull/1720#issuecomment-763027069
https://api.github.com/repos/huggingface/datasets/issues/1720
MDEyOklzc3VlQ29tbWVudDc2MzAyNzA2OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/763027069/reactions
2021-01-19T18:14:42Z
https://api.github.com/repos/huggingface/datasets/issues/comments/763027069
https://avatars.githubusercontent.com/u/173537?v=4
https://api.github.com/users/versae/events{/privacy}
https://api.github.com/users/versae/followers
https://api.github.com/users/versae/following{/other_user}
https://api.github.com/users/versae/gists{/gist_id}
https://github.com/versae
versae
MDQ6VXNlcjE3MzUzNw==
https://api.github.com/users/versae/orgs
https://api.github.com/users/versae/received_events
https://api.github.com/users/versae/repos
https://api.github.com/users/versae/starred{/owner}{/repo}
https://api.github.com/users/versae/subscriptions
User
https://api.github.com/users/versae
public
MEMBER
The first option about having configurations like `bokmaal-7`, `bokmaal-9` etc. would definitely work.

A second option would be to add a parameter `ner_tags_set` to `NorneConfig` and then one could load them with
```python
bokmaal_full = load_dataset("norne", "bokmaal", ner_tags_set="norne-full")
```
for example.

What do you think ?
2021-01-20T13:37:21Z
https://github.com/huggingface/datasets/pull/1720#issuecomment-763610489
https://api.github.com/repos/huggingface/datasets/issues/1720
MDEyOklzc3VlQ29tbWVudDc2MzYxMDQ4OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/763610489/reactions
2021-01-20T13:37:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/763610489
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Hi @versae have you had a chance to consider one of the two options for the config ?
I think both are ok but I have a small preference for the first one since it's simpler to implement.

Feel free to ping me if you have questions or if I can help :) 
2021-01-29T09:58:10Z
https://github.com/huggingface/datasets/pull/1720#issuecomment-769704649
https://api.github.com/repos/huggingface/datasets/issues/1720
MDEyOklzc3VlQ29tbWVudDc2OTcwNDY0OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/769704649/reactions
2021-01-29T09:58:10Z
https://api.github.com/repos/huggingface/datasets/issues/comments/769704649
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Hi @lhoestq. Agree, option 1 seems easier to implement. Just haven't had bandwidth to get to it yet. Hopefully starting next week I'll be able to update the PR.
2021-01-29T21:59:56Z
https://github.com/huggingface/datasets/pull/1720#issuecomment-770073119
https://api.github.com/repos/huggingface/datasets/issues/1720
MDEyOklzc3VlQ29tbWVudDc3MDA3MzExOQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/770073119/reactions
2021-01-29T21:59:56Z
https://api.github.com/repos/huggingface/datasets/issues/comments/770073119
https://avatars.githubusercontent.com/u/173537?v=4
https://api.github.com/users/versae/events{/privacy}
https://api.github.com/users/versae/followers
https://api.github.com/users/versae/following{/other_user}
https://api.github.com/users/versae/gists{/gist_id}
https://github.com/versae
versae
MDQ6VXNlcjE3MzUzNw==
https://api.github.com/users/versae/orgs
https://api.github.com/users/versae/received_events
https://api.github.com/users/versae/repos
https://api.github.com/users/versae/starred{/owner}{/repo}
https://api.github.com/users/versae/subscriptions
User
https://api.github.com/users/versae
public
MEMBER
Hi @versae ! Did you manage to add the configurations ? Let me know if we can help you on this
2021-03-29T12:45:41Z
https://github.com/huggingface/datasets/pull/1720#issuecomment-809347742
https://api.github.com/repos/huggingface/datasets/issues/1720
MDEyOklzc3VlQ29tbWVudDgwOTM0Nzc0Mg==
https://api.github.com/repos/huggingface/datasets/issues/comments/809347742/reactions
2021-03-29T12:45:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/809347742
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Hi @lhoestq, I do actually have to code ready, just need to generate the dummy data for it. 
2021-03-29T13:42:03Z
https://github.com/huggingface/datasets/pull/1720#issuecomment-809386664
https://api.github.com/repos/huggingface/datasets/issues/1720
MDEyOklzc3VlQ29tbWVudDgwOTM4NjY2NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/809386664/reactions
2021-03-29T13:42:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/809386664
https://avatars.githubusercontent.com/u/173537?v=4
https://api.github.com/users/versae/events{/privacy}
https://api.github.com/users/versae/followers
https://api.github.com/users/versae/following{/other_user}
https://api.github.com/users/versae/gists{/gist_id}
https://github.com/versae
versae
MDQ6VXNlcjE3MzUzNw==
https://api.github.com/users/versae/orgs
https://api.github.com/users/versae/received_events
https://api.github.com/users/versae/repos
https://api.github.com/users/versae/starred{/owner}{/repo}
https://api.github.com/users/versae/subscriptions
User
https://api.github.com/users/versae
public
CONTRIBUTOR
One thing I don't know how to do is to make `_info(self)` return the different NER tags in its `DatasetInfo` object depending on the specific config.
2021-03-29T13:44:37Z
https://github.com/huggingface/datasets/pull/1720#issuecomment-809388430
https://api.github.com/repos/huggingface/datasets/issues/1720
MDEyOklzc3VlQ29tbWVudDgwOTM4ODQzMA==
https://api.github.com/repos/huggingface/datasets/issues/comments/809388430/reactions
2021-03-29T13:44:37Z
https://api.github.com/repos/huggingface/datasets/issues/comments/809388430
https://avatars.githubusercontent.com/u/173537?v=4
https://api.github.com/users/versae/events{/privacy}
https://api.github.com/users/versae/followers
https://api.github.com/users/versae/following{/other_user}
https://api.github.com/users/versae/gists{/gist_id}
https://github.com/versae
versae
MDQ6VXNlcjE3MzUzNw==
https://api.github.com/users/versae/orgs
https://api.github.com/users/versae/received_events
https://api.github.com/users/versae/repos
https://api.github.com/users/versae/starred{/owner}{/repo}
https://api.github.com/users/versae/subscriptions
User
https://api.github.com/users/versae
public
CONTRIBUTOR
OK, I think it's ready now.
2021-03-30T15:12:53Z
https://github.com/huggingface/datasets/pull/1720#issuecomment-810343726
https://api.github.com/repos/huggingface/datasets/issues/1720
MDEyOklzc3VlQ29tbWVudDgxMDM0MzcyNg==
https://api.github.com/repos/huggingface/datasets/issues/comments/810343726/reactions
2021-03-30T15:12:53Z
https://api.github.com/repos/huggingface/datasets/issues/comments/810343726
https://avatars.githubusercontent.com/u/173537?v=4
https://api.github.com/users/versae/events{/privacy}
https://api.github.com/users/versae/followers
https://api.github.com/users/versae/following{/other_user}
https://api.github.com/users/versae/gists{/gist_id}
https://github.com/versae
versae
MDQ6VXNlcjE3MzUzNw==
https://api.github.com/users/versae/orgs
https://api.github.com/users/versae/received_events
https://api.github.com/users/versae/repos
https://api.github.com/users/versae/starred{/owner}{/repo}
https://api.github.com/users/versae/subscriptions
User
https://api.github.com/users/versae
public
CONTRIBUTOR
Closing this one and opening a new one with a cleaner commit log.
2021-03-31T14:13:17Z
https://github.com/huggingface/datasets/pull/1720#issuecomment-811101120
https://api.github.com/repos/huggingface/datasets/issues/1720
MDEyOklzc3VlQ29tbWVudDgxMTEwMTEyMA==
https://api.github.com/repos/huggingface/datasets/issues/comments/811101120/reactions
2021-03-31T14:13:17Z
https://api.github.com/repos/huggingface/datasets/issues/comments/811101120
https://avatars.githubusercontent.com/u/173537?v=4
https://api.github.com/users/versae/events{/privacy}
https://api.github.com/users/versae/followers
https://api.github.com/users/versae/following{/other_user}
https://api.github.com/users/versae/gists{/gist_id}
https://github.com/versae
versae
MDQ6VXNlcjE3MzUzNw==
https://api.github.com/users/versae/orgs
https://api.github.com/users/versae/received_events
https://api.github.com/users/versae/repos
https://api.github.com/users/versae/starred{/owner}{/repo}
https://api.github.com/users/versae/subscriptions
User
https://api.github.com/users/versae
public
CONTRIBUTOR
All set now in #2154.
2021-03-31T14:23:48Z
https://github.com/huggingface/datasets/pull/1720#issuecomment-811108785
https://api.github.com/repos/huggingface/datasets/issues/1720
MDEyOklzc3VlQ29tbWVudDgxMTEwODc4NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/811108785/reactions
2021-03-31T14:23:48Z
https://api.github.com/repos/huggingface/datasets/issues/comments/811108785
https://avatars.githubusercontent.com/u/173537?v=4
https://api.github.com/users/versae/events{/privacy}
https://api.github.com/users/versae/followers
https://api.github.com/users/versae/following{/other_user}
https://api.github.com/users/versae/gists{/gist_id}
https://github.com/versae
versae
MDQ6VXNlcjE3MzUzNw==
https://api.github.com/users/versae/orgs
https://api.github.com/users/versae/received_events
https://api.github.com/users/versae/repos
https://api.github.com/users/versae/starred{/owner}{/repo}
https://api.github.com/users/versae/subscriptions
User
https://api.github.com/users/versae
public
NorNE is a manually annotated corpus of named entities which extends the annotation of the existing Norwegian Dependency Treebank. Comprising both of the official standards of written Norwegian (Bokmål and Nynorsk), the corpus contains around 600,000 tokens and annotates a rich set of entity types including persons, organizations, locations, geo-political entities, products, and events, in addition to a class corresponding to nominals derived from names.