The size of CoNLL-2003 is not consistant with the official release.
COLLABORATOR
Hi,

if you inspect the raw data, you can find there are 946 occurrences of `-DOCSTART- -X- -X- O` in the train split and `14041 + 946 = 14987`, which is exactly the number of sentences the authors report. `-DOCSTART-` is a special line that acts as a boundary between two different documents and is filtered out in our implementation.

@lhoestq What do you think about including these lines? ([Link](https://github.com/flairNLP/flair/issues/1097) to a similar issue in the flairNLP repo)
2021-03-04T16:10:07Z
https://github.com/huggingface/datasets/issues/1983#issuecomment-790731836
https://api.github.com/repos/huggingface/datasets/issues/1983
MDEyOklzc3VlQ29tbWVudDc5MDczMTgzNg==
https://api.github.com/repos/huggingface/datasets/issues/comments/790731836/reactions
2021-03-04T16:10:07Z
https://api.github.com/repos/huggingface/datasets/issues/comments/790731836
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
MEMBER
We should mention in the Conll2003 dataset card that these lines have been removed indeed.

If some users are interested in using these lines (maybe to recombine documents ?) then we can add a parameter to the conll2003 dataset to include them.

But IMO the default config should stay the current one (without the `-DOCSTART-` stuff), so that you can directly train NER models without additional preprocessing. Let me know what you think
2021-03-04T17:21:50Z
https://github.com/huggingface/datasets/issues/1983#issuecomment-790786409
https://api.github.com/repos/huggingface/datasets/issues/1983
MDEyOklzc3VlQ29tbWVudDc5MDc4NjQwOQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/790786409/reactions
2021-03-04T17:22:19Z
https://api.github.com/repos/huggingface/datasets/issues/comments/790786409
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
COLLABORATOR
@lhoestq Yes, I agree adding a small note should be sufficient.

Currently, NLTK's `ConllCorpusReader` ignores the `-DOCSTART-` lines so I think it's ok if we do the same. If there is an interest in the future to use these lines, then we can include them.
2021-03-05T13:11:21Z
https://github.com/huggingface/datasets/issues/1983#issuecomment-791410379
https://api.github.com/repos/huggingface/datasets/issues/1983
MDEyOklzc3VlQ29tbWVudDc5MTQxMDM3OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/791410379/reactions
2021-03-05T13:11:21Z
https://api.github.com/repos/huggingface/datasets/issues/comments/791410379
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
MEMBER
I added a mention of this in conll2003's dataset card:
https://github.com/huggingface/datasets/blob/fc9796920da88486c3b97690969aabf03d6b4088/datasets/conll2003/README.md#conll2003

Edit: just saw your PR @mariosasko (noticed it too late ^^)
Let me take a look at it :)
2021-03-08T16:23:32Z
https://github.com/huggingface/datasets/issues/1983#issuecomment-792874213
https://api.github.com/repos/huggingface/datasets/issues/1983
MDEyOklzc3VlQ29tbWVudDc5Mjg3NDIxMw==
https://api.github.com/repos/huggingface/datasets/issues/comments/792874213/reactions
2021-03-08T16:24:25Z
https://api.github.com/repos/huggingface/datasets/issues/comments/792874213
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
Thanks for the dataset sharing! But when I use conll-2003, I meet some questions.
The statistics of conll-2003 in this repo is  : 
\#train 14041  \#dev 3250 \#test 3453
While the official statistics is:
\#train 14987 \#dev 3466 \#test 3684
Wish for your reply~