Sentence Boundaries missing in Dataset: xtreme / udpos
COLLABORATOR
Hi,

This is a known issue. More info on this issue can be found in #2061. If you are looking for an open-source contribution, there are step-by-step instructions in the linked issue that you can follow to fix it.
2021-06-05T18:03:01Z
https://github.com/huggingface/datasets/issues/2444#issuecomment-855274911
https://api.github.com/repos/huggingface/datasets/issues/2444
MDEyOklzc3VlQ29tbWVudDg1NTI3NDkxMQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/855274911/reactions
2021-06-05T18:03:01Z
https://api.github.com/repos/huggingface/datasets/issues/comments/855274911
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
MEMBER
Closed by #2466.
2021-06-18T11:53:43Z
https://github.com/huggingface/datasets/issues/2444#issuecomment-863982700
https://api.github.com/repos/huggingface/datasets/issues/2444
MDEyOklzc3VlQ29tbWVudDg2Mzk4MjcwMA==
https://api.github.com/repos/huggingface/datasets/issues/comments/863982700/reactions
2021-06-18T11:53:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/863982700
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
I was browsing through annotation guidelines, as suggested by the datasets introduction.

The guidlines saids "There must be exactly one blank line after every sentence, including the last sentence in the file. Empty sentences are not allowed." in the [Sentence Boundaries and Comments section](https://universaldependencies.org/format.html#sentence-boundaries-and-comments)

But the sentence boundaries seems not to be represented by huggingface datasets features well. I found out that multiple sentence are concatenated together as a 1D array, without any delimiter.

PAN-x, which is another token classification subset from xtreme do represent the sentence boundary using a 2D array.

You may compare in PAN-x.en and udpos.English in the explorer:
 https://huggingface.co/datasets/viewer/?dataset=xtreme