Token classification labels are strings and we don't have the list of labels
MEMBER
Indeed. Pinging @stefan-it here if he want to give an expert opinion :)
2020-11-03T21:05:42Z
https://github.com/huggingface/datasets/issues/797#issuecomment-721374004
https://api.github.com/repos/huggingface/datasets/issues/797
MDEyOklzc3VlQ29tbWVudDcyMTM3NDAwNA==
https://api.github.com/repos/huggingface/datasets/issues/comments/721374004/reactions
2020-11-03T21:05:42Z
https://api.github.com/repos/huggingface/datasets/issues/comments/721374004
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
MEMBER
Related is https://github.com/huggingface/datasets/pull/636
2020-11-04T08:23:43Z
https://github.com/huggingface/datasets/issues/797#issuecomment-721584573
https://api.github.com/repos/huggingface/datasets/issues/797
MDEyOklzc3VlQ29tbWVudDcyMTU4NDU3Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/721584573/reactions
2020-11-04T08:23:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/721584573
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
CONTRIBUTOR
Should definitely be a ClassLabel 👍 
2020-11-04T11:25:04Z
https://github.com/huggingface/datasets/issues/797#issuecomment-721676053
https://api.github.com/repos/huggingface/datasets/issues/797
MDEyOklzc3VlQ29tbWVudDcyMTY3NjA1Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/721676053/reactions
2020-11-04T11:25:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/721676053
https://avatars.githubusercontent.com/u/20651387?v=4
https://api.github.com/users/stefan-it/events{/privacy}
https://api.github.com/users/stefan-it/followers
https://api.github.com/users/stefan-it/following{/other_user}
https://api.github.com/users/stefan-it/gists{/gist_id}
https://github.com/stefan-it
stefan-it
MDQ6VXNlcjIwNjUxMzg3
https://api.github.com/users/stefan-it/orgs
https://api.github.com/users/stefan-it/received_events
https://api.github.com/users/stefan-it/repos
https://api.github.com/users/stefan-it/starred{/owner}{/repo}
https://api.github.com/users/stefan-it/subscriptions
User
https://api.github.com/users/stefan-it
public
MEMBER
Already done.
2022-02-14T15:41:53Z
https://github.com/huggingface/datasets/issues/797#issuecomment-1039237873
https://api.github.com/repos/huggingface/datasets/issues/797
IC_kwDODunzps498YLx
https://api.github.com/repos/huggingface/datasets/issues/comments/1039237873/reactions
2022-02-14T15:41:53Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1039237873
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
Not sure if this is an issue we want to fix or not, putting it here so it's not forgotten. Right now, in token classification datasets, the labels for NER, POS and the likes are typed as `Sequence` of `strings`, which is wrong in my opinion. These should be `Sequence` of `ClassLabel` or some types that gives easy access to the underlying labels.

The main problem for preprocessing those datasets is that the list of possible labels is not stored inside the `Dataset` object which makes converting the labels to IDs quite difficult (you either have to know the list of labels in advance or run a full pass through the dataset to get the list of labels, the `unique` method being useless with the type `Sequence[str]`).