Allow model labels to be passed during task preparation
COLLABORATOR
We now have the `align_labels_with_mapping` method in the API for this purpose.
2022-10-05T17:37:22Z
https://github.com/huggingface/datasets/issues/2359#issuecomment-1268734373
https://api.github.com/repos/huggingface/datasets/issues/2359
IC_kwDODunzps5Ln1ml
https://api.github.com/repos/huggingface/datasets/issues/comments/1268734373/reactions
2022-10-05T17:37:22Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1268734373
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
Models have a config with label2id. And we have the same for datasets with the ClassLabel feature type. At one point either the model or the dataset must sync with the other. It would be great to do that on the dataset side.

For example for sentiment classification on amazon reviews with you could have these labels:
- "1 star", "2 stars", "3 stars", "4 stars", "5 stars"
- "1", "2", "3", "4", "5"

Some models may use the first set, while other models use the second set.

Here in the `TextClassification` class, the user can only specify one set of labels, while many models could actually be compatible but have different sets of labels. Should we allow users to pass a list of compatible labels sets ?

Then in terms of API, users could use `dataset.prepare_for_task("text-classification", labels=model.labels)` or something like that.

The label set could also be the same but not in the same order. For NLI for example, some models use `["neutral", "entailment", "contradiction"]` and some others use `["neutral", "contradiction", "entailment"]`, so we should take care of updating the order of the labels in the dataset to match the labels order of the model.

Let me know what you think ! This can be done in a future PR

_Originally posted by @lhoestq in https://github.com/huggingface/datasets/pull/2255#discussion_r632412792_