Add align_labels_with_mapping function
MEMBER
@lhoestq i think this is ready for another review 🙂 
2021-06-10T09:32:22Z
https://github.com/huggingface/datasets/pull/2457#issuecomment-858468465
https://api.github.com/repos/huggingface/datasets/issues/2457
MDEyOklzc3VlQ29tbWVudDg1ODQ2ODQ2NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/858468465/reactions
2021-06-10T09:32:22Z
https://api.github.com/repos/huggingface/datasets/issues/comments/858468465
https://avatars.githubusercontent.com/u/26859204?v=4
https://api.github.com/users/lewtun/events{/privacy}
https://api.github.com/users/lewtun/followers
https://api.github.com/users/lewtun/following{/other_user}
https://api.github.com/users/lewtun/gists{/gist_id}
https://github.com/lewtun
lewtun
MDQ6VXNlcjI2ODU5MjA0
https://api.github.com/users/lewtun/orgs
https://api.github.com/users/lewtun/received_events
https://api.github.com/users/lewtun/repos
https://api.github.com/users/lewtun/starred{/owner}{/repo}
https://api.github.com/users/lewtun/subscriptions
User
https://api.github.com/users/lewtun
public
MEMBER
@lhoestq thanks for the feedback - it's now integrated :) 

i also added a comment about sorting the input label IDs
2021-06-17T07:41:39Z
https://github.com/huggingface/datasets/pull/2457#issuecomment-863009131
https://api.github.com/repos/huggingface/datasets/issues/2457
MDEyOklzc3VlQ29tbWVudDg2MzAwOTEzMQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/863009131/reactions
2021-06-17T07:41:39Z
https://api.github.com/repos/huggingface/datasets/issues/comments/863009131
https://avatars.githubusercontent.com/u/26859204?v=4
https://api.github.com/users/lewtun/events{/privacy}
https://api.github.com/users/lewtun/followers
https://api.github.com/users/lewtun/following{/other_user}
https://api.github.com/users/lewtun/gists{/gist_id}
https://github.com/lewtun
lewtun
MDQ6VXNlcjI2ODU5MjA0
https://api.github.com/users/lewtun/orgs
https://api.github.com/users/lewtun/received_events
https://api.github.com/users/lewtun/repos
https://api.github.com/users/lewtun/starred{/owner}{/repo}
https://api.github.com/users/lewtun/subscriptions
User
https://api.github.com/users/lewtun
public
MEMBER
Created the PR here: https://github.com/huggingface/datasets/pull/2510
2021-06-17T10:03:48Z
https://github.com/huggingface/datasets/pull/2457#issuecomment-863107996
https://api.github.com/repos/huggingface/datasets/issues/2457
MDEyOklzc3VlQ29tbWVudDg2MzEwNzk5Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/863107996/reactions
2021-06-17T10:03:48Z
https://api.github.com/repos/huggingface/datasets/issues/comments/863107996
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
> Thanks ! Looks all good now :)
> 
> We will also need to have the `DatasetDict.align_labels_with_mapping` method. Let me quickly add it

thanks a lot! i always forget about `DatasetDict` - will be happy when it's just one "dataset" object :)
2021-06-17T10:17:03Z
https://github.com/huggingface/datasets/pull/2457#issuecomment-863117256
https://api.github.com/repos/huggingface/datasets/issues/2457
MDEyOklzc3VlQ29tbWVudDg2MzExNzI1Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/863117256/reactions
2021-06-17T10:17:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/863117256
https://avatars.githubusercontent.com/u/26859204?v=4
https://api.github.com/users/lewtun/events{/privacy}
https://api.github.com/users/lewtun/followers
https://api.github.com/users/lewtun/following{/other_user}
https://api.github.com/users/lewtun/gists{/gist_id}
https://github.com/lewtun
lewtun
MDQ6VXNlcjI2ODU5MjA0
https://api.github.com/users/lewtun/orgs
https://api.github.com/users/lewtun/received_events
https://api.github.com/users/lewtun/repos
https://api.github.com/users/lewtun/starred{/owner}{/repo}
https://api.github.com/users/lewtun/subscriptions
User
https://api.github.com/users/lewtun
public
NONE
So, there seems to be a problem with the function align_labels_with_mapping for models like this: https://huggingface.co/huggingface/distilbert-base-uncased-finetuned-mnli]. At least with this model, but perhaps also with others, the model.config.label2id values are of type str not int, which crashes said function. After manually converting the model.config.label2id values to int, the script runs smoothly.


2022-01-12T08:57:41Z
https://github.com/huggingface/datasets/pull/2457#issuecomment-1010800087
https://api.github.com/repos/huggingface/datasets/issues/2457
IC_kwDODunzps48P5XX
https://api.github.com/repos/huggingface/datasets/issues/comments/1010800087/reactions
2022-01-12T08:57:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1010800087
https://avatars.githubusercontent.com/u/60972711?v=4
https://api.github.com/users/retoj/events{/privacy}
https://api.github.com/users/retoj/followers
https://api.github.com/users/retoj/following{/other_user}
https://api.github.com/users/retoj/gists{/gist_id}
https://github.com/retoj
retoj
MDQ6VXNlcjYwOTcyNzEx
https://api.github.com/users/retoj/orgs
https://api.github.com/users/retoj/received_events
https://api.github.com/users/retoj/repos
https://api.github.com/users/retoj/starred{/owner}{/repo}
https://api.github.com/users/retoj/subscriptions
User
https://api.github.com/users/retoj
public
This PR adds a helper function to align the `label2id` mapping between a `datasets.Dataset` and a classifier (e.g. a transformer with a `PretrainedConfig.label2id` dict), with the alignment performed on the dataset itself.

This will help us with the Hub evaluation, where we won't know in advance whether a model that is fine-tuned on say MNLI has the same mappings as the MNLI dataset we load from `datasets`.

An example where this is needed is if we naively try to evaluate `microsoft/deberta-base-mnli` on `mnli` because the model config has the following mappings:

```python
  "id2label": {
    "0": "CONTRADICTION",
    "1": "NEUTRAL",
    "2": "ENTAILMENT"
  },
  "label2id": {
    "CONTRADICTION": 0,
    "ENTAILMENT": 2,
    "NEUTRAL": 1
  }
```

while the `mnli` dataset has the `contradiction` and `neutral` labels swapped:

```python
id2label = {0: 'entailment', 1: 'neutral', 2: 'contradiction'}
label2id = {'contradiction': 2, 'entailment': 0, 'neutral': 1}
```

As a result, we get a much lower accuracy during evaluation:

```python
from datasets import load_dataset
from transformers.trainer_utils import EvalPrediction
from transformers import AutoModelForSequenceClassification, Trainer

# load dataset for evaluation
mnli = load_dataset("glue", "mnli", split="test")
# load model
model_ckpt = "microsoft/deberta-base-mnli"
model = AutoModelForSequenceClassification.from_pretrained(checkpoint)
# preprocess, create trainer ...
mnli_enc = ...
trainer = Trainer(model, args=args, tokenizer=tokenizer)
# generate preds
preds = trainer.predict(mnli_enc)
# preds.label_ids misalinged with model.config => returns wrong accuracy (too low)!
compute_metrics(EvalPrediction(preds.predictions, preds.label_ids))
```

The fix is to use the helper function before running the evaluation to make sure the label IDs are aligned:

```python
mnli_enc_aligned = mnli_enc.align_labels_with_mapping(label2id=config.label2id, label_column="label")
# preds now aligned and everyone is happy :)
preds = trainer.predict(mnli_enc_aligned)
```

cc @thomwolf @lhoestq 