Error when calculating glue score
MEMBER
The glue metric for cola is a metric for classification. It expects label ids as integers as inputs.
2020-06-30T16:13:06Z
https://github.com/huggingface/datasets/issues/324#issuecomment-651896268
https://api.github.com/repos/huggingface/datasets/issues/324
MDEyOklzc3VlQ29tbWVudDY1MTg5NjI2OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/651896268/reactions
2020-06-30T16:13:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/651896268
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
I want to evaluate a sentence pair whether they are semantically equivalent, so I used MRPC and it gives the same error, does that mean we have to encode the sentences and parse as input?

using BertTokenizer;
```
encoded_reference=tokenizer.encode(reference, add_special_tokens=False)
encoded_prediction=tokenizer.encode(prediction, add_special_tokens=False)
```

`glue_score = glue_metric.compute(encoded_prediction, encoded_reference)`
```

ValueError                                Traceback (most recent call last)
<ipython-input-9-4c3a3ce7b583> in <module>()
----> 1 glue_score = glue_metric.compute(encoded_prediction, encoded_reference)

6 frames
/usr/local/lib/python3.6/dist-packages/nlp/metric.py in compute(self, predictions, references, timeout, **metrics_kwargs)
    198         predictions = self.data["predictions"]
    199         references = self.data["references"]
--> 200         output = self._compute(predictions=predictions, references=references, **metrics_kwargs)
    201         return output
    202 

/usr/local/lib/python3.6/dist-packages/nlp/metrics/glue/27b1bc63e520833054bd0d7a8d0bc7f6aab84cc9eed1b576e98c806f9466d302/glue.py in _compute(self, predictions, references)
    101             return pearson_and_spearman(predictions, references)
    102         elif self.config_name in ["mrpc", "qqp"]:
--> 103             return acc_and_f1(predictions, references)
    104         elif self.config_name in ["sst2", "mnli", "mnli_mismatched", "mnli_matched", "qnli", "rte", "wnli", "hans"]:
    105             return {"accuracy": simple_accuracy(predictions, references)}

/usr/local/lib/python3.6/dist-packages/nlp/metrics/glue/27b1bc63e520833054bd0d7a8d0bc7f6aab84cc9eed1b576e98c806f9466d302/glue.py in acc_and_f1(preds, labels)
     60 def acc_and_f1(preds, labels):
     61     acc = simple_accuracy(preds, labels)
---> 62     f1 = f1_score(y_true=labels, y_pred=preds)
     63     return {
     64         "accuracy": acc,

/usr/local/lib/python3.6/dist-packages/sklearn/metrics/_classification.py in f1_score(y_true, y_pred, labels, pos_label, average, sample_weight, zero_division)
   1097                        pos_label=pos_label, average=average,
   1098                        sample_weight=sample_weight,
-> 1099                        zero_division=zero_division)
   1100 
   1101 

/usr/local/lib/python3.6/dist-packages/sklearn/metrics/_classification.py in fbeta_score(y_true, y_pred, beta, labels, pos_label, average, sample_weight, zero_division)
   1224                                                  warn_for=('f-score',),
   1225                                                  sample_weight=sample_weight,
-> 1226                                                  zero_division=zero_division)
   1227     return f
   1228 

/usr/local/lib/python3.6/dist-packages/sklearn/metrics/_classification.py in precision_recall_fscore_support(y_true, y_pred, beta, labels, pos_label, average, warn_for, sample_weight, zero_division)
   1482         raise ValueError("beta should be >=0 in the F-beta score")
   1483     labels = _check_set_wise_labels(y_true, y_pred, average, labels,
-> 1484                                     pos_label)
   1485 
   1486     # Calculate tp_sum, pred_sum, true_sum ###

/usr/local/lib/python3.6/dist-packages/sklearn/metrics/_classification.py in _check_set_wise_labels(y_true, y_pred, average, labels, pos_label)
   1314             raise ValueError("Target is %s but average='binary'. Please "
   1315                              "choose another average setting, one of %r."
-> 1316                              % (y_type, average_options))
   1317     elif pos_label not in (None, 1):
   1318         warnings.warn("Note that pos_label (set to %r) is ignored when "

ValueError: Target is multiclass but average='binary'. Please choose another average setting, one of [None, 'micro', 'macro', 'weighted'].

```
2020-07-01T04:48:38Z
https://github.com/huggingface/datasets/issues/324#issuecomment-652187906
https://api.github.com/repos/huggingface/datasets/issues/324
MDEyOklzc3VlQ29tbWVudDY1MjE4NzkwNg==
https://api.github.com/repos/huggingface/datasets/issues/comments/652187906/reactions
2020-07-01T05:11:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/652187906
https://avatars.githubusercontent.com/u/47185867?v=4
https://api.github.com/users/D-i-l-r-u-k-s-h-i/events{/privacy}
https://api.github.com/users/D-i-l-r-u-k-s-h-i/followers
https://api.github.com/users/D-i-l-r-u-k-s-h-i/following{/other_user}
https://api.github.com/users/D-i-l-r-u-k-s-h-i/gists{/gist_id}
https://github.com/D-i-l-r-u-k-s-h-i
D-i-l-r-u-k-s-h-i
MDQ6VXNlcjQ3MTg1ODY3
https://api.github.com/users/D-i-l-r-u-k-s-h-i/orgs
https://api.github.com/users/D-i-l-r-u-k-s-h-i/received_events
https://api.github.com/users/D-i-l-r-u-k-s-h-i/repos
https://api.github.com/users/D-i-l-r-u-k-s-h-i/starred{/owner}{/repo}
https://api.github.com/users/D-i-l-r-u-k-s-h-i/subscriptions
User
https://api.github.com/users/D-i-l-r-u-k-s-h-i
public
MEMBER
MRPC is also a binary classification task, so its metric is a binary classification metric.

To evaluate if pairs of sentences are semantically equivalent, maybe you could take a look at models that compute if one sentence entails the other or not (typically the kinds of model that could work well on the MRPC task).
2020-07-02T14:18:24Z
https://github.com/huggingface/datasets/issues/324#issuecomment-653033370
https://api.github.com/repos/huggingface/datasets/issues/324
MDEyOklzc3VlQ29tbWVudDY1MzAzMzM3MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/653033370/reactions
2020-07-02T14:19:46Z
https://api.github.com/repos/huggingface/datasets/issues/comments/653033370
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Closing this one. Feel free to re-open if you have other questions :)
2020-07-09T09:13:34Z
https://github.com/huggingface/datasets/issues/324#issuecomment-656011778
https://api.github.com/repos/huggingface/datasets/issues/324
MDEyOklzc3VlQ29tbWVudDY1NjAxMTc3OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/656011778/reactions
2020-07-09T09:13:34Z
https://api.github.com/repos/huggingface/datasets/issues/comments/656011778
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
I was trying glue score along with other metrics here. But glue gives me this error;

```
import nlp
glue_metric = nlp.load_metric('glue',name="cola")

glue_score = glue_metric.compute(predictions, references)
```

```
---------------------------------------------------------------------------
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-8-b9210a524504> in <module>()
----> 1 glue_score = glue_metric.compute(predictions, references)

6 frames
/usr/local/lib/python3.6/dist-packages/nlp/metric.py in compute(self, predictions, references, timeout, **metrics_kwargs)
    191         """
    192         if predictions is not None:
--> 193             self.add_batch(predictions=predictions, references=references)
    194         self.finalize(timeout=timeout)
    195 

/usr/local/lib/python3.6/dist-packages/nlp/metric.py in add_batch(self, predictions, references, **kwargs)
    207         if self.writer is None:
    208             self._init_writer()
--> 209         self.writer.write_batch(batch)
    210 
    211     def add(self, prediction=None, reference=None, **kwargs):

/usr/local/lib/python3.6/dist-packages/nlp/arrow_writer.py in write_batch(self, batch_examples, writer_batch_size)
    155         if self.pa_writer is None:
    156             self._build_writer(pa_table=pa.Table.from_pydict(batch_examples))
--> 157         pa_table: pa.Table = pa.Table.from_pydict(batch_examples, schema=self._schema)
    158         if writer_batch_size is None:
    159             writer_batch_size = self.writer_batch_size

/usr/local/lib/python3.6/dist-packages/pyarrow/types.pxi in __iter__()

/usr/local/lib/python3.6/dist-packages/pyarrow/array.pxi in pyarrow.lib.asarray()

/usr/local/lib/python3.6/dist-packages/pyarrow/array.pxi in pyarrow.lib.array()

/usr/local/lib/python3.6/dist-packages/pyarrow/array.pxi in pyarrow.lib._sequence_to_array()

TypeError: an integer is required (got type str)
```
I'm not sure whether I'm doing this wrong or whether it's an issue. I would like to know a workaround. Thank you.