Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Don't Use English Dev: On the Zero-Shot Cross-Lingual Evaluation of
  Contextual Embeddings   Multilingual contextual embeddings have demonstrated state-of-the-art
performance in zero-shot cross-lingual transfer learning, where multilingual
BERT is fine-tuned on one source language and evaluated on a different target
language. However, published results for mBERT zero-shot accuracy vary as much
as 17 points on the MLDoc classification task across four papers. We show that
the standard practice of using English dev accuracy for model selection in the
zero-shot setting makes it difficult to obtain reproducible results on the
MLDoc and XNLI tasks. English dev accuracy is often uncorrelated (or even
anti-correlated) with target language accuracy, and zero-shot performance
varies greatly at different points in the same fine-tuning run and between
different fine-tuning runs. These reproducibility issues are also present for
other tasks with different pre-trained embeddings (e.g., MLQA with XLM-R). We
recommend providing oracle scores alongside zero-shot results: still fine-tune
using English data, but choose a checkpoint with the target dev set. Reporting
this upper bound makes results more consistent by avoiding arbitrarily bad
checkpoints.

29
The categories of this paper are: funct-an
10
The categories of this paper are: q-bio
26
The categories of this paper are: bayes-an
27
The categories of this paper are: comp-gas
36
The categories of this paper are: cmp-lg
19
The categories of this paper are: eess
11
The categories of this paper are: hep-lat
31
The categories of this paper are: ao-sci
8
The categories of this paper are: hep-ex
34
The categories of this paper are: chem-ph
18
The categories of this paper are: econ
9
The categories of this paper are: nlin
28
The categories of this paper are: alg-geom
37
The categories of this paper are: supr-con
35
The categories of this paper are: mtrl-th
14
The categories of this paper are: math-ph