Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Contributions of Transformer Attention Heads in Multi- and Cross-lingual
  Tasks   This paper studies the relative importance of attention heads in
Transformer-based models to aid their interpretability in cross-lingual and
multi-lingual tasks. Prior research has found that only a few attention heads
are important in each mono-lingual Natural Language Processing (NLP) task and
pruning the remaining heads leads to comparable or improved performance of the
model. However, the impact of pruning attention heads is not yet clear in
cross-lingual and multi-lingual tasks. Through extensive experiments, we show
that (1) pruning a number of attention heads in a multi-lingual
Transformer-based model has, in general, positive effects on its performance in
cross-lingual and multi-lingual tasks and (2) the attention heads to be pruned
can be ranked using gradients and identified with a few trial experiments. Our
experiments focus on sequence labeling tasks, with potential applicability on
other cross-lingual and multi-lingual tasks. For comprehensiveness, we examine
two pre-trained multi-lingual models, namely multi-lingual BERT (mBERT) and
XLM-R, on three tasks across 9 languages each. We also discuss the validity of
our findings and their extensibility to truly resource-scarce languages and
other task settings.

20
The categories of this paper are: acc-phys
14
The categories of this paper are: math-ph
37
The categories of this paper are: supr-con
17
The categories of this paper are: q-fin
6
The categories of this paper are: astro-ph
24
The categories of this paper are: dg-ga
31
The categories of this paper are: ao-sci
19
The categories of this paper are: eess
32
The categories of this paper are: atom-ph
27
The categories of this paper are: comp-gas
2
The categories of this paper are: math
11
The categories of this paper are: hep-lat
25
The categories of this paper are: patt-sol
26
The categories of this paper are: bayes-an
0
The categories of this paper are: hep-ph
5
The categories of this paper are: gr-qc