Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Fixed Encoder Self-Attention Patterns in Transformer-Based Machine
  Translation   Transformer-based models have brought a radical change to neural machine
translation. A key feature of the Transformer architecture is the so-called
multi-head attention mechanism, which allows the model to focus simultaneously
on different parts of the input. However, recent works have shown that most
attention heads learn simple, and often redundant, positional patterns. In this
paper, we propose to replace all but one attention head of each encoder layer
with simple fixed -- non-learnable -- attentive patterns that are solely based
on position and do not require any external knowledge. Our experiments with
different data sizes and multiple language pairs show that fixing the attention
heads on the encoder side of the Transformer at training time does not impact
the translation quality and even increases BLEU scores by up to 3 points in
low-resource scenarios.

26
The categories of this paper are: bayes-an
0
The categories of this paper are: hep-ph
14
The categories of this paper are: math-ph
29
The categories of this paper are: funct-an
31
The categories of this paper are: ao-sci
33
The categories of this paper are: plasm-ph
5
The categories of this paper are: gr-qc
36
The categories of this paper are: cmp-lg
13
The categories of this paper are: quant-ph
23
The categories of this paper are: solv-int
9
The categories of this paper are: nlin
22
The categories of this paper are: chao-dyn
32
The categories of this paper are: atom-ph
30
The categories of this paper are: q-alg
12
The categories of this paper are: nucl-th
27
The categories of this paper are: comp-gas