Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Multi-head or Single-head? An Empirical Comparison for Transformer
  Training   Multi-head attention plays a crucial role in the recent success of
Transformer models, which leads to consistent performance improvements over
conventional attention in various applications. The popular belief is that this
effectiveness stems from the ability of jointly attending multiple positions.
In this paper, we first demonstrate that jointly attending multiple positions
is not a unique feature of multi-head attention, as multi-layer single-head
attention also attends multiple positions and is more effective. Then, we
suggest the main advantage of the multi-head attention is the training
stability, since it has less number of layers than the single-head attention,
when attending the same number of positions. For example, 24-layer 16-head
Transformer (BERT-large) and 384-layer single-head Transformer has the same
total attention head number and roughly the same model size, while the
multi-head one is significantly shallower. Meanwhile, we show that, with recent
advances in deep learning, we can successfully stabilize the training of the
384-layer Transformer. As the training difficulty is no longer a bottleneck,
substantially deeper single-head Transformer achieves consistent performance
improvements without tuning hyper-parameters.

13
The categories of this paper are: quant-ph
28
The categories of this paper are: alg-geom
16
The categories of this paper are: stat
5
The categories of this paper are: gr-qc
33
The categories of this paper are: plasm-ph
37
The categories of this paper are: supr-con
30
The categories of this paper are: q-alg
36
The categories of this paper are: cmp-lg
17
The categories of this paper are: q-fin
19
The categories of this paper are: eess
8
The categories of this paper are: hep-ex
2
The categories of this paper are: math
18
The categories of this paper are: econ
25
The categories of this paper are: patt-sol
34
The categories of this paper are: chem-ph
4
The categories of this paper are: cond-mat