Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Multi-branch Attentive Transformer   While the multi-branch architecture is one of the key ingredients to the
success of computer vision tasks, it has not been well investigated in natural
language processing, especially sequence learning tasks. In this work, we
propose a simple yet effective variant of Transformer called multi-branch
attentive Transformer (briefly, MAT), where the attention layer is the average
of multiple branches and each branch is an independent multi-head attention
layer. We leverage two training techniques to regularize the training:
drop-branch, which randomly drops individual branches during training, and
proximal initialization, which uses a pre-trained Transformer model to
initialize multiple branches. Experiments on machine translation, code
generation and natural language understanding demonstrate that such a simple
variant of Transformer brings significant improvements. Our code is available
at \url{https://github.com/HA-Transformer}.

26
The categories of this paper are: bayes-an
3
The categories of this paper are: physics
35
The categories of this paper are: mtrl-th
24
The categories of this paper are: dg-ga
36
The categories of this paper are: cmp-lg
23
The categories of this paper are: solv-int
37
The categories of this paper are: supr-con
32
The categories of this paper are: atom-ph
6
The categories of this paper are: astro-ph
30
The categories of this paper are: q-alg
8
The categories of this paper are: hep-ex
11
The categories of this paper are: hep-lat
29
The categories of this paper are: funct-an
21
The categories of this paper are: adap-org
31
The categories of this paper are: ao-sci
14
The categories of this paper are: math-ph