Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Probing Word Translations in the Transformer and Trading Decoder for
  Encoder Layers   Due to its effectiveness and performance, the Transformer translation model
has attracted wide attention, most recently in terms of probing-based
approaches. Previous work focuses on using or probing source linguistic
features in the encoder. To date, the way word translation evolves in
Transformer layers has not yet been investigated. Naively, one might assume
that encoder layers capture source information while decoder layers translate.
In this work, we show that this is not quite the case: translation already
happens progressively in encoder layers and even in the input embeddings. More
surprisingly, we find that some of the lower decoder layers do not actually do
that much decoding. We show all of this in terms of a probing approach where we
project representations of the layer analyzed to the final trained and frozen
classifier level of the Transformer decoder to measure word translation
accuracy. Our findings motivate and explain a Transformer configuration change:
if translation already happens in the encoder layers, perhaps we can increase
the number of encoder layers, while decreasing the number of decoder layers,
boosting decoding speed, without loss in translation quality? Our experiments
show that this is indeed the case: we can increase speed by up to a factor 2.3
with small gains in translation quality, while an 18-4 deep encoder
configuration boosts translation quality by +1.42 BLEU (En-De) at a speed-up of
1.4.

15
The categories of this paper are: nucl-ex
29
The categories of this paper are: funct-an
27
The categories of this paper are: comp-gas
19
The categories of this paper are: eess
3
The categories of this paper are: physics
24
The categories of this paper are: dg-ga
16
The categories of this paper are: stat
0
The categories of this paper are: hep-ph
2
The categories of this paper are: math
20
The categories of this paper are: acc-phys
23
The categories of this paper are: solv-int
6
The categories of this paper are: astro-ph
25
The categories of this paper are: patt-sol
28
The categories of this paper are: alg-geom
22
The categories of this paper are: chao-dyn
34
The categories of this paper are: chem-ph