Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Improving Generalization of Transformer for Speech Recognition with
  Parallel Schedule Sampling and Relative Positional Embedding   Transformer has shown promising results in many sequence to sequence
transformation tasks recently. It utilizes a number of feed-forward
self-attention layers to replace the recurrent neural networks (RNN) in
attention-based encoder decoder (AED) architecture. Self-attention layer learns
temporal dependence by incorporating sinusoidal positional embedding of tokens
in a sequence for parallel computing. Quicker iteration speed in training than
sequential operation of RNN can be obtained. Deeper layers of the transformer
also make it perform better than RNN-based AED. However, this parallelization
ability is lost when applying scheduled sampling training. Self-attention with
sinusoidal positional embedding may cause performance degradations for longer
sequences that have similar acoustic or semantic information at different
positions as well. To address these problems, we propose to use parallel
scheduled sampling (PSS) and relative positional embedding (RPE) to help the
transformer generalize to unseen data. Our proposed methods achieve a 7%
relative improvement for short utterances and a 70% relative gain for long
utterances on a 10,000-hour Mandarin ASR task.

21
The categories of this paper are: adap-org
12
The categories of this paper are: nucl-th
29
The categories of this paper are: funct-an
17
The categories of this paper are: q-fin
15
The categories of this paper are: nucl-ex
14
The categories of this paper are: math-ph
37
The categories of this paper are: supr-con
9
The categories of this paper are: nlin
20
The categories of this paper are: acc-phys
33
The categories of this paper are: plasm-ph
6
The categories of this paper are: astro-ph
25
The categories of this paper are: patt-sol
5
The categories of this paper are: gr-qc
30
The categories of this paper are: q-alg
2
The categories of this paper are: math
34
The categories of this paper are: chem-ph