Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: SpeechMoE: Scaling to Large Acoustic Models with Dynamic Routing Mixture
  of Experts   Recently, Mixture of Experts (MoE) based Transformer has shown promising
results in many domains. This is largely due to the following advantages of
this architecture: firstly, MoE based Transformer can increase model capacity
without computational cost increasing both at training and inference time.
Besides, MoE based Transformer is a dynamic network which can adapt to the
varying complexity of input instances in realworld applications. In this work,
we explore the MoE based model for speech recognition, named SpeechMoE. To
further control the sparsity of router activation and improve the diversity of
gate values, we propose a sparsity L1 loss and a mean importance loss
respectively. In addition, a new router architecture is used in SpeechMoE which
can simultaneously utilize the information from a shared embedding network and
the hierarchical representation of different MoE layers. Experimental results
show that SpeechMoE can achieve lower character error rate (CER) with
comparable computation cost than traditional static networks, providing
7.0%-23.0% relative CER improvements on four evaluation datasets.

10
The categories of this paper are: q-bio
8
The categories of this paper are: hep-ex
36
The categories of this paper are: cmp-lg
18
The categories of this paper are: econ
12
The categories of this paper are: nucl-th
17
The categories of this paper are: q-fin
26
The categories of this paper are: bayes-an
15
The categories of this paper are: nucl-ex
5
The categories of this paper are: gr-qc
34
The categories of this paper are: chem-ph
37
The categories of this paper are: supr-con
11
The categories of this paper are: hep-lat
0
The categories of this paper are: hep-ph
2
The categories of this paper are: math
9
The categories of this paper are: nlin
4
The categories of this paper are: cond-mat