Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Experiments on Parallel Training of Deep Neural Network using Model
  Averaging   In this work we apply model averaging to parallel training of deep neural
network (DNN). Parallelization is done in a model averaging manner. Data is
partitioned and distributed to different nodes for local model updates, and
model averaging across nodes is done every few minibatches. We use multiple
GPUs for data parallelization, and Message Passing Interface (MPI) for
communication between nodes, which allows us to perform model averaging
frequently without losing much time on communication. We investigate the
effectiveness of Natural Gradient Stochastic Gradient Descent (NG-SGD) and
Restricted Boltzmann Machine (RBM) pretraining for parallel training in
model-averaging framework, and explore the best setups in term of different
learning rate schedules, averaging frequencies and minibatch sizes. It is shown
that NG-SGD and RBM pretraining benefits parameter-averaging based model
training. On the 300h Switchboard dataset, a 9.3 times speedup is achieved
using 16 GPUs and 17 times speedup using 32 GPUs with limited decoding accuracy
loss.

32
The categories of this paper are: atom-ph
36
The categories of this paper are: cmp-lg
26
The categories of this paper are: bayes-an
34
The categories of this paper are: chem-ph
10
The categories of this paper are: q-bio
12
The categories of this paper are: nucl-th
17
The categories of this paper are: q-fin
9
The categories of this paper are: nlin
14
The categories of this paper are: math-ph
0
The categories of this paper are: hep-ph
4
The categories of this paper are: cond-mat
27
The categories of this paper are: comp-gas
6
The categories of this paper are: astro-ph
21
The categories of this paper are: adap-org
29
The categories of this paper are: funct-an
13
The categories of this paper are: quant-ph