Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Knowledge Distillation for Small-footprint Highway Networks   Deep learning has significantly advanced state-of-the-art of speech
recognition in the past few years. However, compared to conventional Gaussian
mixture acoustic models, neural network models are usually much larger, and are
therefore not very deployable in embedded devices. Previously, we investigated
a compact highway deep neural network (HDNN) for acoustic modelling, which is a
type of depth-gated feedforward neural network. We have shown that HDNN-based
acoustic models can achieve comparable recognition accuracy with much smaller
number of model parameters compared to plain deep neural network (DNN) acoustic
models. In this paper, we push the boundary further by leveraging on the
knowledge distillation technique that is also known as {\it teacher-student}
training, i.e., we train the compact HDNN model with the supervision of a high
accuracy cumbersome model. Furthermore, we also investigate sequence training
and adaptation in the context of teacher-student training. Our experiments were
performed on the AMI meeting speech recognition corpus. With this technique, we
significantly improved the recognition accuracy of the HDNN acoustic model with
less than 0.8 million parameters, and narrowed the gap between this model and
the plain DNN with 30 million parameters.

13
The categories of this paper are: quant-ph
30
The categories of this paper are: q-alg
31
The categories of this paper are: ao-sci
18
The categories of this paper are: econ
21
The categories of this paper are: adap-org
0
The categories of this paper are: hep-ph
9
The categories of this paper are: nlin
20
The categories of this paper are: acc-phys
23
The categories of this paper are: solv-int
26
The categories of this paper are: bayes-an
22
The categories of this paper are: chao-dyn
16
The categories of this paper are: stat
24
The categories of this paper are: dg-ga
11
The categories of this paper are: hep-lat
5
The categories of this paper are: gr-qc
19
The categories of this paper are: eess