Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: DistilHuBERT: Speech Representation Learning by Layer-wise Distillation
  of Hidden-unit BERT   Self-supervised speech representation learning methods like wav2vec 2.0 and
Hidden-unit BERT (HuBERT) leverage unlabeled speech data for pre-training and
offer good representations for numerous speech processing tasks. Despite the
success of these methods, they require large memory and high pre-training
costs, making them inaccessible for researchers in academia and small
companies. Therefore, this paper introduces DistilHuBERT, a novel multi-task
learning framework to distill hidden representations from a HuBERT model
directly. This method reduces HuBERT's size by 75% and 73% faster while
retaining most performance in ten different tasks. Moreover, DistilHuBERT
required little training time and data, opening the possibilities of
pre-training personal and on-device SSL models for speech.

32
The categories of this paper are: atom-ph
26
The categories of this paper are: bayes-an
13
The categories of this paper are: quant-ph
18
The categories of this paper are: econ
0
The categories of this paper are: hep-ph
17
The categories of this paper are: q-fin
8
The categories of this paper are: hep-ex
6
The categories of this paper are: astro-ph
11
The categories of this paper are: hep-lat
21
The categories of this paper are: adap-org
27
The categories of this paper are: comp-gas
7
The categories of this paper are: hep-th
20
The categories of this paper are: acc-phys
24
The categories of this paper are: dg-ga
36
The categories of this paper are: cmp-lg
35
The categories of this paper are: mtrl-th