  Self-supervised speech representation learning (speech SSL) has demonstrated
the benefit of scale in learning rich representations for Automatic Speech
Recognition (ASR) with limited paired data, such as wav2vec 2.0. We investigate
the existence of sparse subnetworks in pre-trained speech SSL models that
achieve even better low-resource ASR results. However, directly applying widely
adopted pruning methods such as the Lottery Ticket Hypothesis (LTH) is
suboptimal in the computational cost needed. Moreover, we show that the
discovered subnetworks yield minimal performance gain compared to the original
dense network. We present Prune-Adjust-Re-Prune (PARP), which discovers and
finetunes subnetworks for much better performance, while only requiring a
single downstream ASR finetuning run. PARP is inspired by our surprising
observation that subnetworks pruned for pre-training tasks need merely a slight
adjustment to achieve a sizeable performance boost in downstream ASR tasks.
Extensive experiments on low-resource ASR verify (1) sparse subnetworks exist
in mono-lingual/multi-lingual pre-trained speech SSL, and (2) the computational
advantage and performance gain of PARP over baseline pruning methods. In
particular, on the 10min Librispeech split without LM decoding, PARP discovers
subnetworks from wav2vec 2.0 with an absolute 10.9%/12.6% WER decrease compared
to the full model. We further demonstrate the effectiveness of PARP via:
cross-lingual pruning without any phone recognition degradation, the discovery
of a multi-lingual subnetwork for 10 spoken languages in 1 finetuning run, and
its applicability to pre-trained BERT/XLNet for natural language tasks.
