  Deep neural networks (DNNs) have been widely and successfully adopted and
deployed in various applications of speech recognition. Recently, a few works
revealed that these models are vulnerable to backdoor attacks, where the
adversaries can implant malicious prediction behaviors into victim models by
poisoning their training process. In this paper, we revisit poison-only
backdoor attacks against speech recognition. We reveal that existing methods
are not stealthy since their trigger patterns are perceptible to humans or
machine detection. This limitation is mostly because their trigger patterns are
simple noises or separable and distinctive clips. Motivated by these findings,
we propose to exploit elements of sound ($e.g.$, pitch and timbre) to design
more stealthy yet effective poison-only backdoor attacks. Specifically, we
insert a short-duration high-pitched signal as the trigger and increase the
pitch of remaining audio clips to `mask' it for designing stealthy pitch-based
triggers. We manipulate timbre features of victim audios to design the stealthy
timbre-based attack and design a voiceprint selection module to facilitate the
multi-backdoor attack. Our attacks can generate more `natural' poisoned samples
and therefore are more stealthy. Extensive experiments are conducted on
benchmark datasets, which verify the effectiveness of our attacks under
different settings ($e.g.$, all-to-one, all-to-all, clean-label, physical, and
multi-backdoor settings) and their stealthiness. The code for reproducing main
experiments are available at \url{https://github.com/HanboCai/BadSpeech_SoE}.
