  Nowadays, it is common to release audio content to the public. However, with
the rise of voice cloning technology, attackers have the potential to easily
impersonate a specific person by utilizing his publicly released audio without
any permission. Therefore, it becomes significant to detect any potential
misuse of the released audio content and protect its timbre from being
impersonated. To this end, we introduce a novel concept, "Timbre Watermarking",
which embeds watermark information into the target individual's speech,
eventually defeating the voice cloning attacks. To ensure the watermark is
robust to the voice cloning model's learning process, we design an end-to-end
voice cloning-resistant detection framework. The core idea of our solution is
to embed and extract the watermark in the frequency domain in a temporally
invariant manner. To acquire generalization across different voice cloning
attacks, we modulate their shared process and integrate it into our framework
as a distortion layer. Experiments demonstrate that the proposed timbre
watermarking can defend against different voice cloning attacks, exhibit strong
resistance against various adaptive attacks (e.g., reconstruction-based removal
attacks, watermark overwriting attacks), and achieve practicality in real-world
services such as PaddleSpeech, Voice-Cloning-App, and so-vits-svc. In addition,
ablation studies are also conducted to verify the effectiveness of our design.
Some audio samples are available at
https://timbrewatermarking.github.io/samples.
