  The exponential growth of social media platforms such as Twitter and Facebook
has revolutionized textual communication and textual content publication in
human society. However, they have been increasingly exploited to propagate
toxic content, such as hate speech, malicious advertisement, and pornography,
which can lead to highly negative impacts (e.g., harmful effects on teen mental
health). Researchers and practitioners have been enthusiastically developing
and extensively deploying textual content moderation software to address this
problem. However, we find that malicious users can evade moderation by changing
only a few words in the toxic content. Moreover, modern content moderation
software performance against malicious inputs remains underexplored. To this
end, we propose MTTM, a Metamorphic Testing framework for Textual content
Moderation software. Specifically, we conduct a pilot study on 2,000 text
messages collected from real users and summarize eleven metamorphic relations
across three perturbation levels: character, word, and sentence. MTTM employs
these metamorphic relations on toxic textual contents to generate test cases,
which are still toxic yet likely to evade moderation. In our evaluation, we
employ MTTM to test three commercial textual content moderation software and
two state-of-the-art moderation algorithms against three kinds of toxic
content. The results show that MTTM achieves up to 83.9%, 51%, and 82.5% error
finding rates (EFR) when testing commercial moderation software provided by
Google, Baidu, and Huawei, respectively, and it obtains up to 91.2% EFR when
testing the state-of-the-art algorithms from the academy. In addition, we
leverage the test cases generated by MTTM to retrain the model we explored,
which largely improves model robustness (0% to 5.9% EFR) while maintaining the
accuracy on the original test set.
