  Machine translation systems have been widely adopted in our daily life,
making life easier and more convenient. Unfortunately, erroneous translations
may result in severe consequences, such as financial losses. This requires to
improve the accuracy and the reliability of machine translation systems.
However, it is challenging to test machine translation systems because of the
complexity and intractability of the underlying neural models. To tackle these
challenges, we propose a novel metamorphic testing approach by syntactic tree
pruning (STP) to validate machine translation systems. Our key insight is that
a pruned sentence should have similar crucial semantics compared with the
original sentence. Specifically, STP (1) proposes a core semantics-preserving
pruning strategy by basic sentence structure and dependency relations on the
level of syntactic tree representation; (2) generates source sentence pairs
based on the metamorphic relation; (3) reports suspicious issues whose
translations break the consistency property by a bag-of-words model. We further
evaluate STP on two state-of-the-art machine translation systems (i.e., Google
Translate and Bing Microsoft Translator) with 1,200 source sentences as inputs.
The results show that STP can accurately find 5,073 unique erroneous
translations in Google Translate and 5,100 unique erroneous translations in
Bing Microsoft Translator (400% more than state-of-the-art techniques), with
64.5% and 65.4% precision, respectively. The reported erroneous translations
vary in types and more than 90% of them cannot be found by state-of-the-art
techniques. There are 9,393 erroneous translations unique to STP, which is
711.9% more than state-of-the-art techniques. Moreover, STP is quite effective
to detect translation errors for the original sentences with a recall reaching
74.0%, improving state-of-the-art techniques by 55.1% on average.
