  With the wide application of machine translation, the testing of Machine
Translation Systems (MTSs) has attracted much attention. Recent works apply
Metamorphic Testing (MT) to address the oracle problem in MTS testing. Existing
MT methods for MTS generally follow the workflow of input transformation and
output relation comparison, which generates a follow-up input sentence by
mutating the source input and compares the source and follow-up output
translations to detect translation errors, respectively. These methods use
various input transformations to generate test case pairs and have successfully
triggered numerous translation errors. However, they have limitations in
performing fine-grained and rigorous output relation comparison and thus may
report false alarms and miss true errors. In this paper, we propose a word
closure-based output comparison method to address the limitations of the
existing MTS MT methods. Specifically, we first build a new comparison unit
called word closure, where each closure includes a group of correlated input
and output words in the test case pair. Word closures suggest the linkages
between the appropriate fragment in the source output translation and its
counterpart in the follow-up output for comparison. Next, we compare the
semantics on the level of word closure to identify the translation errors. In
this way, we perform a fine-grained and rigorous semantic comparison for the
outputs and thus realize more effective violation identification. We evaluate
our method with the test cases generated by five existing input transformations
and translation outputs from three popular MTSs. Results show that our method
significantly outperforms the existing works in violation identification by
improving the precision and recall and achieving an average increase of 29.8%
in F1 score. It also helps to increase the F1 score of translation error
localization by 35.9%.
