  The Image Captioning (IC) technique is widely used to describe images in
natural language. Recently, some IC system testing methods have been proposed.
However, these methods still rely on pre-annotated information and hence cannot
really alleviate the oracle problem in testing. Besides, their method
artificially manipulates objects, which may generate unreal images as test
cases and thus lead to less meaningful testing results. Thirdly, existing
methods have various requirements on the eligibility of source test cases, and
hence cannot fully utilize the given images to perform testing. To tackle these
issues, in this paper, we propose REIC to perform metamorphic testing for IC
systems with some image-level reduction transformations like image cropping and
stretching. Instead of relying on the pre-annotated information, REIC uses a
localization method to align objects in the caption with corresponding objects
in the image, and checks whether each object is correctly described or deleted
in the caption after transformation. With the image-level reduction
transformations, REIC does not artificially manipulate any objects and hence
can avoid generating unreal follow-up images. Besides, it eliminates the
requirement on the eligibility of source test cases in the metamorphic
transformation process, as well as decreases the ambiguity and boosts the
diversity among the follow-up test cases, which consequently enables testing to
be performed on any test image and reveals more distinct valid violations. We
employ REIC to test five popular IC systems. The results demonstrate that REIC
can sufficiently leverage the provided test images to generate follow-up cases
of good reality, and effectively detect a great number of distinct violations,
without the need for any pre-annotated information.
