  The goal of unpaired image captioning (UIC) is to describe images without
using image-caption pairs in the training phase. Although challenging, we
except the task can be accomplished by leveraging a training set of images
aligned with visual concepts. Most existing studies use off-the-shelf
algorithms to obtain the visual concepts because the Bounding Box (BBox) labels
or relationship-triplet labels used for the training are expensive to acquire.
In order to resolve the problem in expensive annotations, we propose a novel
approach to achieve cost-effective UIC. Specifically, we adopt image-level
labels for the optimization of the UIC model in a weakly-supervised manner. For
each image, we assume that only the image-level labels are available without
specific locations and numbers. The image-level labels are utilized to train a
weakly-supervised object recognition model to extract object information (e.g.,
instance) in an image, and the extracted instances are adopted to infer the
relationships among different objects based on an enhanced graph neural network
(GNN). The proposed approach achieves comparable or even better performance
compared with previous methods without the expensive cost of annotations.
Furthermore, we design an unrecognized object (UnO) loss combined with a visual
concept reward to improve the alignment of the inferred object and relationship
information with the images. It can effectively alleviate the issue encountered
by existing UIC models about generating sentences with nonexistent objects. To
the best of our knowledge, this is the first attempt to solve the problem of
Weakly-Supervised visual concept recognition for UIC (WS-UIC) based only on
image-level labels. Extensive experiments have been carried out to demonstrate
that the proposed WS-UIC model achieves inspiring results on the COCO dataset
while significantly reducing the cost of labeling.
