Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Learning to Generate Grounded Visual Captions without Localization
  Supervision   When automatically generating a sentence description for an image or video,
it often remains unclear how well the generated caption is grounded, that is
whether the model uses the correct image regions to output particular words, or
if the model is hallucinating based on priors in the dataset and/or the
language model. The most common way of relating image regions with words in
caption models is through an attention mechanism over the regions that are used
as input to predict the next word. The model must therefore learn to predict
the attentional weights without knowing the word it should localize. This is
difficult to train without grounding supervision since recurrent models can
propagate past information and there is no explicit signal to force the
captioning model to properly ground the individual decoded words. In this work,
we help the model to achieve this via a novel cyclical training regimen that
forces the model to localize each word in the image after the sentence decoder
generates it, and then reconstruct the sentence from the localized image
region(s) to match the ground-truth. Our proposed framework only requires
learning one extra fully-connected layer (the localizer), a layer that can be
removed at test time. We show that our model significantly improves grounding
accuracy without relying on grounding supervision or introducing extra
computation during inference, for both image and video captioning tasks. Code
is available at https://github.com/chihyaoma/cyclical-visual-captioning .

19
The categories of this paper are: eess
13
The categories of this paper are: quant-ph
35
The categories of this paper are: mtrl-th
6
The categories of this paper are: astro-ph
0
The categories of this paper are: hep-ph
28
The categories of this paper are: alg-geom
3
The categories of this paper are: physics
37
The categories of this paper are: supr-con
2
The categories of this paper are: math
31
The categories of this paper are: ao-sci
17
The categories of this paper are: q-fin
32
The categories of this paper are: atom-ph
18
The categories of this paper are: econ
25
The categories of this paper are: patt-sol
30
The categories of this paper are: q-alg
24
The categories of this paper are: dg-ga