Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Cross-modal Scene Graph Matching for Relationship-aware Image-Text
  Retrieval   Image-text retrieval of natural scenes has been a popular research topic.
Since image and text are heterogeneous cross-modal data, one of the key
challenges is how to learn comprehensive yet unified representations to express
the multi-modal data. A natural scene image mainly involves two kinds of visual
concepts, objects and their relationships, which are equally essential to
image-text retrieval. Therefore, a good representation should account for both
of them. In the light of recent success of scene graph in many CV and NLP tasks
for describing complex natural scenes, we propose to represent image and text
with two kinds of scene graphs: visual scene graph (VSG) and textual scene
graph (TSG), each of which is exploited to jointly characterize objects and
relationships in the corresponding modality. The image-text retrieval task is
then naturally formulated as cross-modal scene graph matching. Specifically, we
design two particular scene graph encoders in our model for VSG and TSG, which
can refine the representation of each node on the graph by aggregating
neighborhood information. As a result, both object-level and relationship-level
cross-modal features can be obtained, which favorably enables us to evaluate
the similarity of image and text in the two levels in a more plausible way. We
achieve state-of-the-art results on Flickr30k and MSCOCO, which verifies the
advantages of our graph matching based approach for image-text retrieval.

11
The categories of this paper are: hep-lat
25
The categories of this paper are: patt-sol
27
The categories of this paper are: comp-gas
9
The categories of this paper are: nlin
26
The categories of this paper are: bayes-an
6
The categories of this paper are: astro-ph
30
The categories of this paper are: q-alg
19
The categories of this paper are: eess
15
The categories of this paper are: nucl-ex
20
The categories of this paper are: acc-phys
0
The categories of this paper are: hep-ph
3
The categories of this paper are: physics
12
The categories of this paper are: nucl-th
21
The categories of this paper are: adap-org
4
The categories of this paper are: cond-mat
28
The categories of this paper are: alg-geom