Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Evaluating Multimodal Representations on Visual Semantic Textual
  Similarity   The combination of visual and textual representations has produced excellent
results in tasks such as image captioning and visual question answering, but
the inference capabilities of multimodal representations are largely untested.
In the case of textual representations, inference tasks such as Textual
Entailment and Semantic Textual Similarity have been often used to benchmark
the quality of textual representations. The long term goal of our research is
to devise multimodal representation techniques that improve current inference
capabilities. We thus present a novel task, Visual Semantic Textual Similarity
(vSTS), where such inference ability can be tested directly. Given two items
comprised each by an image and its accompanying caption, vSTS systems need to
assess the degree to which the captions in context are semantically equivalent
to each other. Our experiments using simple multimodal representations show
that the addition of image representations produces better inference, compared
to text-only representations. The improvement is observed both when directly
computing the similarity between the representations of the two items, and when
learning a siamese network based on vSTS training data. Our work shows, for the
first time, the successful contribution of visual information to textual
inference, with ample room for benchmarking more complex multimodal
representation options.

7
The categories of this paper are: hep-th
21
The categories of this paper are: adap-org
17
The categories of this paper are: q-fin
12
The categories of this paper are: nucl-th
18
The categories of this paper are: econ
10
The categories of this paper are: q-bio
33
The categories of this paper are: plasm-ph
37
The categories of this paper are: supr-con
14
The categories of this paper are: math-ph
31
The categories of this paper are: ao-sci
26
The categories of this paper are: bayes-an
22
The categories of this paper are: chao-dyn
16
The categories of this paper are: stat
27
The categories of this paper are: comp-gas
6
The categories of this paper are: astro-ph
5
The categories of this paper are: gr-qc