Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Premise-based Multimodal Reasoning: Conditional Inference on Joint
  Textual and Visual Clues   It is a common practice for recent works in vision language cross-modal
reasoning to adopt a binary or multi-choice classification formulation taking
as input a set of source image(s) and textual query. In this work, we take a
sober look at such an unconditional formulation in the sense that no prior
knowledge is specified with respect to the source image(s). Inspired by the
designs of both visual commonsense reasoning and natural language inference
tasks, we propose a new task termed Premise-based Multi-modal Reasoning(PMR)
where a textual premise is the background presumption on each source image. The
PMR dataset contains 15,360 manually annotated samples which are created by a
multi-phase crowd-sourcing process. With selected high-quality movie
screenshots and human-curated premise templates from 6 pre-defined categories,
we ask crowd-source workers to write one true hypothesis and three distractors
(4 choices) given the premise and image through a cross-check procedure.
Besides, we generate adversarial samples to alleviate the annotation artifacts
and double the size of PMR. We benchmark various state-of-the-art (pretrained)
multi-modal inference models on PMR and conduct comprehensive experimental
analyses to showcase the utility of our dataset.

16
The categories of this paper are: stat
0
The categories of this paper are: hep-ph
24
The categories of this paper are: dg-ga
30
The categories of this paper are: q-alg
23
The categories of this paper are: solv-int
22
The categories of this paper are: chao-dyn
14
The categories of this paper are: math-ph
13
The categories of this paper are: quant-ph
17
The categories of this paper are: q-fin
34
The categories of this paper are: chem-ph
7
The categories of this paper are: hep-th
26
The categories of this paper are: bayes-an
32
The categories of this paper are: atom-ph
19
The categories of this paper are: eess
27
The categories of this paper are: comp-gas
20
The categories of this paper are: acc-phys