Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Point and Ask: Incorporating Pointing into Visual Question Answering   Visual Question Answering (VQA) has become one of the key benchmarks of
visual recognition progress. Multiple VQA extensions have been explored to
better simulate real-world settings: different question formulations, changing
training and test distributions, conversational consistency in dialogues, and
explanation-based answering. In this work, we further expand this space by
considering visual questions that include a spatial point of reference.
Pointing is a nearly universal gesture among humans, and real-world VQA is
likely to involve a gesture towards the target region.
  Concretely, we (1) introduce and motivate point-input questions as an
extension of VQA, (2) define three novel classes of questions within this
space, and (3) for each class, introduce both a benchmark dataset and a series
of baseline models to handle its unique challenges. There are two key
distinctions from prior work. First, we explicitly design the benchmarks to
require the point input, i.e., we ensure that the visual question cannot be
answered accurately without the spatial reference. Second, we explicitly
explore the more realistic point spatial input rather than the standard but
unnatural bounding box input. Through our exploration we uncover and address
several visual recognition challenges, including the ability to infer human
intent, reason both locally and globally about the image, and effectively
combine visual, language and spatial inputs. Code is available at:
https://github.com/princetonvisualai/pointingqa .

6
The categories of this paper are: astro-ph
12
The categories of this paper are: nucl-th
4
The categories of this paper are: cond-mat
18
The categories of this paper are: econ
37
The categories of this paper are: supr-con
36
The categories of this paper are: cmp-lg
21
The categories of this paper are: adap-org
23
The categories of this paper are: solv-int
26
The categories of this paper are: bayes-an
3
The categories of this paper are: physics
31
The categories of this paper are: ao-sci
28
The categories of this paper are: alg-geom
17
The categories of this paper are: q-fin
16
The categories of this paper are: stat
2
The categories of this paper are: math
33
The categories of this paper are: plasm-ph