  Existing synthetic datasets (FigureQA, DVQA) for reasoning over plots do not
contain variability in data labels, real-valued data, or complex reasoning
questions. Consequently, proposed models for these datasets do not fully
address the challenge of reasoning over plots. In particular, they assume that
the answer comes either from a small fixed size vocabulary or from a bounding
box within the image. However, in practice, this is an unrealistic assumption
because many questions require reasoning and thus have real-valued answers
which appear neither in a small fixed size vocabulary nor in the image. In this
work, we aim to bridge this gap between existing datasets and real-world plots.
Specifically, we propose PlotQA with 28.9 million question-answer pairs over
224,377 plots on data from real-world sources and questions based on
crowd-sourced question templates. Further, 80.76% of the out-of-vocabulary
(OOV) questions in PlotQA have answers that are not in a fixed vocabulary.
Analysis of existing models on PlotQA reveals that they cannot deal with OOV
questions: their overall accuracy on our dataset is in single digits. This is
not surprising given that these models were not designed for such questions. As
a step towards a more holistic model which can address fixed vocabulary as well
as OOV questions, we propose a hybrid approach: Specific questions are answered
by choosing the answer from a fixed vocabulary or by extracting it from a
predicted bounding box in the plot, while other questions are answered with a
table question-answering engine which is fed with a structured table generated
by detecting visual elements from the image. On the existing DVQA dataset, our
model has an accuracy of 58%, significantly improving on the highest reported
accuracy of 46%. On PlotQA, our model has an accuracy of 22.52%, which is
significantly better than state of the art models.
