Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Learning to Solve Complex Tasks by Talking to Agents   Humans often solve complex problems by interacting (in natural language) with
existing agents, such as AI assistants, that can solve simpler sub-tasks. These
agents themselves can be powerful systems built using extensive resources and
privately held data. In contrast, common NLP benchmarks aim for the development
of self-sufficient models for every task. To address this gap and facilitate
research towards ``green'' AI systems that build upon existing agents, we
propose a new benchmark called CommaQA that contains three kinds of complex
reasoning tasks that are designed to be solved by ``talking'' to four agents
with different capabilities. We demonstrate that state-of-the-art black-box
models, which are unable to leverage existing agents, struggle on CommaQA
(exact match score only reaches 40pts) even when given access to the agents'
internal knowledge and gold fact supervision. On the other hand, models using
gold question decomposition supervision can indeed solve CommaQA to a high
accuracy (over 96\% exact match) by learning to utilize the agents. Even these
additional supervision models, however, do not solve our compositional
generalization test set. Finally the end-goal of learning to solve complex
tasks by communicating with existing agents \emph{without relying on any
additional supervision} remains unsolved and we hope CommaQA serves as a novel
benchmark to enable the development of such systems.

29
The categories of this paper are: funct-an
17
The categories of this paper are: q-fin
8
The categories of this paper are: hep-ex
14
The categories of this paper are: math-ph
33
The categories of this paper are: plasm-ph
7
The categories of this paper are: hep-th
13
The categories of this paper are: quant-ph
16
The categories of this paper are: stat
0
The categories of this paper are: hep-ph
10
The categories of this paper are: q-bio
20
The categories of this paper are: acc-phys
30
The categories of this paper are: q-alg
26
The categories of this paper are: bayes-an
35
The categories of this paper are: mtrl-th
5
The categories of this paper are: gr-qc
24
The categories of this paper are: dg-ga