Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Learning UI Navigation through Demonstrations composed of Macro Actions   We have developed a framework to reliably build agents capable of UI
navigation. The state space is simplified from raw-pixels to a set of UI
elements extracted from screen understanding, such as OCR and icon detection.
The action space is restricted to the UI elements plus a few global actions.
Actions can be customized for tasks and each action is a sequence of basic
operations conditioned on status checks. With such a design, we are able to
train DQfD and BC agents with a small number of demonstration episodes. We
propose demo augmentation that significantly reduces the required number of
human demonstrations. We made a customization of DQfD to allow demos collected
on screenshots to facilitate the demo coverage of rare cases. Demos are only
collected for the failed cases during the evaluation of the previous version of
the agent. With 10s of iterations looping over evaluation, demo collection, and
training, the agent reaches a 98.7\% success rate on the search task in an
environment of 80+ apps and websites where initial states and viewing
parameters are randomized.

20
The categories of this paper are: acc-phys
18
The categories of this paper are: econ
14
The categories of this paper are: math-ph
30
The categories of this paper are: q-alg
5
The categories of this paper are: gr-qc
23
The categories of this paper are: solv-int
29
The categories of this paper are: funct-an
31
The categories of this paper are: ao-sci
2
The categories of this paper are: math
19
The categories of this paper are: eess
4
The categories of this paper are: cond-mat
33
The categories of this paper are: plasm-ph
15
The categories of this paper are: nucl-ex
11
The categories of this paper are: hep-lat
17
The categories of this paper are: q-fin
8
The categories of this paper are: hep-ex