  Robots assisting us in environments such as factories or homes must learn to
make use of objects as tools to perform tasks, for instance using a tray to
carry objects. We consider the problem of learning commonsense knowledge of
when a tool may be useful and how its use may be composed with other tools to
accomplish a high-level task instructed by a human. Specifically, we introduce
a novel neural model, termed TOOLTANGO, that first predicts the next tool to be
used, and then uses this information to predict the next action. We show that
this joint model can inform learning of a fine-grained policy enabling the
robot to use a particular tool in sequence and adds a significant value in
making the model more accurate. TOOLTANGO encodes the world state, comprising
objects and symbolic relationships between them, using a graph neural network
and is trained using demonstrations from human teachers instructing a virtual
robot in a physics simulator. The model learns to attend over the scene using
knowledge of the goal and the action history, finally decoding the symbolic
action to execute. Crucially, we address generalization to unseen environments
where some known tools are missing, but alternative unseen tools are present.
We show that by augmenting the representation of the environment with
pre-trained embeddings derived from a knowledge-base, the model can generalize
effectively to novel environments. Experimental results show at least
48.8-58.1% absolute improvement over the baselines in predicting successful
symbolic plans for a simulated mobile manipulator in novel environments with
unseen objects. This work takes a step in the direction of enabling robots to
rapidly synthesize robust plans for complex tasks, particularly in novel
settings
