  In this paper we present a neurosymbolic architecture for coupling
language-guided visual reasoning with robot manipulation. A non-expert human
user can prompt the robot using unconstrained natural language, providing a
referring expression (REF), a question (VQA), or a grasp action instruction.
The system tackles all cases in a task-agnostic fashion through the utilization
of a shared library of primitive skills. Each primitive handles an independent
sub-task, such as reasoning about visual attributes, spatial relation
comprehension, logic and enumeration, as well as arm control. A language parser
maps the input query to an executable program composed of such primitives,
depending on the context. While some primitives are purely symbolic operations
(e.g. counting), others are trainable neural functions (e.g. visual grounding),
therefore marrying the interpretability and systematic generalization benefits
of discrete symbolic approaches with the scalability and representational power
of deep networks. We generate a 3D vision-and-language synthetic dataset of
tabletop scenes in a simulation environment to train our approach and perform
extensive evaluations in both synthetic and real-world scenes. Results showcase
the benefits of our approach in terms of accuracy, sample-efficiency, and
robustness to the user's vocabulary, while being transferable to real-world
scenes with few-shot visual fine-tuning. Finally, we integrate our method with
a robot framework and demonstrate how it can serve as an interpretable solution
for an interactive object-picking task, both in simulation and with a real
robot. We make our datasets available in
https://gtziafas.github.io/neurosymbolic-manipulation.
