  Knowledge graphs are often used to represent structured information in a
flexible and efficient manner, but their use in situated dialogue remains
under-explored. This paper presents a novel conversational model for
human--robot interaction that rests upon a graph-based representation of the
dialogue state. The knowledge graph representing the dialogue state is
continuously updated with new observations from the robot sensors, including
linguistic, situated and multimodal inputs, and is further enriched by other
modules, in particular for spatial understanding. The neural conversational
model employed to respond to user utterances relies on a simple but effective
graph-to-text mechanism that traverses the dialogue state graph and converts
the traversals into a natural language form. This conversion of the state graph
into text is performed using a set of parameterized functions, and the values
for those parameters are optimized based on a small set of Wizard-of-Oz
interactions. After this conversion, the text representation of the dialogue
state graph is included as part of the prompt of a large language model used to
decode the agent response. The proposed approach is empirically evaluated
through a user study with a humanoid robot that acts as conversation partner to
evaluate the impact of the graph-to-text mechanism on the response generation.
After moving a robot along a tour of an indoor environment, participants
interacted with the robot using spoken dialogue and evaluated how well the
robot was able to answer questions about what the robot observed during the
tour. User scores show a statistically significant improvement in the perceived
factuality of the robot responses when the graph-to-text approach is employed,
compared to a baseline using inputs structured as semantic triples.
