  Recently developed pretrained models can encode rich world knowledge
expressed in multiple modalities, such as text and images. However, the outputs
of these models cannot be integrated into algorithms to solve sequential
decision-making tasks. We develop an algorithm that utilizes the knowledge from
pretrained models to construct and verify controllers for sequential
decision-making tasks, and to ground these controllers to task environments
through visual observations. In particular, the algorithm queries a pretrained
model with a user-provided, text-based task description and uses the model's
output to construct an automaton-based controller that encodes the model's
task-relevant knowledge. It then verifies whether the knowledge encoded in the
controller is consistent with other independently available knowledge, which
may include abstract information on the environment or user-provided
specifications. If this verification step discovers any inconsistency, the
algorithm automatically refines the controller to resolve the inconsistency.
Next, the algorithm leverages the vision and language capabilities of
pretrained models to ground the controller to the task environment. It collects
image-based observations from the task environment and uses the pretrained
model to link these observations to the text-based control logic encoded in the
controller (e.g., actions and conditions that trigger the actions). We propose
a mechanism to ensure the controller satisfies the user-provided specification
even when perceptual uncertainties are present. We demonstrate the algorithm's
ability to construct, verify, and ground automaton-based controllers through a
suite of real-world tasks, including daily life and robot manipulation tasks.
