  Finetuning pre-trained language models (LMs) enhances the models'
capabilities. Prior techniques fine-tune a pre-trained LM on input-output pairs
(e.g., instruction fine-tuning), or with numerical rewards that gauge the
quality of its outputs (e.g., reinforcement learning from human feedback). We
explore LMs' potential to learn from textual interactions (LeTI) that not only
check their correctness with binary labels, but also pinpoint and explain
errors in their outputs through textual feedback. Our investigation focuses on
the code generation task, where the model produces code pieces in response to
natural language instructions. This setting invites a natural and scalable way
to acquire the textual feedback: the error messages and stack traces from code
execution using a Python interpreter. LeTI iteratively fine-tunes the model,
using the LM objective, on a concatenation of natural language instructions,
LM-generated programs, and textual feedback, which is only provided when the
generated program fails to solve the task. Prepended to this fine-tuning text,
a binary reward token is used to differentiate correct and buggy solutions. On
MBPP, a code generation dataset, LeTI substantially improves the performance of
two base LMs of different scales. LeTI requires no ground-truth outputs for
training and even outperforms a fine-tuned baseline that does. LeTI's strong
performance generalizes to other datasets. Trained on MBPP, it achieves
comparable or better performance than the base LMs on unseen problems in
HumanEval. Furthermore, compared to binary feedback, we observe that textual
feedback leads to improved generation quality and sample efficiency, achieving
the same performance with fewer than half of the gradient steps. LeTI is
equally applicable in natural language tasks when they can be formulated as
code generation, which we empirically verified on event argument extraction.
