  With their remarkable ability to generate code, large language models (LLMs)
are a transformative technology for computing education practice. They have
created an urgent need for educators to rethink pedagogical approaches and
teaching strategies for newly emerging skill sets. Traditional approaches to
learning programming have focused on frequent and repeated practice at writing
code. The ease with which code can now be generated has resulted in a shift in
focus towards reading, understanding and evaluating LLM-generated code. In
parallel with this shift, a new essential skill is emerging -- the ability to
construct good prompts for code-generating models. This paper introduces a
novel pedagogical concept known as a `Prompt Problem', designed to help
students learn how to craft effective prompts for LLMs. A Prompt Problem
challenges a student to create a natural language prompt that leads an LLM to
produce the correct code for a specific problem. To support the delivery of
Prompt Problems at scale, in this paper we also present a novel tool called
Promptly which hosts a repository of Prompt Problems and automates the
evaluation of prompt-generated code. We report empirical findings from a field
study in which Promptly was deployed in a first-year Python programming course
(n=54). We explore student interactions with the tool and their perceptions of
the Prompt Problem concept. We found that Promptly was largely well-received by
students for its ability to engage their computational thinking skills and
expose them to new programming constructs. We also discuss avenues for future
work, including variations on the design of Prompt Problems and the need to
study their integration into the curriculum and teaching practice.
