  Programming can be challenging for novices, and it is difficult to provide
high-quality, comprehensive, and timely support at scale. Generative AI and its
products, like ChatGPT, can create a solution for most introductory programming
problems. However, students may become overly reliant on these tools for quick
code generation and homework completion, which might cause reduced engagement
and limited learning. In this work, we present CodeTailor, a system that
leverages large language models (LLMs) while still encouraging students'
cognitive engagement. CodeTailor provides a personalized Parsons puzzle to
support struggling students. In a Parsons puzzle, students place mixed-up code
blocks in the correct order to solve it. A technical evaluation with 800
incorrect student code demonstrated that CodeTailor can efficiently create
high-quality (correct, personalized, and concise) Parsons puzzles for students.
In a within-subjects experiment with 18 novice programmers, students rated
using CodeTailor as more engaging, and they recalled more newly acquired
elements from the supported practice in the posttest after using CodeTailor,
compared to when they simply received an AI-generated solution. In addition,
most students preferred to use CodeTailor over receiving an AI-generated
solution to support learning. Qualitative observations and interviews also
provided evidence for the benefits of CodeTailor, including emphasizing
thinking about solution construction, fostering continuity in learning,
promoting reflection, and boosting student confidence. We conclude by
suggesting future design ideas for applying generative AI to facilitate active
learning opportunities and minimize over-reliance.
