thetically generated textbooks and exercises with GPT-3.5. Despite its small scale, phi-1 achieves impressive accuracy on HumanEval and MBPP, two benchmarks for evaluating code generation models.

The authors argue that the quality of the training data is a crucial factor in the performance of the model. They suggest that standard sources of text data for code generation, such as The Stack and StackOverflow, are not optimal for teaching the model how to reason and plan algorithmically. Instead, they propose