  Machine learning-based automatic scoring can be challenging if students'
responses are unbalanced across scoring categories, as it introduces
uncertainty in the machine training process. To meet this challenge, we
introduce a novel text data augmentation framework using GPT-4, a generative
large language model, specifically tailored for unbalanced datasets in
automatic scoring. Our experimental dataset comprised student-written responses
to two science items. We crafted prompts for GPT-4 to generate responses
resembling student-written answers, particularly for the minority scoring
classes, to augment the data. We then finetuned DistillBERT for automatic
scoring based on the augmented and original datasets. Model performance was
assessed using accuracy, precision, recall, and F1 score. We incorporate varied
amounts of augmented data to examine scoring performance, and our findings
revealed remarkedly improved model performance. The average maximum increase
observed across two items is: 3.5% for accuracy, 30.6% for precision, 21.1% for
recall, and 24.2% for F1 score. Notably, using just 5% of the augmented data
led to substantial improvements: 2.6%, 29.2%, 15.1%, and 19.6%. Interestingly,
the extent of improvement varied depending on specific datasets. Moreover, we
found that a varying amount of augmented data (5%-40%) was needed to obtain a
stable improvement. We also compare models trained with GPT-4 augmented data
and those trained with additional student-written responses. The findings
indicate that former ones match or even exceed the performance of the latter.
Specifically, there is an average difference of 1.7%, 1.9%, 11.0%, and 7.8% for
four metrics separately. This research underscores the potential and
effectiveness of data augmentation techniques utilizing GPT-4 in addressing
unbalanced datasets within automated assessment.
