  Automated Program Repair (APR) techniques have shown more and more promising
results in fixing real-world bugs. Despite the effectiveness, APR techniques
still face an overfitting problem: a generated patch can be incorrect although
it passes all tests. It is time-consuming to manually evaluate the correctness
of generated patches that can pass all tests. To address this problem, many
approaches have been proposed to automatically assess the correctness of
patches generated by APR techniques. However, existing approaches require a
large set of manually labeled patches as the training data. To mitigate the
issue, in this study, we propose PatchZero, the patch correctness assessment by
adopting large pre-trained models. Specifically, for patches generated by a new
or unseen APR tool, PatchZero does not need labeled patches of this new or
unseen APR tool for training (i.e., zero-shot) but directly queries the large
pre-trained model to get predictions on the correctness labels without
training. In this way, PatchZero can reduce the manual labeling effort when
building a model to automatically assess the correctness of generated patches
of new APR tools. To provide knowledge regarding the automatic patch
correctness assessment (APCA) task to the large pre-trained models, we also
design an instance-wise demonstration formation strategy by using contrastive
learning. Specifically, PatchZero selects semantically similar patches to help
the large pre-trained model to give more accurate predictions on the unlabeled
patches. Our experimental results showed that PatchZero can achieve an accuracy
of 82.7% and an F1-score of 86.0% on average although no labeled patch of the
new or unseen APR tool is available. In addition, our proposed technique
outperformed the prior state-of-the-art by a large margin.
