  Previous researchers conducting Just-In-Time (JIT) defect prediction tasks
have primarily focused on the performance of individual pre-trained models,
without exploring the relationship between different pre-trained models as
backbones. In this study, we build six models: RoBERTaJIT, CodeBERTJIT,
BARTJIT, PLBARTJIT, GPT2JIT, and CodeGPTJIT, each with a distinct pre-trained
model as its backbone. We systematically explore the differences and
connections between these models. Specifically, we investigate the performance
of the models when using Commit code and Commit message as inputs, as well as
the relationship between training efficiency and model distribution among these
six models. Additionally, we conduct an ablation experiment to explore the
sensitivity of each model to inputs. Furthermore, we investigate how the models
perform in zero-shot and few-shot scenarios. Our findings indicate that each
model based on different backbones shows improvements, and when the backbone's
pre-training model is similar, the training resources that need to be consumed
are much more closer. We also observe that Commit code plays a significant role
in defect detection, and different pre-trained models demonstrate better defect
detection ability with a balanced dataset under few-shot scenarios. These
results provide new insights for optimizing JIT defect prediction tasks using
pre-trained models and highlight the factors that require more attention when
constructing such models. Additionally, CodeGPTJIT and GPT2JIT achieved better
performance than DeepJIT and CC2Vec on the two datasets respectively under 2000
training samples. These findings emphasize the effectiveness of
transformer-based pre-trained models in JIT defect prediction tasks, especially
in scenarios with limited training data.
