  Background: Understanding maintenance activities performed in a source code
repository could help practitioners reduce uncertainty and improve
cost-effectiveness by planning ahead and pre-allocating resources towards
source code maintenance. The research community uses 3 main classification
categories for maintenance activities: Corrective: fault fixing; Perfective:
system improvements; Adaptive: new feature introduction. Previous work in this
area has mostly concentrated on evaluating commit classification (into
maintenance activities) models in the scope of a single software project. Aims:
In this work we seek to design a commit classification model capable of
providing high accuracy and Kappa across different projects. In addition, we
wish to compare the accuracy and kappa characteristics of classification models
that utilize word frequency analysis, source code changes, and combination
thereof. Method: We suggest a novel method for automatically classifying
commits into maintenance activities by utilizing source code changes (e.g,
statement added, method removed, etc.). The results we report are based on
studying 11 popular open source projects from various professional domains from
which we had manually classified 1151 commits, over 100 from each of the
studied projects. Our models were trained using 85% of the dataset, while the
remaining 15% were used as a test set. Results: Our method shows a promising
accuracy of 76% and Cohen's kappa of 63% (considered "Good" in this context)
for the test dataset, an improvement of over 20 percentage points, and a
relative boost of ~40% in the context of cross-project classification.
Conclusions: We show that by using source code changes in combination with
commit message word frequency analysis we are able to considerably boost
classification quality in a project agnostic manner.
