  Text analytics based on supervised machine learning classifiers has shown
great promise in a multitude of domains, but has yet to be applied to
Seismology. We test various standard models (Naive Bayes, k-Nearest Neighbors,
Support Vector Machines, and Random Forests) on a seismological corpus of 100
articles related to the topic of precursory accelerating seismicity, spanning
from 1988 to 2010. This corpus was labelled in Mignan (2011) with the precursor
whether explained by critical processes (i.e., cascade triggering) or by other
processes (such as signature of main fault loading). We investigate rather the
classification process can be automatized to help analyze larger corpora in
order to better understand trends in earthquake predictability research. We
find that the Naive Bayes model performs best, in agreement with the machine
learning literature for the case of small datasets, with cross-validation
accuracies of 86% for binary classification. For a refined multiclass
classification ('non-critical process' < 'agnostic' < 'critical process
assumed' < 'critical process demonstrated'), we obtain up to 78% accuracy.
Prediction on a dozen of articles published since 2011 shows however a weak
generalization with a F1-score of 60%, only slightly better than a random
classifier, which can be explained by a change of authorship and use of
different terminologies. Yet, the model shows F1-scores greater than 80% for
the two multiclass extremes ('non-critical process' versus 'critical process
demonstrated') while it falls to random classifier results (around 25%) for
papers labelled 'agnostic' or 'critical process assumed'. Those results are
encouraging in view of the small size of the corpus and of the high degree of
abstraction of the labelling. Domain knowledge engineering remains essential
but can be made transparent by an investigation of Naive Bayes keyword
posterior probabilities.
