  Context: Topic modeling finds human-readable structures in unstructured
textual data. A widely used topic modeler is Latent Dirichlet allocation. When
run on different datasets, LDA suffers from "order effects" i.e. different
topics are generated if the order of training data is shuffled. Such order
effects introduce a systematic error for any study. This error can relate to
misleading results;specifically, inaccurate topic descriptions and a reduction
in the efficacy of text mining classification results. Objective: To provide a
method in which distributions generated by LDA are more stable and can be used
for further analysis. Method: We use LDADE, a search-based software engineering
tool that tunes LDA's parameters using DE (Differential Evolution). LDADE is
evaluated on data from a programmer information exchange site (Stackoverflow),
title and abstract text of thousands ofSoftware Engineering (SE) papers, and
software defect reports from NASA. Results were collected across different
implementations of LDA (Python+Scikit-Learn, Scala+Spark); across different
platforms (Linux, Macintosh) and for different kinds of LDAs (VEM,or using
Gibbs sampling). Results were scored via topic stability and text mining
classification accuracy. Results: In all treatments: (i) standard LDA exhibits
very large topic instability; (ii) LDADE's tunings dramatically reduce cluster
instability; (iii) LDADE also leads to improved performances for supervised as
well as unsupervised learning. Conclusion: Due to topic instability, using
standard LDA with its "off-the-shelf" settings should now be depreciated. Also,
in future, we should require SE papers that use LDA to test and (if needed)
mitigate LDA topic instability. Finally, LDADE is a candidate technology for
effectively and efficiently reducing that instability.
