  Given the large number of publications in software engineering, frequent
literature reviews are required to keep current on work in specific areas. One
tedious work in literature reviews is to find relevant studies amongst
thousands of non-relevant search results. In theory, expert systems can assist
in finding relevant work but those systems have primarily been tested in
simulations rather than in application to actual literature reviews. Hence, few
researchers have faith in such expert systems. Accordingly, using a realistic
case study, this paper assesses how well our state-of-the-art expert system can
help with literature reviews.
  The assessed literature review aimed at identifying test case prioritization
techniques for automated UI testing, specifically from 8,349 papers on IEEE
Xplore. This corpus was studied with an expert system that incorporates an
incrementally updated human-in-the-loop active learning tool. Using that expert
system, in three hours, we found 242 relevant papers from which we identified
12 techniques representing the state-of-the-art in test case prioritization
when source code information is not available. These results were then
validated by six other graduate students manually exploring the same corpus.
Without the expert system, this task would have required 53 hours and would
have found 27 additional papers. That is, our expert system achieved 90% recall
with 6% of the human effort cost when compared to a conventional manual method.
Significantly, the same 12 state-of-the-art test case prioritization techniques
were identified by both the expert system and the manual method. That is, the
27 papers missed by the expert system would not have changed the conclusion of
the literature review.
  Hence, if this result generalizes, it endorses the use of our expert system
to assist in literature reviews.
