  One type of machine learning, text classification, is now regularly applied
in the legal matters involving voluminous document populations because it can
reduce the time and expense associated with the review of those documents. One
form of machine learning - Active Learning - has drawn attention from the legal
community because it offers the potential to make the machine learning process
even more effective. Active Learning, applied to legal documents, is considered
a new technology in the legal domain and is continuously applied to all
documents in a legal matter until an insignificant number of relevant documents
are left for review. This implementation is slightly different than traditional
implementations of Active Learning where the process stops once achieving
acceptable model performance. The purpose of this paper is twofold: (i) to
question whether Active Learning actually is a superior learning methodology
and (ii) to highlight the ways that Active Learning can be most effectively
applied to real legal industry data. Unlike other studies, our experiments were
performed against large data sets taken from recent, real-world legal matters
covering a variety of areas. We conclude that, although these experiments show
the Active Learning strategy popularly used in legal document review can
quickly identify informative training documents, it becomes less effective over
time. In particular, our findings suggest this most popular form of Active
Learning in the legal arena, where the highest-scoring documents are selected
as training examples, is in fact not the most efficient approach in most
instances. Ultimately, a different Active Learning strategy may be best suited
to initiate the predictive modeling process but not to continue through the
entire document review.
