  Background: A significant barrier to conducting systematic reviews and
meta-analysis is efficiently finding scientifically sound relevant articles.
Typically, less than 1% of articles match this requirement which leads to a
highly imbalanced task. Although feature-engineered and early neural networks
models were studied for this task, there is an opportunity to improve the
results.
  Methods: We framed the problem of filtering articles as a classification
task, and trained and tested several ensemble architectures of SciBERT, a
variant of BERT pre-trained on scientific articles, on a manually annotated
dataset of about 50K articles from MEDLINE. Since scientifically sound articles
are identified through a multi-step process we proposed a novel cascade
ensemble analogous to the selection process. We compared the performance of the
cascade ensemble with a single integrated model and other types of ensembles as
well as with results from previous studies.
  Results: The cascade ensemble architecture achieved 0.7505 F measure, an
impressive 49.1% error rate reduction, compared to a CNN model that was
previously proposed and evaluated on a selected subset of the 50K articles. On
the full dataset, the cascade ensemble achieved 0.7639 F measure, resulting in
an error rate reduction of 19.7% compared to the best performance reported in a
previous study that used the full dataset.
  Conclusion: Pre-trained contextual encoder neural networks (e.g. SciBERT)
perform better than the models studied previously and manually created search
filters in filtering for scientifically sound relevant articles. The superior
performance achieved by the cascade ensemble is a significant result that
generalizes beyond this task and the dataset, and is analogous to query
optimization in IR and databases.
