  Random Forests (RF) are among the state-of-the-art in many machine learning
applications. With the ongoing integration of ML models into everyday life, the
deployment and continuous application of models becomes more and more an
important issue. Hence, small models which offer good predictive performance
but use small amounts of memory are required. Ensemble pruning is a standard
technique to remove unnecessary classifiers from an ensemble to reduce the
overall resource consumption and sometimes even improve the performance of the
original ensemble. In this paper, we revisit ensemble pruning in the context of
`modernly' trained Random Forests where trees are very large. We show that the
improvement effects of pruning diminishes for ensembles of large trees but that
pruning has an overall better accuracy-memory trade-off than RF. However,
pruning does not offer fine-grained control over this trade-off because it
removes entire trees from the ensemble. To further improve the accuracy-memory
trade-off we present a simple, yet surprisingly effective algorithm that
refines the predictions in the leaf nodes in the forest via stochastic gradient
descent. We evaluate our method against 7 state-of-the-art pruning methods and
show that our method outperforms the other methods on 11 of 16 datasets with a
statistically significant better accuracy-memory trade-off compared to most
methods. We conclude our experimental evaluation with a case study showing that
our method can be applied in a real-world setting.
