  In the era of big data and cloud computing, large amounts of data are
generated from user applications and need to be processed in the datacenter.
Data-parallel computing frameworks, such as Apache Spark, are widely used to
perform such data processing at scale. Specifically, Spark leverages
distributed memory to cache the intermediate results, represented as Resilient
Distributed Datasets (RDDs). This gives Spark an advantage over other parallel
frameworks for implementations of iterative machine learning and data mining
algorithms, by avoiding repeated computation or hard disk accesses to retrieve
RDDs. By default, caching decisions are left at the programmer's discretion,
and the LRU policy is used for evicting RDDs when the cache is full. However,
when the objective is to minimize total work, LRU is woefully inadequate,
leading to arbitrarily suboptimal caching decisions. In this paper, we design
an algorithm for multi-stage big data processing platforms to adaptively
determine and cache the most valuable intermediate datasets that can be reused
in the future. Our solution automates the decision of which RDDs to cache: this
amounts to identifying nodes in a direct acyclic graph (DAG) representing
computations whose outputs should persist in the memory. Our experiment results
show that our proposed cache optimization solution can improve the performance
of machine learning applications on Spark decreasing the total work to
recompute RDDs by 12%.
