  We study the problem of optimizing data storage and access costs on the cloud
while ensuring that the desired performance or latency is unaffected. We first
propose an optimizer that optimizes the data placement tier (on the cloud) and
the choice of compression schemes to apply, for given data partitions with
temporal access predictions. Secondly, we propose a model to learn the
compression performance of multiple algorithms across data partitions in
different formats to generate compression performance predictions on the fly,
as inputs to the optimizer. Thirdly, we propose to approach the data
partitioning problem fundamentally differently than the current default in most
data lakes where partitioning is in the form of ingestion batches. We propose
access pattern aware data partitioning and formulate an optimization problem
that optimizes the size and reading costs of partitions subject to access
patterns.
  We study the various optimization problems theoretically as well as
empirically, and provide theoretical bounds as well as hardness results. We
propose a unified pipeline of cost minimization, called SCOPe that combines the
different modules. We extensively compare the performance of our methods with
related baselines from the literature on TPC-H data as well as enterprise
datasets (ranging from GB to PB in volume) and show that SCOPe substantially
improves over the baselines. We show significant cost savings compared to
platform baselines, of the order of 50% to 83% on enterprise Data Lake datasets
that range from terabytes to petabytes in volume.
