  Solid-state drives (SSDs) are extensively used to deploy persistent data
stores, as they provide low latency random access, high write throughput, high
data density, and low cost. Tree-based data structures are widely used to build
persistent data stores, and indeed they lie at the backbone of many of the data
management systems used in production and research today. In this paper, we
show that benchmarking a persistent tree-based data structure on an SSD is a
complex process, which may easily incur subtle pitfalls that can lead to an
inaccurate performance assessment. At a high-level, these pitfalls stem from
the interaction of complex software running on complex hardware. On one hand,
tree structures implement internal operations that have nontrivial effects on
performance. On the other hand, SSDs employ firmware logic to deal with the
idiosyncrasies of the underlying flash memory, which are well known to lead to
complex performance dynamics. We identify seven benchmarking pitfalls using
RocksDB and WiredTiger, two widespread implementations of an LSM-Tree and a
B+Tree, respectively. We show that such pitfalls can lead to incorrect
measurements of key performance indicators, hinder the reproducibility and the
representativeness of the results, and lead to suboptimal deployments in
production environments. We also provide guidelines on how to avoid these
pitfalls to obtain more reliable performance measurements, and to perform more
thorough and fair comparison among different design points.
