  In this paper we analyze the influence that lower layers (file system, OS,
SSD) have on HDFS' ability to extract maximum performance from SSDs on the read
path. We uncover and analyze three surprising performance slowdowns induced by
lower layers that result in HDFS read throughput loss. First, intrinsic
slowdown affects reads from every new file system extent for a variable amount
of time. Second, temporal slowdown appears temporarily and periodically and is
workload-agnostic. Third, in permanent slowdown, some files can individually
and permanently become slower after a period of time. We analyze the impact of
these slowdowns on HDFS and show significant throughput loss. Individually,
each of the slowdowns can cause a read throughput loss of 10-15%. However,
their effect is cumulative. When all slowdowns happen concurrently, read
throughput drops by as much as 30%. We further analyze mitigation techniques
and show that two of the three slowdowns could be addressed via increased IO
request parallelism in the lower layers. Unfortunately, HDFS cannot
automatically adapt to use such additional parallelism. Our results point to a
need for adaptability in storage stacks. The reason is that an access pattern
that maximizes performance in the common case is not necessarily the same one
that can mask performance fluctuations.
