  As memory capacity has outstripped TLB coverage, large data applications
suffer from frequent page table walks. We investigate two complementary
techniques for addressing this cost: reducing the number of accesses required
and reducing the latency of each access. The first approach is accomplished by
opportunistically "flattening" the page table: merging two levels of
traditional 4KB page table nodes into a single 2MB node, thereby reducing the
table's depth and the number of indirections required to search it. The second
is accomplished by biasing the cache replacement algorithm to keep page table
entries during periods of high TLB miss rates, as these periods also see high
data miss rates and are therefore more likely to benefit from having the
smaller page table in the cache than to suffer from increased data cache
misses.
  We evaluate these approaches for both native and virtualized systems and
across a range of realistic memory fragmentation scenarios, describe the
limited changes needed in our kernel implementation and hardware design,
identify and address challenges related to self-referencing page tables and
kernel memory allocation, and compare results across server and mobile systems
using both academic and industrial simulators for robustness.
  We find that flattening does reduce the number of accesses required on a page
walk (to 1.0), but its performance impact (+2.3%) is small due to Page Walker
Caches (already 1.5 accesses). Prioritizing caching has a larger effect
(+6.8%), and the combination improves performance by +9.2%. Flattening is more
effective on virtualized systems (4.4 to 2.8 accesses, +7.1% performance), due
to 2D page walks. By combining the two techniques we demonstrate a
state-of-the-art +14.0% performance gain and -8.7% dynamic cache energy and
-4.7% dynamic DRAM energy for virtualized execution with very simple hardware
and software changes.
