  Modern enterprise servers are increasingly embracing tiered memory systems
with a combination of low latency DRAMs and large capacity but high latency
non-volatile main memories (NVMMs) such as Intel's Optane DC PMM. Prior works
have focused on efficient placement and migration of data on a tiered memory
system, but have not studied the optimal placement of page tables.
  Explicit and efficient placement of page tables is crucial for large memory
footprint applications with high TLB miss rates because they incur dramatically
higher page walk latency when page table pages are placed in NVMM. We show that
(i) page table pages can end up on NVMM even when enough DRAM memory is
available and (ii) page table pages that spill over to NVMM due to DRAM memory
pressure are not migrated back later when memory is available in DRAM.
  We study the performance impact of page table placement in a tiered memory
system and propose an efficient and transparent page table management technique
that (i) applies different placement policies for data and page table pages,
(ii) introduces a differentiating policy for page table pages by placing a
small but critical part of the page table in DRAM, and (iii) dynamically and
judiciously manages the rest of the page table by transparently migrating the
page table pages between DRAM and NVMM. Our implementation on a real system
equipped with Intel's Optane NVMM running Linux reduces the page table walk
cycles by 12% and total cycles by 20% on an average. This improves the runtime
by 20% on an average for a set of synthetic and real-world large memory
footprint applications when compared with various default Linux kernel
techniques.
