  The increasing demand for memory in hyperscale applications has led to memory
becoming a large portion of the overall datacenter spend. The emergence of
coherent interfaces like CXL enables main memory expansion and offers an
efficient solution to this problem. In such systems, the main memory can
constitute different memory technologies with varied characteristics. In this
paper, we characterize memory usage patterns of a wide range of datacenter
applications across the server fleet of Meta. We, therefore, demonstrate the
opportunities to offload colder pages to slower memory tiers for these
applications. Without efficient memory management, however, such systems can
significantly degrade performance.
  We propose a novel OS-level application-transparent page placement mechanism
(TPP) for CXL-enabled memory. TPP employs a lightweight mechanism to identify
and place hot/cold pages to appropriate memory tiers. It enables a proactive
page demotion from local memory to CXL-Memory. This technique ensures a memory
headroom for new page allocations that are often related to request processing
and tend to be short-lived and hot. At the same time, TPP can promptly promote
performance-critical hot pages trapped in the slow CXL-Memory to the fast local
memory, while minimizing both sampling overhead and unnecessary migrations. TPP
works transparently without any application-specific knowledge and can be
deployed globally as a kernel release.
  We evaluate TPP in the production server fleet with early samples of new x86
CPUs with CXL 1.1 support. TPP makes a tiered memory system performant as an
ideal baseline (<1% gap) that has all the memory in the local tier. It is 18%
better than today's Linux, and 5-17% better than existing solutions including
NUMA Balancing and AutoTiering. Most of the TPP patches have been merged in the
Linux v5.18 release.
