  Multi-tenancy is essential for unleashing SmartNIC's potential in
datacenters. Our systematic analysis in this work shows that existing on-path
SmartNICs have resource multiplexing limitations. For example, existing
solutions lack multi-tenancy capabilities such as performance isolation and QoS
provisioning for compute and IO resources. Compared to standard NIC data paths
with a well-defined set of offloaded functions, unpredictable execution times
of SmartNIC kernels make conventional approaches for multi-tenancy and QoS
insufficient. We fill this gap with OSMOSIS, a SmartNICs resource manager
co-design. OSMOSIS extends existing OS mechanisms to enable dynamic hardware
resource multiplexing on top of the on-path packet processing data plane. We
implement OSMOSIS within an open-source RISC-V-based 400Gbit/s SmartNIC. Our
performance results demonstrate that OSMOSIS fully supports multi-tenancy and
enables broader adoption of SmartNICs in datacenters with low overhead.
