  FlexTOE is a flexible, yet high-performance TCP offload engine (TOE) to
SmartNICs. FlexTOE eliminates almost all host data-path TCP processing and is
fully customizable. FlexTOE interoperates well with other TCP stacks, is robust
under adverse network conditions, and supports POSIX sockets.
  FlexTOE focuses on data-path offload of established connections, avoiding
complex control logic and packet buffering in the NIC. FlexTOE leverages
fine-grained parallelization of the TCP data-path and segment reordering for
high performance on wimpy SmartNIC architectures, while remaining flexible via
a modular design. We compare FlexTOE on an Agilio-CX40 to host TCP stacks Linux
and TAS, and to the Chelsio Terminator TOE. We find that Memcached scales up to
38% better on FlexTOE versus TAS, while saving up to 81% host CPU cycles versus
Chelsio. FlexTOE provides competitive performance for RPCs, even with wimpy
SmartNICs. FlexTOE cuts 99.99th-percentile RPC RTT by 3.2$\times$ and 50%
versus Chelsio and TAS, respectively. FlexTOE's data-path parallelism
generalizes across hardware architectures, improving single connection RPC
throughput up to 2.4$\times$ on x86 and 4$\times$ on BlueField. FlexTOE
supports C and XDP programs written in eBPF. It allows us to implement popular
data center transport features, such as TCP tracing, packet filtering and
capture, VLAN stripping, flow classification, firewalling, and connection
splicing.
