  We present and evaluate the ExaNeSt Prototype, a liquid-cooled rack prototype
consisting of 256 Xilinx ZU9EG MPSoCs, 4 TBytes of DRAM, 16 TBytes of SSD, and
configurable interconnection 10-Gbps hardware. We developed this testbed in
2016-2019 to validate the flexibility of FPGAs for experimenting with efficient
hardware support for HPC communication among tens of thousands of processors
and accelerators in the quest towards Exascale systems and beyond. We present
our key design choices reagrding overall system architecture, PCBs and runtime
software, and summarize insights resulting from measurement and analysis. Of
particular note, our custom interconnect includes a low-cost low-latency
network interface, offering user-level zero-copy RDMA, which we have tightly
coupled with the ARMv8 processors in the MPSoCs. We have developed a system
software runtime on top of these features, and have been able to run MPI. We
have evaluated our testbed through MPI microbenchmarks, mini, and full MPI
applications. Single hop, one way latency is $1.3$~$\mu$s; approximately
$0.47$~$\mu$s out of these are attributed to network interface and the
user-space library that exposes its functionality to the runtime. Latency over
longer paths increases as expected, reaching $2.55$~$\mu$s for a five-hop path.
Bandwidth tests show that, for a single hop, link utilization reaches $82\%$ of
the theoretical capacity. Microbenchmarks based on MPI collectives reveal that
broadcast latency scales as expected when the number of participating ranks
increases. We also implemented a custom Allreduce accelerator in the network
interface, which reduces the latency of such collectives by up to $88\%$. We
assess performance scaling through weak and strong scaling tests for HPCG,
LAMMPS, and the miniFE mini application; for all these tests, parallelization
efficiency is at least $69\%$, or better.
