  Neural Radiance Field (NeRF) based rendering has attracted growing attention
thanks to its state-of-the-art (SOTA) rendering quality and wide applications
in Augmented and Virtual Reality (AR/VR). However, immersive real-time (> 30
FPS) NeRF based rendering enabled interactions are still limited due to the low
achievable throughput on AR/VR devices. To this end, we first profile SOTA
efficient NeRF algorithms on commercial devices and identify two primary causes
of the aforementioned inefficiency: (1) the uniform point sampling and (2) the
dense accesses and computations of the required embeddings in NeRF.
Furthermore, we propose RT-NeRF, which to the best of our knowledge is the
first algorithm-hardware co-design acceleration of NeRF. Specifically, on the
algorithm level, RT-NeRF integrates an efficient rendering pipeline for largely
alleviating the inefficiency due to the commonly adopted uniform point sampling
method in NeRF by directly computing the geometry of pre-existing points.
Additionally, RT-NeRF leverages a coarse-grained view-dependent computing
ordering scheme for eliminating the (unnecessary) processing of invisible
points. On the hardware level, our proposed RT-NeRF accelerator (1) adopts a
hybrid encoding scheme to adaptively switch between a bitmap- or
coordinate-based sparsity encoding format for NeRF's sparse embeddings, aiming
to maximize the storage savings and thus reduce the required DRAM accesses
while supporting efficient NeRF decoding; and (2) integrates both a
dual-purpose bi-direction adder & search tree and a high-density sparse search
unit to coordinate the two aforementioned encoding formats. Extensive
experiments on eight datasets consistently validate the effectiveness of
RT-NeRF, achieving a large throughput improvement (e.g., 9.7x - 3,201x) while
maintaining the rendering quality as compared with SOTA efficient NeRF
solutions.
