  The recent research explosion around implicit neural representations, such as
NeRF, shows that there is immense potential for implicitly storing high-quality
scene and lighting information in compact neural networks. However, one major
limitation preventing the use of NeRF in real-time rendering applications is
the prohibitive computational cost of excessive network evaluations along each
view ray, requiring dozens of petaFLOPS. In this work, we bring compact neural
representations closer to practical rendering of synthetic content in real-time
applications, such as games and virtual reality. We show that the number of
samples required for each view ray can be significantly reduced when samples
are placed around surfaces in the scene without compromising image quality. To
this end, we propose a depth oracle network that predicts ray sample locations
for each view ray with a single network evaluation. We show that using a
classification network around logarithmically discretized and spherically
warped depth values is essential to encode surface locations rather than
directly estimating depth. The combination of these techniques leads to DONeRF,
our compact dual network design with a depth oracle network as its first step
and a locally sampled shading network for ray accumulation. With DONeRF, we
reduce the inference costs by up to 48x compared to NeRF when conditioning on
available ground truth depth information. Compared to concurrent acceleration
methods for raymarching-based neural representations, DONeRF does not require
additional memory for explicit caching or acceleration structures, and can
render interactively (20 frames per second) on a single GPU.
