  Recent works in spatiotemporal radiance fields can produce photorealistic
free-viewpoint videos. However, they are inherently unsuitable for interactive
streaming scenarios (e.g. video conferencing, telepresence) because have an
inevitable lag even if the training is instantaneous. This is because these
approaches consume videos and thus have to buffer chunks of frames (often
seconds) before processing. In this work, we take a step towards interactive
streaming via a frame-by-frame approach naturally free of lag. Conventional
wisdom believes that per-frame NeRFs are impractical due to prohibitive
training costs and storage. We break this belief by introducing Incremental
Neural Videos (INV), a per-frame NeRF that is efficiently trained and
streamable. We designed INV based on two insights: (1) Our main finding is that
MLPs naturally partition themselves into Structure and Color Layers, which
store structural and color/texture information respectively. (2) We leverage
this property to retain and improve upon knowledge from previous frames, thus
amortizing training across frames and reducing redundant learning. As a result,
with negligible changes to NeRF, INV can achieve good qualities (>28.6db) in
8min/frame. It can also outperform prior SOTA in 19% less training time.
Additionally, our Temporal Weight Compression reduces the per-frame size to
0.3MB/frame (6.6% of NeRF). More importantly, INV is free from buffer lag and
is naturally fit for streaming. While this work does not achieve real-time
training, it shows that incremental approaches like INV present new
possibilities in interactive 3D streaming. Moreover, our discovery of natural
information partition leads to a better understanding and manipulation of MLPs.
Code and dataset will be released soon.
