  The light transport (LT) of a scene describes how it appears under different
lighting and viewing directions, and complete knowledge of a scene's LT enables
the synthesis of novel views under arbitrary lighting. In this paper, we focus
on image-based LT acquisition, primarily for human bodies within a light stage
setup. We propose a semi-parametric approach to learn a neural representation
of LT that is embedded in the space of a texture atlas of known geometric
properties, and model all non-diffuse and global LT as residuals added to a
physically-accurate diffuse base rendering. In particular, we show how to fuse
previously seen observations of illuminants and views to synthesize a new image
of the same scene under a desired lighting condition from a chosen viewpoint.
This strategy allows the network to learn complex material effects (such as
subsurface scattering) and global illumination, while guaranteeing the physical
correctness of the diffuse LT (such as hard shadows). With this learned LT, one
can relight the scene photorealistically with a directional light or an HDRI
map, synthesize novel views with view-dependent effects, or do both
simultaneously, all in a unified framework using a set of sparse, previously
seen observations. Qualitative and quantitative experiments demonstrate that
our neural LT (NLT) outperforms state-of-the-art solutions for relighting and
view synthesis, without separate treatment for both problems that prior work
requires.
