  While recent work has shown progress on extracting clothed 3D human avatars
from a single image, video, or a set of 3D scans, several limitations remain.
Most methods use a holistic representation to jointly model the body and
clothing, which means that the clothing and body cannot be separated for
applications like virtual try-on. Other methods separately model the body and
clothing, but they require training from a large set of 3D clothed human meshes
obtained from 3D/4D scanners or physics simulations. Our insight is that the
body and clothing have different modeling requirements. While the body is well
represented by a mesh-based parametric 3D model, implicit representations and
neural radiance fields are better suited to capturing the large variety in
shape and appearance present in clothing. Building on this insight, we propose
SCARF (Segmented Clothed Avatar Radiance Field), a hybrid model combining a
mesh-based body with a neural radiance field. Integrating the mesh into the
volumetric rendering in combination with a differentiable rasterizer enables us
to optimize SCARF directly from monocular videos, without any 3D supervision.
The hybrid modeling enables SCARF to (i) animate the clothed body avatar by
changing body poses (including hand articulation and facial expressions), (ii)
synthesize novel views of the avatar, and (iii) transfer clothing between
avatars in virtual try-on applications. We demonstrate that SCARF reconstructs
clothing with higher visual quality than existing methods, that the clothing
deforms with changing body pose and body shape, and that clothing can be
successfully transferred between avatars of different subjects. The code and
models are available at https://github.com/YadiraF/SCARF.
