  We address the problem of learning person-specific facial priors from a small
number (e.g., 20) of portrait photos of the same person. This enables us to
edit this specific person's facial appearance, such as expression and lighting,
while preserving their identity and high-frequency facial details. Key to our
approach, which we dub DiffusionRig, is a diffusion model conditioned on, or
"rigged by," crude 3D face models estimated from single in-the-wild images by
an off-the-shelf estimator. On a high level, DiffusionRig learns to map
simplistic renderings of 3D face models to realistic photos of a given person.
Specifically, DiffusionRig is trained in two stages: It first learns generic
facial priors from a large-scale face dataset and then person-specific priors
from a small portrait photo collection of the person of interest. By learning
the CGI-to-photo mapping with such personalized priors, DiffusionRig can "rig"
the lighting, facial expression, head pose, etc. of a portrait photo,
conditioned only on coarse 3D models while preserving this person's identity
and other high-frequency characteristics. Qualitative and quantitative
experiments show that DiffusionRig outperforms existing approaches in both
identity preservation and photorealism. Please see the project website:
https://diffusionrig.github.io for the supplemental material, video, code, and
data.
