  We introduce a method for assigning photorealistic relightable materials to
3D shapes in an automatic manner. Our method takes as input a photo exemplar of
a real object and a 3D object with segmentation, and uses the exemplar to guide
the assignment of materials to the parts of the shape, so that the appearance
of the resulting shape is as similar as possible to the exemplar. To accomplish
this goal, our method combines an image translation neural network with a
material assignment neural network. The image translation network translates
the color from the exemplar to a projection of the 3D shape and the part
segmentation from the projection to the exemplar. Then, the material prediction
network assigns materials from a collection of realistic materials to the
projected parts, based on the translated images and perceptual similarity of
the materials. One key idea of our method is to use the translation network to
establish a correspondence between the exemplar and shape projection, which
allows us to transfer materials between objects with diverse structures.
Another key idea of our method is to use the two pairs of (color, segmentation)
images provided by the image translation to guide the material assignment,
which enables us to ensure the consistency in the assignment. We demonstrate
that our method allows us to assign materials to shapes so that their
appearances better resemble the input exemplars, improving the quality of the
results over the state-of-the-art method, and allowing us to automatically
create thousands of shapes with high-quality photorealistic materials. Code and
data for this paper are available at https://github.com/XiangyuSu611/TMT.
