  Learning about the three-dimensional world from two-dimensional images is a
fundamental problem in computer vision. An ideal neural network architecture
for such tasks would leverage the fact that objects can be rotated and
translated in three dimensions to make predictions about novel images. However,
imposing SO(3)-equivariance on two-dimensional inputs is difficult because the
group of three-dimensional rotations does not have a natural action on the
two-dimensional plane. Specifically, it is possible that an element of SO(3)
will rotate an image out of plane. We show that an algorithm that learns a
three-dimensional representation of the world from two dimensional images must
satisfy certain geometric consistency properties which we formulate as
SO(2)-equivariance constraints. We use the induced and restricted
representations of SO(2) on SO(3) to construct and classify architectures which
satisfy these geometric consistency constraints. We prove that any architecture
which respects said consistency constraints can be realized as an instance of
our construction. We show that three previously proposed neural architectures
for 3D pose prediction are special cases of our construction. We propose a new
algorithm that is a learnable generalization of previously considered methods.
We test our architecture on three pose predictions task and achieve SOTA
results on both the PASCAL3D+ and SYMSOL pose estimation tasks.
