  We study the non-convex optimization landscape for maximum likelihood
estimation in the discrete orbit recovery model with Gaussian noise. This model
is motivated by applications in molecular microscopy and image processing,
where each measurement of an unknown object is subject to an independent random
rotation from a rotational group. Equivalently, it is a Gaussian mixture model
where the mixture centers belong to a group orbit.
  We show that fundamental properties of the likelihood landscape depend on the
signal-to-noise ratio and the group structure. At low noise, this landscape is
"benign" for any discrete group, possessing no spurious local optima and only
strict saddle points. At high noise, this landscape may develop spurious local
optima, depending on the specific group. We discuss several positive and
negative examples, and provide a general condition that ensures a globally
benign landscape. For cyclic permutations of coordinates on $\mathbb{R}^d$
(multi-reference alignment), there may be spurious local optima when $d \geq
6$, and we establish a correspondence between these local optima and those of a
surrogate function of the phase variables in the Fourier domain.
  We show that the Fisher information matrix transitions from resembling that
of a single Gaussian in low noise to having a graded eigenvalue structure in
high noise, which is determined by the graded algebra of invariant polynomials
under the group action. In a local neighborhood of the true object, the
likelihood landscape is strongly convex in a reparametrized system of variables
given by a transcendence basis of this polynomial algebra. We discuss
implications for optimization algorithms, including slow convergence of
expectation-maximization, and possible advantages of momentum-based
acceleration and variable reparametrization for first- and second-order descent
methods.
