  In this paper, we propose a general procedure for establishing the geometric
landscape connections of a Riemannian optimization problem under the embedded
and quotient geometries. By applying the general procedure to the fixed-rank
positive semidefinite (PSD) and general matrix optimization, we establish an
exact Riemannian gradient connection under two geometries at every point on the
manifold and sandwich inequalities between the spectra of Riemannian Hessians
at Riemannian first-order stationary points (FOSPs). These results immediately
imply an equivalence on the sets of Riemannian FOSPs, Riemannian second-order
stationary points (SOSPs), and strict saddles of fixed-rank matrix optimization
under the embedded and the quotient geometries. To the best of our knowledge,
this is the first geometric landscape connection between the embedded and the
quotient geometries for fixed-rank matrix optimization and it provides a
concrete example of how these two geometries are connected in Riemannian
optimization. In addition, the effects of the Riemannian metric and quotient
structure on the landscape connection are discussed. We also observe an
algorithmic connection between two geometries with some specific Riemannian
metrics in fixed-rank matrix optimization: there is an equivalence between
gradient flows under two geometries with shared spectra of Riemannian Hessians.
A number of novel ideas and technical ingredients including a unified treatment
for different Riemannian metrics, novel metrics for the Stiefel manifold, and
new horizontal space representations under quotient geometries are developed to
obtain our results. The results in this paper deepen our understanding of
geometric and algorithmic connections of Riemannian optimization under
different Riemannian geometries and provide a few new theoretical insights to
unanswered questions in the literature.
