text: Various applications in signal processing and machine learning give rise to highly structured spectral optimization problems characterized by low-rank solutions. Two important examples that motivate this work are optimization problems from phase retrieval and from blind deconvolution, which are designed to yield rank-1 solutions. An algorithm is described that is based on solving a certain constrained eigenvalue optimization problem that corresponds to the gauge dual which, unlike the more typical Lagrange dual, has an especially simple constraint. The dominant cost at each iteration is the computation of rightmost eigenpairs of a Hermitian operator. A range of numerical examples illustrate the scalability of the approach.
  @cite_8: A new nonmonotone line search algorithm is proposed and analyzed. In our scheme, we require that an average of the successive function values decreases, while the traditional nonmonotone approach of Grippo, Lampariello, and Lucidi [SIAM J. Numer. Anal., 23 (1986), pp. 707--716] requires that a maximum of recent function values decreases. We prove global convergence for nonconvex, smooth functions, and R-linear convergence for strongly convex functions. For the L-BFGS method and the unconstrained optimization problems in the CUTE library, the new nonmonotone line search algorithm used fewer function and gradient evaluations, on average, than either the monotone or the traditional nonmonotone scheme.
  @cite_18: We present a hybrid algorithm for optimizing a convex, smooth function over the cone of positive semidefinite matrices. Our algorithm converges to the global optimal solution and can be used to solve general large-scale semidefinite programs and hence can be readily applied to a variety of machine learning problems. We show experimental results on three machine learning problems. Our approach outperforms state-of-the-art algorithms.
  @cite_11: We propose an algorithm for approximately maximizing a concave function over the bounded semi-definite cone, which produces sparse solutions. Sparsity for SDP corresponds to low rank matrices, and is a important property for both computational as well as learning theoretic reasons. As an application, building on Aaronson's recent work, we derive a linear time algorithm for Quantum State Tomography.
summary: Other researchers have recognized the need for algorithms with low per-iteration costs that scale well for large-scale, low-rank spectral optimization problems. Notable efforts include @cite_11 , @cite_18 , and @cite_8 , who advocate variations of the Frank-Wolfe (FW) method to solve some version of the problem where @math is a differentiable function. For example, the choice @math yields a problem related to . The asymmetric version of the problem with @math is easily accommodated by simply replacing the above constraints. For simplicity, here we focus on the symmetric case, though our approach applies equally to the asymmetric case. The main benefit of using FW for this problem is that each iteration requires only a rightmost eigenvalue of the gradient @math , and therefore has the same per-iteration cost of the method that we consider, which requires a rightmost eigenvalue of the same-sized matrix @math . The same Krylov-based eigensolvers apply in both cases.