text: We present Nesterov-type acceleration techniques for Alternating Least Squares (ALS) methods applied to canonical tensor decomposition. While Nesterov acceleration turns gradient descent into an optimal first-order method for convex problems by adding a momentum term with a specific weight sequence, a direct application of this method and weight sequence to ALS results in erratic convergence behaviour or divergence. This is so because the tensor decomposition problem is non-convex and ALS is accelerated instead of gradient descent. We investigate how line search or restart mechanisms can be used to obtain effective acceleration. We first consider a cubic line search (LS) strategy for determining the momentum weight, showing numerically that the combined Nesterov-ALS-LS approach is competitive with or superior to other recently developed nonlinear acceleration techniques for ALS, including acceleration by nonlinear conjugate gradients (NCG) and LBFGS. As an alternative, we consider various restarting techniques, some of which are inspired by previously proposed restarting mechanisms for Nesterov's accelerated gradient method. We study how two key parameters, the momentum weight and the restart condition, should be set. Our extensive empirical results show that the Nesterov-accelerated ALS methods with restart can be dramatically more efficient than the stand-alone ALS or Nesterov accelerated gradient method, when problems are ill-conditioned or accurate solutions are required. The resulting methods perform competitively with or superior to existing acceleration methods for ALS, and additionally enjoy the benefit of being much simpler and easier to implement. On a large and ill-conditioned 71 x 1000 x 900 tensor consisting of readings from chemical sensors used for tracking hazardous gases, the restarted Nesterov-ALS method outperforms any of the existing methods by a large factor.
  @cite_13: Optimization plays a key role in machine learning. Recently, stochastic second-order methods have attracted much attention due to their low computational cost in each iteration. However, these algorithms might perform poorly especially if it is hard to approximate the Hessian well and efficiently. As far as we know, there is no effective way to handle this problem. In this paper, we resort to Nesterov's acceleration technique to improve the convergence performance of a class of second-order methods called approximate Newton. We give a theoretical analysis that Nesterov's acceleration technique can improve the convergence performance for approximate Newton just like for first-order methods. We accordingly propose an accelerated regularized sub-sampled Newton. Our accelerated algorithm performs much better than the original regularized sub-sampled Newton in experiments, which validates our theory empirically. Besides, the accelerated regularized sub-sampled Newton has good performance comparable to or even better than classical algorithms.
  @cite_12: Alternating direction methods are a common tool for general mathematical programming and optimization. These methods have become particularly important in the field of variational image processing, which frequently requires the minimization of nondifferentiable objectives. This paper considers accelerated (i.e., fast) variants of two common alternating direction methods: the alternating direction method of multipliers (ADMM) and the alternating minimization algorithm (AMA). The proposed acceleration is of the form first proposed by Nesterov for gradient descent methods. In the case that the objective function is strongly convex, global convergence bounds are provided for both classical and accelerated variants of the methods. Numerical examples are presented to demonstrate the superior performance of the fast methods for a wide variety of problems.
summary: Nesterov's technique has also been used to accelerate non-gradient based methods. In @cite_12 it was used to accelerate ADMM, and @cite_13 used it to accelerate an approximate Newton method.