  In large-scale unconstrained optimization algorithms such as limited memory
BFGS (LBFGS), a common subproblem is a line search minimizing the loss function
along a descent direction. Commonly used line searches iteratively find an
approximate solution for which the Wolfe conditions are satisfied, typically
requiring multiple function and gradient evaluations per line search, which is
expensive in parallel due to communication requirements. In this paper we
propose a new line search approach for cases where the loss function is
analytic, as in least squares regression, logistic regression, or low rank
matrix factorization. We approximate the loss function by a truncated Taylor
polynomial, whose coefficients may be computed efficiently in parallel with
less communication than evaluating the gradient, after which this polynomial
may be minimized with high accuracy in a neighbourhood of the expansion point.
Our Polynomial Expansion Line Search (PELS) was implemented in the Apache Spark
framework and used to accelerate the training of a logistic regression model on
binary classification datasets from the LIBSVM repository with LBFGS and the
Nonlinear Conjugate Gradient (NCG) method. In large-scale numerical experiments
in parallel on a 16-node cluster with 256 cores using the URL, KDDA, and KDDB
datasets, the PELS approach produced significant convergence improvements
compared to the use of classical Wolfe line searches. For example, to reach the
final training label prediction accuracies, LBFGS using PELS had speedup
factors of 1.8--2 over LBFGS using a Wolfe line search, measured by both the
number of iterations and the time required, due to the better accuracy of step
sizes computed in the line search. PELS has the potential to significantly
accelerate large-scale regression and factorization computations, and is
applicable to continuous optimization problems with smooth loss functions.
