Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Stochastic Training of Neural Networks via Successive Convex
  Approximations   This paper proposes a new family of algorithms for training neural networks
(NNs). These are based on recent developments in the field of non-convex
optimization, going under the general name of successive convex approximation
(SCA) techniques. The basic idea is to iteratively replace the original
(non-convex, highly dimensional) learning problem with a sequence of (strongly
convex) approximations, which are both accurate and simple to optimize.
Differently from similar ideas (e.g., quasi-Newton algorithms), the
approximations can be constructed using only first-order information of the
neural network function, in a stochastic fashion, while exploiting the overall
structure of the learning problem for a faster convergence. We discuss several
use cases, based on different choices for the loss function (e.g., squared loss
and cross-entropy loss), and for the regularization of the NN's weights. We
experiment on several medium-sized benchmark problems, and on a large-scale
dataset involving simulated physical data. The results show how the algorithm
outperforms state-of-the-art techniques, providing faster convergence to a
better minimum. Additionally, we show how the algorithm can be easily
parallelized over multiple computational units without hindering its
performance. In particular, each computational unit can optimize a tailored
surrogate function defined on a randomly assigned subset of the input
variables, whose dimension can be selected depending entirely on the available
computational power.

6
The categories of this paper are: astro-ph
31
The categories of this paper are: ao-sci
21
The categories of this paper are: adap-org
3
The categories of this paper are: physics
26
The categories of this paper are: bayes-an
8
The categories of this paper are: hep-ex
15
The categories of this paper are: nucl-ex
34
The categories of this paper are: chem-ph
28
The categories of this paper are: alg-geom
35
The categories of this paper are: mtrl-th
17
The categories of this paper are: q-fin
9
The categories of this paper are: nlin
0
The categories of this paper are: hep-ph
25
The categories of this paper are: patt-sol
14
The categories of this paper are: math-ph
22
The categories of this paper are: chao-dyn