Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Accelerating Natural Gradient with Higher-Order Invariance   An appealing property of the natural gradient is that it is invariant to
arbitrary differentiable reparameterizations of the model. However, this
invariance property requires infinitesimal steps and is lost in practical
implementations with small but finite step sizes. In this paper, we study
invariance properties from a combined perspective of Riemannian geometry and
numerical differential equation solving. We define the order of invariance of a
numerical method to be its convergence order to an invariant solution. We
propose to use higher-order integrators and geodesic corrections to obtain more
invariant optimization trajectories. We prove the numerical convergence
properties of geodesic corrected updates and show that they can be as
computationally efficient as plain natural gradient. Experimentally, we
demonstrate that invariance leads to faster optimization and our techniques
improve on traditional natural gradient in deep neural network training and
natural policy gradient for reinforcement learning.

10
The categories of this paper are: q-bio
24
The categories of this paper are: dg-ga
30
The categories of this paper are: q-alg
18
The categories of this paper are: econ
6
The categories of this paper are: astro-ph
13
The categories of this paper are: quant-ph
3
The categories of this paper are: physics
2
The categories of this paper are: math
17
The categories of this paper are: q-fin
26
The categories of this paper are: bayes-an
21
The categories of this paper are: adap-org
36
The categories of this paper are: cmp-lg
15
The categories of this paper are: nucl-ex
29
The categories of this paper are: funct-an
20
The categories of this paper are: acc-phys
8
The categories of this paper are: hep-ex