Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: A Random Matrix Theory Approach to Damping in Deep Learning   We conjecture that the inherent difference in generalisation between adaptive
and non-adaptive gradient methods in deep learning stems from the increased
estimation noise in the flattest directions of the true loss surface. We
demonstrate that typical schedules used for adaptive methods (with low
numerical stability or damping constants) serve to bias relative movement
towards flat directions relative to sharp directions, effectively amplifying
the noise-to-signal ratio and harming generalisation. We further demonstrate
that the numerical damping constant used in these methods can be decomposed
into a learning rate reduction and linear shrinkage of the estimated curvature
matrix. We then demonstrate significant generalisation improvements by
increasing the shrinkage coefficient, closing the generalisation gap entirely
in both logistic regression and several deep neural network experiments.
Extending this line further, we develop a novel random matrix theory based
damping learner for second order optimiser inspired by linear shrinkage
estimation. We experimentally demonstrate our learner to be very insensitive to
the initialised value and to allow for extremely fast convergence in
conjunction with continued stable training and competitive generalisation.

36
The categories of this paper are: cmp-lg
20
The categories of this paper are: acc-phys
22
The categories of this paper are: chao-dyn
31
The categories of this paper are: ao-sci
24
The categories of this paper are: dg-ga
4
The categories of this paper are: cond-mat
13
The categories of this paper are: quant-ph
2
The categories of this paper are: math
11
The categories of this paper are: hep-lat
19
The categories of this paper are: eess
8
The categories of this paper are: hep-ex
32
The categories of this paper are: atom-ph
33
The categories of this paper are: plasm-ph
12
The categories of this paper are: nucl-th
26
The categories of this paper are: bayes-an
34
The categories of this paper are: chem-ph