Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Gradient Starvation: A Learning Proclivity in Neural Networks   We identify and formalize a fundamental gradient descent phenomenon resulting
in a learning proclivity in over-parameterized neural networks. Gradient
Starvation arises when cross-entropy loss is minimized by capturing only a
subset of features relevant for the task, despite the presence of other
predictive features that fail to be discovered. This work provides a
theoretical explanation for the emergence of such feature imbalance in neural
networks. Using tools from Dynamical Systems theory, we identify simple
properties of learning dynamics during gradient descent that lead to this
imbalance, and prove that such a situation can be expected given certain
statistical structure in training data. Based on our proposed formalism, we
develop guarantees for a novel regularization method aimed at decoupling
feature learning dynamics, improving accuracy and robustness in cases hindered
by gradient starvation. We illustrate our findings with simple and real-world
out-of-distribution (OOD) generalization experiments.

32
The categories of this paper are: atom-ph
29
The categories of this paper are: funct-an
0
The categories of this paper are: hep-ph
34
The categories of this paper are: chem-ph
15
The categories of this paper are: nucl-ex
19
The categories of this paper are: eess
8
The categories of this paper are: hep-ex
31
The categories of this paper are: ao-sci
27
The categories of this paper are: comp-gas
21
The categories of this paper are: adap-org
7
The categories of this paper are: hep-th
24
The categories of this paper are: dg-ga
33
The categories of this paper are: plasm-ph
30
The categories of this paper are: q-alg
11
The categories of this paper are: hep-lat
35
The categories of this paper are: mtrl-th