Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: An Analytical Formula of Population Gradient for two-layered ReLU
  network and its Applications in Convergence and Critical Point Analysis   In this paper, we explore theoretical properties of training a two-layered
ReLU network $g(\mathbf{x}; \mathbf{w}) = \sum_{j=1}^K
\sigma(\mathbf{w}_j^T\mathbf{x})$ with centered $d$-dimensional spherical
Gaussian input $\mathbf{x}$ ($\sigma$=ReLU). We train our network with gradient
descent on $\mathbf{w}$ to mimic the output of a teacher network with the same
architecture and fixed parameters $\mathbf{w}^*$. We show that its population
gradient has an analytical formula, leading to interesting theoretical analysis
of critical points and convergence behaviors. First, we prove that critical
points outside the hyperplane spanned by the teacher parameters
("out-of-plane") are not isolated and form manifolds, and characterize in-plane
critical-point-free regions for two ReLU case. On the other hand, convergence
to $\mathbf{w}^*$ for one ReLU node is guaranteed with at least
$(1-\epsilon)/2$ probability, if weights are initialized randomly with standard
deviation upper-bounded by $O(\epsilon/\sqrt{d})$, consistent with empirical
practice. For network with many ReLU nodes, we prove that an infinitesimal
perturbation of weight initialization results in convergence towards
$\mathbf{w}^*$ (or its permutation), a phenomenon known as spontaneous
symmetric-breaking (SSB) in physics. We assume no independence of ReLU
activations. Simulation verifies our findings.

37
The categories of this paper are: supr-con
15
The categories of this paper are: nucl-ex
14
The categories of this paper are: math-ph
13
The categories of this paper are: quant-ph
12
The categories of this paper are: nucl-th
35
The categories of this paper are: mtrl-th
8
The categories of this paper are: hep-ex
0
The categories of this paper are: hep-ph
2
The categories of this paper are: math
32
The categories of this paper are: atom-ph
23
The categories of this paper are: solv-int
10
The categories of this paper are: q-bio
31
The categories of this paper are: ao-sci
19
The categories of this paper are: eess
4
The categories of this paper are: cond-mat
34
The categories of this paper are: chem-ph