Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Why bigger is not always better: on finite and infinite neural networks   Recent work has argued that neural networks can be understood theoretically
by taking the number of channels to infinity, at which point the outputs become
Gaussian process (GP) distributed. However, we note that infinite Bayesian
neural networks lack a key facet of the behaviour of real neural networks: the
fixed kernel, determined only by network hyperparameters, implies that they
cannot do any form of representation learning. The lack of representation or
equivalently kernel learning leads to less flexibility and hence worse
performance, giving a potential explanation for the inferior performance of
infinite networks observed in the literature (e.g. Novak et al. 2019). We give
analytic results characterising the prior over representations and
representation learning in finite deep linear networks. We show empirically
that the representations in SOTA architectures such as ResNets trained with SGD
are much closer to those suggested by our deep linear results than by the
corresponding infinite network. This motivates the introduction of a new class
of network: infinite networks with bottlenecks, which inherit the theoretical
tractability of infinite networks while at the same time allowing
representation learning.

32
The categories of this paper are: atom-ph
3
The categories of this paper are: physics
33
The categories of this paper are: plasm-ph
28
The categories of this paper are: alg-geom
20
The categories of this paper are: acc-phys
8
The categories of this paper are: hep-ex
13
The categories of this paper are: quant-ph
10
The categories of this paper are: q-bio
26
The categories of this paper are: bayes-an
7
The categories of this paper are: hep-th
21
The categories of this paper are: adap-org
24
The categories of this paper are: dg-ga
14
The categories of this paper are: math-ph
6
The categories of this paper are: astro-ph
0
The categories of this paper are: hep-ph
25
The categories of this paper are: patt-sol