  The geometric structure of an optimization landscape is argued to be
fundamentally important to support the success of deep neural network learning.
A direct computation of the landscape beyond two layers is hard. Therefore, to
capture the global view of the landscape, an interpretable model of the
network-parameter (or weight) space must be established. However, the model is
lacking so far. Furthermore, it remains unknown what the landscape looks like
for deep networks of binary synapses, which plays a key role in robust and
energy efficient neuromorphic computation. Here, we propose a statistical
mechanics framework by directly building a least structured model of the
high-dimensional weight space, considering realistic structured data,
stochastic gradient descent training, and the computational depth of neural
networks. We also consider whether the number of network parameters outnumbers
the number of supplied training data, namely, over- or under-parametrization.
Our least structured model reveals that the weight spaces of the
under-parametrization and over-parameterization cases belong to the same class,
in the sense that these weight spaces are well-connected without any
hierarchical clustering structure. In contrast, the shallow-network has a
broken weight space, characterized by a discontinuous phase transition, thereby
clarifying the benefit of depth in deep learning from the angle of high
dimensional geometry. Our effective model also reveals that inside a deep
network, there exists a liquid-like central part of the architecture in the
sense that the weights in this part behave as randomly as possible, providing
algorithmic implications. Our data-driven model thus provides a statistical
mechanics insight about why deep learning is unreasonably effective in terms of
the high-dimensional weight space, and how deep networks are different from
shallow ones.
