  In theoretical ML, the teacher-student paradigm is often employed as an
effective metaphor for real-life tuition. The above scheme proves particularly
relevant when the student network is overparameterized as compared to the
teacher network. Under these operating conditions, it is tempting to speculate
that the student ability to handle the given task could be eventually stored in
a sub-portion of the whole network. This latter should be to some extent
reminiscent of the frozen teacher structure, according to suitable metrics,
while being approximately invariant across different architectures of the
student candidate network. Unfortunately, state-of-the-art conventional
learning techniques could not help in identifying the existence of such an
invariant subnetwork, due to the inherent degree of non-convexity that
characterizes the examined problem. In this work, we take a leap forward by
proposing a radically different optimization scheme which builds on a spectral
representation of the linear transfer of information between layers. The
gradient is hence calculated with respect to both eigenvalues and eigenvectors
with negligible increase in terms of computational and complexity load, as
compared to standard training algorithms. Working in this framework, we could
isolate a stable student substructure, that mirrors the true complexity of the
teacher in terms of computing neurons, path distribution and topological
attributes. When pruning unimportant nodes of the trained student, as follows a
ranking that reflects the optimized eigenvalues, no degradation in the recorded
performance is seen above a threshold that corresponds to the effective teacher
size. The observed behavior can be pictured as a genuine second-order phase
transition that bears universality traits.
