  Optimal Learning Machines (OLM) are systems that extract maximally
informative representation of the environment they are in contact with, or of
the data they are presented. It has recently been suggested that these systems
are characterised by an exponential distribution of energy levels. In order to
understand the peculiar properties of OLM within a broader framework, I
consider an ensemble of optimisation problems over functions of many variables,
part of which describe a sub-system and the rest account for its interaction
with a random environment. The number of states of the sub-system with a given
value of the objective function obeys a stretched exponential distribution,
with exponent $\gamma$, and the interaction part is drawn at random from the
same distribution, independently for each configuration of the whole system.
Systems with $\gamma=1$ then correspond to OLM, and we find that they sit at
the boundary between two regions with markedly different properties. For all
$\gamma>0$ the system exhibits a freezing phase transition. The transition is
discontinuous for $\gamma<1$ and it is continuous for $\gamma>1$. The region
$\gamma>1$ corresponds to learnable energy landscapes and the behaviour of the
sub-system becomes predictable as the size of the environment exceeds a
critical threshold. For $\gamma<1$, instead, the energy landscape is
unlearnable and the behaviour of the system becomes more and more unpredictable
as the size of the environment increases. Sub-systems with $\gamma=1$ (OLM)
feature a behaviour which is independent of the relative size of the
environment. This is consistent with the expectation that efficient
representations should be largely independent of the level of detail of the
description of the environment.
