  It is well-known that the parameterized family of functions representable by
fully-connected feedforward neural networks with ReLU activation function is
precisely the class of piecewise linear functions with finitely many pieces. It
is less well-known that for every fixed architecture of ReLU neural network,
the parameter space admits positive-dimensional spaces of symmetries, and hence
the local functional dimension near any given parameter is lower than the
parametric dimension. In this work we carefully define the notion of functional
dimension, show that it is inhomogeneous across the parameter space of ReLU
neural network functions, and continue an investigation - initiated in [14] and
[5] - into when the functional dimension achieves its theoretical maximum. We
also study the quotient space and fibers of the realization map from parameter
space to function space, supplying examples of fibers that are disconnected,
fibers upon which functional dimension is non-constant, and fibers upon which
the symmetry group acts non-transitively.
