  This paper addresses the following question of neural network
identifiability: Does the input-output map realized by a feed-forward neural
network with respect to a given nonlinearity uniquely specify the network
architecture, weights, and biases? Existing literature on the subject Sussman
1992, Albertini, Sontag et al. 1993, Fefferman 1994 suggests that the answer
should be yes, up to certain symmetries induced by the nonlinearity, and
provided the networks under consideration satisfy certain "genericity
conditions". The results in Sussman 1992 and Albertini, Sontag et al. 1993
apply to networks with a single hidden layer and in Fefferman 1994 the networks
need to be fully connected. In an effort to answer the identifiability question
in greater generality, we derive necessary genericity conditions for the
identifiability of neural networks of arbitrary depth and connectivity with an
arbitrary nonlinearity. Moreover, we construct a family of nonlinearities for
which these genericity conditions are minimal, i.e., both necessary and
sufficient. This family is large enough to approximate many commonly
encountered nonlinearities to within arbitrary precision in the uniform norm.
