  Recurrent Neural Networks (RNNs) were recently successfully used to model the
way neural activity drives task-related behavior in animals, operating under
the implicit assumption that the obtained solutions are universal. Observations
in both neuroscience and machine learning challenge this assumption. Animals
can approach a given task with a variety of strategies, and training machine
learning algorithms introduces the phenomenon of underspecification. These
observations imply that every task is associated with a space of solutions. To
date, the structure of this space is not understood, limiting the approach of
comparing RNNs with neural data. Here, we characterize the space of solutions
associated with various tasks. We first study a simple two-neuron network on a
task that leads to multiple solutions. We trace the nature of the final
solution back to the network's initial connectivity and identify discrete
dynamical regimes that underlie this diversity. We then examine three
neuroscience-inspired tasks: Delayed and interval discrimination, and Time
reproduction. For each task, we find a rich set of solutions. Variability can
be found directly in the neural activity of the networks, and additionally by
testing the trained networks' ability to extrapolate, as a perturbation to a
system often reveals hidden structure. Furthermore, we relate extrapolation
patterns to specific dynamical objects and effective algorithms found by the
networks. We introduce a tool to derive the reduced dynamics of networks by
generating a compact directed graph describing the essence of the dynamics with
regards to behavioral inputs and outputs. Using this representation, we can
partition the solutions to each task into a handful of types and partially
predict them from neural features. Our results shed light on the concept of the
space of solutions and its uses in Machine learning and in Neuroscience.
