  Neuro-evolution and neural architecture search algorithms have gained
increasing interest due to the challenges involved in designing optimal
artificial neural networks (ANNs). While these algorithms have been shown to
possess the potential to outperform the best human crafted architectures, a
less common use of them is as a tool for analysis of ANN structural components
and connectivity structures. In this work, we focus on this particular use-case
to develop a rigorous examination and comparison framework for analyzing
recurrent neural networks (RNNs) applied to time series prediction using the
novel neuro-evolutionary process known as Evolutionary eXploration of
Augmenting Memory Models (EXAMM). Specifically, we use our EXAMM-based analysis
to investigate the capabilities of recurrent memory cells and the
generalization ability afforded by various complex recurrent connectivity
patterns that span one or more steps in time, i.e., deep recurrent connections.
EXAMM, in this study, was used to train over 10.56 million RNNs in 5,280
repeated experiments with varying components. While many modern, often
hand-crafted RNNs rely on complex memory cells (which have internal recurrent
connections that only span a single time step) operating under the assumption
that these sufficiently latch information and handle long term dependencies,
our results show that networks evolved with deep recurrent connections perform
significantly better than those without. More importantly, in some cases, the
best performing RNNs consisted of only simple neurons and deep time skip
connections, without any memory cells. These results strongly suggest that
utilizing deep time skip connections in RNNs for time series data prediction
not only deserves further, dedicated study, but also demonstrate the potential
of neuro-evolution as a means to better study, understand, and train effective
RNNs.
