  Recurrent neural networks (RNNs) are known to be universal approximators of
dynamic systems under fairly mild and general assumptions, making them good
tools to process temporal information. However, RNNs usually suffer from the
issues of vanishing and exploding gradients in the standard RNN training.
Reservoir computing (RC), a special RNN where the recurrent weights are
randomized and left untrained, has been introduced to overcome these issues and
has demonstrated superior empirical performance in fields as diverse as natural
language processing and wireless communications especially in scenarios where
training samples are extremely limited. On the contrary, the theoretical
grounding to support this observed performance has not been fully developed at
the same pace. In this work, we show that RNNs can provide universal
approximation of linear time-invariant (LTI) systems. Specifically, we show
that RC can universally approximate a general LTI system. We present a clear
signal processing interpretation of RC and utilize this understanding in the
problem of simulating a generic LTI system through RC. Under this setup, we
analytically characterize the optimal probability distribution function for
generating the recurrent weights of the underlying RNN of the RC. We provide
extensive numerical evaluations to validate the optimality of the derived
optimum distribution of the recurrent weights of the RC for the LTI system
simulation problem. Our work results in clear signal processing-based model
interpretability of RC and provides theoretical explanation for the power of
randomness in setting instead of training RC's recurrent weights. It further
provides a complete optimum analytical characterization for the untrained
recurrent weights, marking an important step towards explainable machine
learning (XML) which is extremely important for applications where training
samples are limited.
