  Previous work has established that RNNs with an unbounded activation function
have the capacity to count exactly. However, it has also been shown that RNNs
are challenging to train effectively and generally do not learn exact counting
behaviour. In this paper, we focus on this problem by studying the simplest
possible RNN, a linear single-cell network. We conduct a theoretical analysis
of linear RNNs and identify conditions for the models to exhibit exact counting
behaviour. We provide a formal proof that these conditions are necessary and
sufficient. We also conduct an empirical analysis using tasks involving a
Dyck-1-like Balanced Bracket language under two different settings. We observe
that linear RNNs generally do not meet the necessary and sufficient conditions
for counting behaviour when trained with the standard approach. We investigate
how varying the length of training sequences and utilising different target
classes impacts model behaviour during training and the ability of linear RNN
models to effectively approximate the indicator conditions.
