  Counter machines have achieved a newfound relevance to the field of natural
language processing (NLP): recent work suggests some strong-performing
recurrent neural networks utilize their memory as counters. Thus, one potential
way to understand the success of these networks is to revisit the theory of
counter computation. Therefore, we study the abilities of real-time counter
machines as formal grammars, focusing on formal properties that are relevant
for NLP models. We first show that several variants of the counter machine
converge to express the same class of formal languages. We also prove that
counter languages are closed under complement, union, intersection, and many
other common set operations. Next, we show that counter machines cannot
evaluate boolean expressions, even though they can weakly validate their
syntax. This has implications for the interpretability and evaluation of neural
network systems: successfully matching syntactic patterns does not guarantee
that counter memory accurately encodes compositional semantics. Finally, we
consider whether counter languages are semilinear. This work makes general
contributions to the theory of formal languages that are of potential interest
for understanding recurrent neural networks.
