  Inspired by number series tests to measure human intelligence, we suggest
number sequence prediction tasks to assess neural network models' computational
powers for solving algorithmic problems. We define the complexity and
difficulty of a number sequence prediction task with the structure of the
smallest automaton that can generate the sequence. We suggest two types of
number sequence prediction problems: the number-level and the digit-level
problems. The number-level problems format sequences as 2-dimensional grids of
digits and the digit-level problems provide a single digit input per a time
step. The complexity of a number-level sequence prediction can be defined with
the depth of an equivalent combinatorial logic, and the complexity of a
digit-level sequence prediction can be defined with an equivalent state
automaton for the generation rule. Experiments with number-level sequences
suggest that CNN models are capable of learning the compound operations of
sequence generation rules, but the depths of the compound operations are
limited. For the digit-level problems, simple GRU and LSTM models can solve
some problems with the complexity of finite state automata. Memory augmented
models such as Stack-RNN, Attention, and Neural Turing Machines can solve the
reverse-order task which has the complexity of simple pushdown automaton.
However, all of above cannot solve general Fibonacci, Arithmetic or Geometric
sequence generation problems that represent the complexity of queue automata or
Turing machines. The results show that our number sequence prediction problems
effectively evaluate machine learning models' computational capabilities.
