  Self-delimiting (SLIM) programs are a central concept of theoretical computer
science, particularly algorithmic information & probability theory, and
asymptotically optimal program search (AOPS). To apply AOPS to (possibly
recurrent) neural networks (NNs), I introduce SLIM NNs. Neurons of a typical
SLIM NN have threshold activation functions. During a computational episode,
activations are spreading from input neurons through the SLIM NN until the
computation activates a special halt neuron. Weights of the NN's used
connections define its program. Halting programs form a prefix code. The reset
of the initial NN state does not cost more than the latest program execution.
Since prefixes of SLIM programs influence their suffixes (weight changes
occurring early in an episode influence which weights are considered later),
SLIM NN learning algorithms (LAs) should execute weight changes online during
activation spreading. This can be achieved by applying AOPS to growing SLIM
NNs. To efficiently teach a SLIM NN to solve many tasks, such as correctly
classifying many different patterns, or solving many different robot control
tasks, each connection keeps a list of tasks it is used for. The lists may be
efficiently updated during training. To evaluate the overall effect of
currently tested weight changes, a SLIM NN LA needs to re-test performance only
on the efficiently computable union of tasks potentially affected by the
current weight changes. Future SLIM NNs will be implemented on 3-dimensional
brain-like multi-processor hardware. Their LAs will minimize task-specific
total wire length of used connections, to encourage efficient solutions of
subtasks by subsets of neurons that are physically close. The novel class of
SLIM NN LAs is currently being probed in ongoing experiments to be reported in
separate papers.
