text: We introduce a neural network with a recurrent attention model over a possibly large external memory. The architecture is a form of Memory Network (, 2015) but unlike the model in that work, it is trained end-to-end, and hence requires significantly less supervision during training, making it more generally applicable in realistic settings. It can also be seen as an extension of RNNsearch to the case where multiple computational steps (hops) are performed per output symbol. The flexibility of the model allows us to apply it to tasks as diverse as (synthetic) question answering and to language modeling. For the former our approach is competitive with Memory Networks, but with less supervision. For the latter, on the Penn TreeBank and Text8 datasets our approach demonstrates comparable performance to RNNs and LSTMs. In both cases we show that the key concept of multiple computational hops yields improved results.
  @cite_25: The problem of synthesizing apparatus that will automatically simulate man's ability to recognize and to learn to recognize patterns is discussed and it is concluded that analogue circuits, rather than the digital switching circuits that have been employed in the past, provide the simpler solution. A new circuit unit that possesses many of the essential functional characteristics exhibited by nerve cells in the brain is derived from earlier work on the electrical simulation of nervous-system functional activity and forms the basic element of the circuits. The new analogue apparatus consists of a number of distinct functional circuits arranged in a definite sequence, through which signals derived from the patterns to be recognized pass simultaneously on their way to the final output terminals. Classification information may be built into the apparatus initially if it is available, but if not, it can be stored automatically in a special unit during a setting-up procedure in which samples of the pattern types that the apparatus will be required to recognize are presented, together with identification signals. Low-resolution automatic pattern-recognition apparatus is described, and examples illustrate the setting-up procedure and subsequent performance of the apparatus.
  @cite_2: The paper gives a survey of the learning circuits which became known as learning matrices and some of their possible technological applications. The first section describes the principle of learning matrices. So-called conditioned connections between the characteristics of an object and the meaning of an object are formed in the learning phase. During the operation of connecting the characteristics of an object with its meaning (EB operation of the knowing phase) upon presenting the object characteristics, the associated most similar meaning is realized in the form of a signal by maximum likelihood decoding. Conversely, in operation from the meaning of an object to its characteristics (BE operation) the associated object characteristics are obtained as signals by parallel reading upon application of an object meaning. According to the characteristic signals processed (binary or analog signals) discrimination must be made between binary and nonbinary learning matrices. In the case of the binary learning matrix the conditioned connections are a statistical measure for the frequency of the coordination of object characteristics and object meaning, in the case of the nonbinary learning matrix they are a measure for an analog value proportional to a characteristic. Both types of matrices allow for the characteristic sets applied during EB operation to be unsystematically disturbed within limits. Moreover, the nonbinary learning matrix is invariant to systematic deviations between presented and learned characteristic sets (invariance to affine transformation, translation and rotated skewness).
summary: Some of the very early work on neural networks by Steinbuch and Piske @cite_2 and Taylor @cite_25 considered a memory that performed nearest-neighbor operations on stored input vectors and then fit parametric models to the retrieved sets. This has similarities to a single layer version of our model.