  Neural networks have become the key technology of artificial intelligence and
have contributed to breakthroughs in several machine learning tasks, primarily
owing to advances in deep learning applied to Artificial Neural Networks
(ANNs). Simultaneously, Spiking Neural Networks (SNNs) incorporating
biologically-feasible spiking neurons have held great promise because of their
rich temporal dynamics and high-power efficiency. However, the developments in
SNNs were proceeding separately from those in ANNs, effectively limiting the
adoption of deep learning research insights. Here we show an alternative
perspective on the spiking neuron that casts it as a particular ANN construct
called Spiking Neural Unit (SNU), and a soft SNU (sSNU) variant that
generalizes its dynamics to a novel recurrent ANN unit. SNUs bridge the
biologically-inspired SNNs with ANNs and provide a methodology for seamless
inclusion of spiking neurons in deep learning architectures. Furthermore, SNU
enables highly-efficient in-memory acceleration of SNNs trained with
backpropagation through time, implemented with the hardware in-the-loop. We
apply SNUs to tasks ranging from hand-written digit recognition, language
modelling, to music prediction. We obtain accuracy comparable to, or better
than, that of state-of-the-art ANNs, and we experimentally verify the efficacy
of the in-memory-based SNN realization for the music-prediction task using
52,800 phase-change memory devices. The new generation of neural units
introduced in this paper incorporate biologically-inspired neural dynamics in
deep learning. In addition, they provide a systematic methodology for training
neuromorphic computing hardware. Thus, they open a new avenue for a widespread
adoption of SNNs in practical applications.
