  Spiking Neural Networks (SNNs) have emerged as a hardware efficient
architecture for classification tasks. The challenge of spike-based encoding
has been the lack of a universal training mechanism performed entirely using
spikes. There have been several attempts to adopt the powerful backpropagation
(BP) technique used in non-spiking artificial neural networks (ANN): (1) SNNs
can be trained by externally computed numerical gradients. (2) A major
advancement towards native spike-based learning has been the use of approximate
Backpropagation using spike-time dependent plasticity (STDP) with phased
forward/backward passes. However, the transfer of information between such
phases for gradient and weight update calculation necessitates external memory
and computational access. This is a challenge for standard neuromorphic
hardware implementations. In this paper, we propose a stochastic SNN based
Back-Prop (SSNN-BP) algorithm that utilizes a composite neuron to
simultaneously compute the forward pass activations and backward pass gradients
explicitly with spikes. Although signed gradient values are a challenge for
spike-based representation, we tackle this by splitting the gradient signal
into positive and negative streams. We show that our method approaches BP ANN
baseline with sufficiently long spike-trains. Finally, we show that the
well-performing softmax cross-entropy loss function can be implemented through
inhibitory lateral connections enforcing a Winner Take All (WTA) rule. Our SNN
with a 2-layer network shows excellent generalization through comparable
performance to ANNs with equivalent architecture and regularization parameters
on static image datasets like MNIST, Fashion-MNIST, Extended MNIST, and
temporally encoded image datasets like Neuromorphic MNIST datasets. Thus,
SSNN-BP enables BP compatible with purely spike-based neuromorphic hardware.
