  Exploiting sparsity is a key technique in accelerating quantized
convolutional neural network (CNN) inference on mobile devices. Prior sparse
CNN accelerators largely exploit un-structured sparsity and achieve significant
speedups. Due to the unbounded, largely unpredictable sparsity patterns,
however, exploiting unstructured sparsity requires complicated hardware design
with significant energy and area overhead, which is particularly detrimental to
mobile/IoT inference scenarios where energy and area efficiency are crucial. We
propose to exploit structured sparsity, more specifically, Density Bound Block
(DBB) sparsity for both weights and activations. DBB block tensors bound the
maximum number of non-zeros per block. DBB thus exposes statically predictable
sparsity patterns that enable lean sparsity-exploiting hardware. We propose new
hardware primitives to implement DBB sparsity for (static) weights and
(dynamic) activations, respectively, with very low overheads. Building on top
of the primitives, we describe S2TA, a systolic array-based CNN accelerator
that exploits joint weight and activation DBB sparsity and new dimensions of
data reuse unavailable on the traditional systolic array. S2TA in 16nm achieves
more than 2x speedup and energy reduction compared to a strong baseline of a
systolic array with zero-value clock gating, over five popular CNN benchmarks.
Compared to two recent non-systolic sparse accelerators, Eyeriss v2 (65nm) and
SparTen (45nm), S2TA in 65nm uses about 2.2x and 3.1x less energy per
inference, respectively.
