  Semantic understanding and completion of real world scenes is a foundational
primitive of 3D Visual perception widely used in high-level applications such
as robotics, medical imaging, autonomous driving and navigation. Due to the
curse of dimensionality, compute and memory requirements for 3D scene
understanding grow in cubic complexity with voxel resolution, posing a huge
impediment to realizing real-time energy efficient deployments. The inherent
spatial sparsity present in the 3D world due to free space is fundamentally
different from the channel-wise sparsity that has been extensively studied. We
present ACCELERATOR FOR SPATIALLY SPARSE 3D DNNs (AccSS3D), the first
end-to-end solution for accelerating 3D scene understanding by exploiting the
ample spatial sparsity. As an algorithm-dataflow-architecture co-designed
system specialized for spatially-sparse 3D scene understanding, AccSS3D
includes novel spatial locality-aware metadata structures, a near-zero latency
and spatial sparsity-aware dataflow optimizer, a surface orientation aware
pointcloud reordering algorithm and a codesigned hardware accelerator for
spatial sparsity that exploits data reuse through systolic and multicast
interconnects. The SSpNNA accelerator core together with the 64 KB of L1 memory
requires 0.92 mm2 of area in 16nm process at 1 GHz. Overall, AccSS3D achieves
16.8x speedup and a 2232x energy efficiency improvement for 3D sparse
convolution compared to an Intel-i7-8700K 4-core CPU, which translates to a
11.8x end-to-end 3D semantic segmentation speedup and a 24.8x energy efficiency
improvement (iso technology node)
