  Visual sensors, including 3D LiDAR, neuromorphic DVS sensors, and
conventional frame cameras, are increasingly integrated into edge-side
intelligent machines. Realizing intensive multi-sensory data analysis directly
on edge intelligent machines is crucial for numerous emerging edge
applications, such as augmented and virtual reality and unmanned aerial
vehicles, which necessitates unified data representation, unprecedented
hardware energy efficiency and rapid model training. However, multi-sensory
data are intrinsically heterogeneous, causing significant complexity in the
system development for edge-side intelligent machines. In addition, the
performance of conventional digital hardware is limited by the physically
separated processing and memory units, known as the von Neumann bottleneck, and
the physical limit of transistor scaling, which contributes to the slowdown of
Moore's law. These limitations are further intensified by the tedious training
of models with ever-increasing sizes. We propose a novel hardware-software
co-design, random resistive memory-based deep extreme point learning machine
(DEPLM), that offers efficient unified point set analysis. We show the system's
versatility across various data modalities and two different learning tasks.
Compared to a conventional digital hardware-based system, our co-design system
achieves huge energy efficiency improvements and training cost reduction when
compared to conventional systems. Our random resistive memory-based deep
extreme point learning machine may pave the way for energy-efficient and
training-friendly edge AI across various data modalities and tasks.
