  The demand to process vast amounts of data generated from state-of-the-art
high resolution cameras has motivated novel energy-efficient on-device AI
solutions. Visual data in such cameras are usually captured in the form of
analog voltages by a sensor pixel array, and then converted to the digital
domain for subsequent AI processing using analog-to-digital converters (ADC).
Recent research has tried to take advantage of massively parallel low-power
analog/digital computing in the form of near- and in-sensor processing, in
which the AI computation is performed partly in the periphery of the pixel
array and partly in a separate on-board CPU/accelerator. Unfortunately,
high-resolution input images still need to be streamed between the camera and
the AI processing unit, frame by frame, causing energy, bandwidth, and security
bottlenecks. To mitigate this problem, we propose a novel
Processing-in-Pixel-in-memory (P2M) paradigm, that customizes the pixel array
by adding support for analog multi-channel, multi-bit convolution, batch
normalization, and ReLU (Rectified Linear Units). Our solution includes a
holistic algorithm-circuit co-design approach and the resulting P2M paradigm
can be used as a drop-in replacement for embedding memory-intensive first few
layers of convolutional neural network (CNN) models within
foundry-manufacturable CMOS image sensor platforms. Our experimental results
indicate that P2M reduces data transfer bandwidth from sensors and analog to
digital conversions by ~21x, and the energy-delay product (EDP) incurred in
processing a MobileNetV2 model on a TinyML use case for visual wake words
dataset (VWW) by up to ~11x compared to standard near-processing or in-sensor
implementations, without any significant drop in test accuracy.
