  In contemporary speech understanding (SU), a sophisticated pipeline is
employed, encompassing the ingestion of streaming voice input. The pipeline
executes beam search iteratively, invoking a deep neural network to generate
tentative outputs (referred to as hypotheses) in an autoregressive manner.
Periodically, the pipeline assesses attention and Connectionist Temporal
Classification (CTC) scores.
  This paper aims to enhance SU performance on edge devices with limited
resources. Adopting a hybrid strategy, our approach focuses on accelerating
on-device execution and offloading inputs surpassing the device's capacity.
While this approach is established, we tackle SU's distinctive challenges
through innovative techniques: (1) Late Contextualization: This involves the
parallel execution of a model's attentive encoder during input ingestion. (2)
Pilot Inference: Addressing temporal load imbalances in the SU pipeline, this
technique aims to mitigate them effectively. (3) Autoregression Offramps:
Decisions regarding offloading are made solely based on hypotheses, presenting
a novel approach.
  These techniques are designed to seamlessly integrate with existing speech
models, pipelines, and frameworks, offering flexibility for independent or
combined application. Collectively, they form a hybrid solution for edge SU.
Our prototype, named XYZ, has undergone testing on Arm platforms featuring 6 to
8 cores, demonstrating state-of-the-art accuracy. Notably, it achieves a 2x
reduction in end-to-end latency and a corresponding 2x decrease in offloading
requirements.
