  As Large Language Models (LLMs) have made significant advancements across
various tasks, such as question answering, translation, text summarization, and
dialogue systems, the need for accuracy in information becomes crucial,
especially for serious financial products serving billions of users like
Alipay. To address this, Alipay has developed a Retrieval-Augmented Generation
(RAG) system that grounds LLMs on the most accurate and up-to-date information.
However, for a real-world product serving millions of users, the inference
speed of LLMs becomes a critical factor compared to a mere experimental model.
  Hence, this paper presents a generic framework for accelerating the inference
process, resulting in a substantial increase in speed and cost reduction for
our RAG system, with lossless generation accuracy. In the traditional inference
process, each token is generated sequentially by the LLM, leading to a time
consumption proportional to the number of generated tokens. To enhance this
process, our framework, named \textit{lookahead}, introduces a
\textit{multi-branch} strategy. Instead of generating a single token at a time,
we propose a \textit{Trie-based Retrieval} (TR) process that enables the
generation of multiple branches simultaneously, each of which is a sequence of
tokens. Subsequently, for each branch, a \textit{Verification and Accept} (VA)
process is performed to identify the longest correct sub-sequence as the final
output. Our strategy offers two distinct advantages: (1) it guarantees absolute
correctness of the output, avoiding any approximation algorithms, and (2) the
worst-case performance of our approach is equivalent to the conventional
process. We conduct extensive experiments to demonstrate the significant
improvements achieved by applying our inference acceleration framework. Code is
avaliable: https://github.com/alipay/PainlessInferenceAcceleration.
