  The rapid advancement of Large Language Models (LLMs) has led to extensive
discourse regarding their potential to boost the return of quantitative stock
trading strategies. This discourse primarily revolves around harnessing the
remarkable comprehension capabilities of LLMs to extract sentiment factors
which facilitate informed and high-frequency investment portfolio adjustments.
To ensure successful implementations of these LLMs into the analysis of Chinese
financial texts and the subsequent trading strategy development within the
Chinese stock market, we provide a rigorous and encompassing benchmark as well
as a standardized back-testing framework aiming at objectively assessing the
efficacy of various types of LLMs in the specialized domain of sentiment factor
extraction from Chinese news text data. To illustrate how our benchmark works,
we reference three distinctive models: 1) the generative LLM (ChatGPT), 2) the
Chinese language-specific pre-trained LLM (Erlangshen-RoBERTa), and 3) the
financial domain-specific fine-tuned LLM classifier(Chinese FinBERT). We apply
them directly to the task of sentiment factor extraction from large volumes of
Chinese news summary texts. We then proceed to building quantitative trading
strategies and running back-tests under realistic trading scenarios based on
the derived sentiment factors and evaluate their performances with our
benchmark. By constructing such a comparative analysis, we invoke the question
of what constitutes the most important element for improving a LLM's
performance on extracting sentiment factors. And by ensuring that the LLMs are
evaluated on the same benchmark, following the same standardized experimental
procedures that are designed with sufficient expertise in quantitative trading,
we make the first stride toward answering such a question.
