  High-frequency trading (HFT) uses computer algorithms to make trading
decisions in short time scales (e.g., second-level), which is widely used in
the Cryptocurrency (Crypto) market (e.g., Bitcoin). Reinforcement learning (RL)
in financial research has shown stellar performance on many quantitative
trading tasks. However, most methods focus on low-frequency trading, e.g.,
day-level, which cannot be directly applied to HFT because of two challenges.
First, RL for HFT involves dealing with extremely long trajectories (e.g., 2.4
million steps per month), which is hard to optimize and evaluate. Second, the
dramatic price fluctuations and market trend changes of Crypto make existing
algorithms fail to maintain satisfactory performance. To tackle these
challenges, we propose an Efficient hieArchical Reinforcement learNing method
for High Frequency Trading (EarnHFT), a novel three-stage hierarchical RL
framework for HFT. In stage I, we compute a Q-teacher, i.e., the optimal action
value based on dynamic programming, for enhancing the performance and training
efficiency of second-level RL agents. In stage II, we construct a pool of
diverse RL agents for different market trends, distinguished by return rates,
where hundreds of RL agents are trained with different preferences of return
rates and only a tiny fraction of them will be selected into the pool based on
their profitability. In stage III, we train a minute-level router which
dynamically picks a second-level agent from the pool to achieve stable
performance across different markets. Through extensive experiments in various
market trends on Crypto markets in a high-fidelity simulation trading
environment, we demonstrate that EarnHFT significantly outperforms 6
state-of-art baselines in 6 popular financial criteria, exceeding the runner-up
by 30% in profitability.
