  Software testing is a crucial aspect of software development, and the
creation of high-quality tests that adhere to best practices is essential for
effective maintenance. Recently, Large Language Models (LLMs) have gained
popularity for code generation, including the automated creation of test cases.
However, these LLMs are often trained on vast amounts of publicly available
code, which may include test cases that do not adhere to best practices and may
even contain test smells (anti-patterns). To address this issue, we propose a
novel technique called Reinforcement Learning from Static Quality Metrics
(RLSQM). To begin, we analyze the anti-patterns generated by the LLM and show
that LLMs can generate undesirable test smells. Thus, we train specific reward
models for each static quality metric, then utilize Proximal Policy
Optimization (PPO) to train models for optimizing a single quality metric at a
time. Furthermore, we amalgamate these rewards into a unified reward model
aimed at capturing different best practices and quality aspects of tests. By
comparing RL-trained models with those trained using supervised learning, we
provide insights into how reliably utilize RL to improve test generation
quality and into the effects of various training strategies. Our experimental
results demonstrate that the RL-optimized model consistently generated
high-quality test cases compared to the base LLM, improving the model by up to
21%, and successfully generates nearly 100% syntactically correct code. RLSQM
also outperformed GPT-4 on four out of seven metrics. This represents a
significant step towards enhancing the overall efficiency and reliability of
software testing through Reinforcement Learning and static quality metrics. Our
data are available at this link: https://figshare.com/s/ded476c8d4c221222849.
