  Unit testing is essential in detecting bugs in functionally-discrete program
units. Manually writing high-quality unit tests is time-consuming and
laborious. Although traditional techniques can generate tests with reasonable
coverage, they exhibit low readability and cannot be directly adopted by
developers. Recent work has shown the large potential of large language models
(LLMs) in unit test generation, which can generate more human-like and
meaningful test code. ChatGPT, the latest LLM incorporating instruction tuning
and reinforcement learning, has performed well in various domains. However, It
remains unclear how effective ChatGPT is in unit test generation.
  In this work, we perform the first empirical study to evaluate ChatGPT's
capability of unit test generation. Specifically, we conduct a quantitative
analysis and a user study to systematically investigate the quality of its
generated tests regarding the correctness, sufficiency, readability, and
usability. The tests generated by ChatGPT still suffer from correctness issues,
including diverse compilation errors and execution failures. Still, the passing
tests generated by ChatGPT resemble manually-written tests by achieving
comparable coverage, readability, and even sometimes developers' preference.
Our findings indicate that generating unit tests with ChatGPT could be very
promising if the correctness of its generated tests could be further improved.
  Inspired by our findings above, we propose ChatTESTER, a novel ChatGPT-based
unit test generation approach, which leverages ChatGPT itself to improve the
quality of its generated tests. ChatTESTER incorporates an initial test
generator and an iterative test refiner. Our evaluation demonstrates the
effectiveness of ChatTESTER by generating 34.3% more compilable tests and 18.7%
more tests with correct assertions than the default ChatGPT.
