  Fuzz drivers are necessary for library API fuzzing. Automatic fuzz driver
generation is challenging since it requires generating high quality API usage
code which is correct and robust. Large language model based fuzz driver
generator is a promising direction. Compared to traditional program analysis
based generators, it is a text-based approach which is more lightweight and
general. It can easily leverage various sources of API usage information for
generation and generate human-friendly code. Nonetheless, there still lacks the
basic understanding on this direction. To fill this gap, we did a study aiming
the core issues of using LLMs on effective fuzz driver generation. For
systematic understanding, 5 query strategies are designed and analyzed from
basic to enhanced. For evaluation in scale, we built a semi-automatic
framework, containing a quiz with 86 driver generation questions collected from
30 popular C projects, and a set of criteria for precise driver effectiveness
validation. In total, 189,628 fuzz drivers using 0.22 billion tokens are
generated and evaluated. Besides, generated drivers were compared with
industrial used ones to obtain practical insights (3.12 CPU year fuzzing
experiments). Our study revealed: 1) LLM-based generation has shown promising
practicality. 64% questions can be solved entirely automatically and the number
rises to 91% if manual semantic validators are incorporated. Moreover, the
generated drivers exhibited competitive performance to those commonly employed
in the industry; 2) LLMs struggle to generate fuzz drivers that require complex
API usage specifics. Three key designs can help: repeatedly querying, querying
with examples, and iteratively querying. Combining them yields a dominant
strategy; 3) Significant rooms for improvement are still left, such as
automatic semantic correctness validation, API usage expansion, and semantic
oracle generation.
