  Recent code large language models (LLMs) have shown promising performance in
generating standalone functions but face limitations in repository-level code
generation due to their lack of awareness of repository-level dependencies
(e.g., user-defined attributes), resulting in dependency errors such as
undefined-variable and no-member errors. In this work, we introduce ToolGen, an
approach that integrates autocompletion tools into the code LLM generation
process to address these dependencies. ToolGen comprises two main phases:
Trigger Insertion and Model Fine-tuning (Offline), and Tool-integrated Code
Generation (Online). During the offline phase, ToolGen augments functions
within a given code corpus with a special mark token, indicating positions to
trigger autocompletion tools. These augmented functions, along with their
corresponding docstrings, are then used to fine-tune a selected code LLM. In
the online phase, ToolGen iteratively generates functions by predicting tokens
step-by-step using the fine-tuned LLM. Whenever a mark token is encountered,
ToolGen invokes the autocompletion tool to suggest code completions and selects
the most appropriate one.
  We conduct comprehensive experiments to evaluate ToolGen's effectiveness in
repository-level code generation. To facilitate this evaluation, we create a
benchmark comprising 680 real-world code repositories and introduce two new
repository-level metrics: Dependency Coverage and Static Validity Rate. The
results demonstrate that ToolGen significantly improves Dependency Coverage by
15.2% to 45.8% and Static Validity Rate by 10.9% to 42.2% across three distinct
code LLMs, while maintaining competitive performance in widely-recognized
similarity metrics. Furthermore, our generalizability evaluation confirms
ToolGen's consistent performance when applied to diverse code LLMs, including
various model architectures and scales.
