  Large language models (LLMs) are often augmented with tools to solve complex
tasks. By generating code snippets and executing them through task-specific
Application Programming Interfaces (APIs), they can offload certain functions
to dedicated external modules, such as image encoding and performing
calculations. However, most existing approaches to augment LLMs with tools are
constrained by general-purpose APIs and lack the flexibility for tailoring them
to specific tasks. In this work, we present CRAFT, a general tool creation and
retrieval framework for LLMs. It creates toolsets specifically curated for the
tasks and equips LLMs with a component that retrieves tools from these sets to
enhance their capability to solve complex tasks. For each task, we collect
specific code solutions by prompting GPT-4 to solve the training examples.
Following a validation step ensuring the correctness, these solutions are
abstracted into code snippets to enhance reusability, and deduplicated for
higher quality. At inference time, the language model retrieves snippets from
the toolsets and then executes them or generates the output conditioning on the
retrieved snippets. Our method is designed to be flexible and offers a
plug-and-play approach to adapt off-the-shelf LLMs to unseen domains and
modalities, without any finetuning. Experiments on vision-language, tabular
processing, and mathematical reasoning tasks show that our approach achieves
substantial improvements compared to strong baselines. In addition, our
in-depth analysis reveals that: (1) consistent performance improvement can be
achieved by scaling up the number of tools and the capability of the backbone
models; (2) each component of our approach contributes to the performance
gains; (3) the created tools are well-structured and reliable with low
complexity and atomicity. The code is available at
\url{https://github.com/lifan-yuan/CRAFT}.
