  Pre-trained large language models have significantly improved code
generation. As these models scale up, there is an increasing need for the
output to handle more intricate tasks and to be appropriately specialized to
particular domains. Here, we target bioinformatics due to the amount of
specialized domain knowledge, algorithms, and data operations this discipline
requires. We present BioCoder, a benchmark developed to evaluate large language
models (LLMs) in generating bioinformatics-specific code. BioCoder spans a
broad spectrum of the field and covers cross-file dependencies, class
declarations, and global variables. It incorporates 1026 Python functions and
1243 Java methods extracted from GitHub, along with 253 examples from the
Rosalind Project, all pertaining to bioinformatics. Using topic modeling we
show that overall coverage of the included code is representative of the full
spectrum of bioinformatics calculations. BioCoder incorporates a fuzz-testing
framework for evaluation. We have applied it to evaluate many models including
InCoder, CodeGen, CodeGen2, SantaCoder, StarCoder, StarCoder+, InstructCodeT5+,
GPT-3.5, and GPT-4. Furthermore, we finetuned StarCoder, demonstrating how our
dataset can effectively enhance the performance of LLMs on our benchmark (by
>15% in terms of Pass@K in certain prompt configurations and always >3%). The
results highlight two key aspects of successful models: (1) Successful models
accommodate a long prompt (> ~2600 tokens) with full context, for functional
dependencies. (2) They contain specific domain knowledge of bioinformatics,
beyond just general coding knowledge. This is evident from the performance gain
of GPT-3.5/4 compared to the smaller models on the benchmark (50% vs up to
~25%). Our dataset, benchmark, Docker images, and scripts required for testing
are all available at https://github.com/gersteinlab/biocoder.
