  With the advent of large language models (LLMs) like GPT-3, a natural
question is the extent to which these models can be utilized for source code
optimization. This paper presents methodologically stringent case studies
applied to well-known open source python libraries pillow and numpy. We find
that contemporary LLM ChatGPT-4 (state September and October 2023) is
surprisingly adept at optimizing energy and compute efficiency. However, this
is only the case in interactive use, with a human expert in the loop. Aware of
experimenter bias, we document our qualitative approach in detail, and provide
transcript and source code. We start by providing a detailed description of our
approach in conversing with the LLM to optimize the _getextrema function in the
pillow library, and a quantitative evaluation of the performance improvement.
To demonstrate qualitative replicability, we report further attempts on another
locus in the pillow library, and one code locus in the numpy library, to
demonstrate generalization within and beyond a library. In all attempts, the
performance improvement is significant (factor up to 38). We have also not
omitted reporting of failed attempts (there were none). We conclude that LLMs
are a promising tool for code optimization in open source libraries, but that
the human expert in the loop is essential for success. Nonetheless, we were
surprised by how few iterations were required to achieve substantial
performance improvements that were not obvious to the expert in the loop. We
would like bring attention to the qualitative nature of this study, more robust
quantitative studies would need to introduce a layer of selecting experts in a
representative sample -- we invite the community to collaborate.
