  We present the first study to investigate Large Language Models (LLMs) in
answering radiation oncology physics questions. Because popular exams like AP
Physics, LSAT, and GRE have large test-taker populations and ample test
preparation resources in circulation, they may not allow for accurately
assessing the true potential of LLMs. This paper proposes evaluating LLMs on a
highly-specialized topic, radiation oncology physics, which may be more
pertinent to scientific and medical communities in addition to being a valuable
benchmark of LLMs. We developed an exam consisting of 100 radiation oncology
physics questions based on our expertise at Mayo Clinic. Four LLMs, ChatGPT
(GPT-3.5), ChatGPT (GPT-4), Bard (LaMDA), and BLOOMZ, were evaluated against
medical physicists and non-experts. ChatGPT (GPT-4) outperformed all other LLMs
as well as medical physicists, on average. The performance of ChatGPT (GPT-4)
was further improved when prompted to explain first, then answer. ChatGPT
(GPT-3.5 and GPT-4) showed a high level of consistency in its answer choices
across a number of trials, whether correct or incorrect, a characteristic that
was not observed in the human test groups. In evaluating ChatGPTs (GPT-4)
deductive reasoning ability using a novel approach (substituting the correct
answer with "None of the above choices is the correct answer."), ChatGPT
(GPT-4) demonstrated surprising accuracy, suggesting the potential presence of
an emergent ability. Finally, although ChatGPT (GPT-4) performed well overall,
its intrinsic properties did not allow for further improvement when scoring
based on a majority vote across trials. In contrast, a team of medical
physicists were able to greatly outperform ChatGPT (GPT-4) using a majority
vote. This study suggests a great potential for LLMs to work alongside
radiation oncology experts as highly knowledgeable assistants.
