  Generative Artificial Intelligence (AI) holds immense potential in medical
applications. Numerous studies have explored the efficacy of various generative
AI models within healthcare contexts, but there is a lack of a comprehensive
and systematic evaluation framework. Given that some studies evaluating the
ability of generative AI for medical applications have deficiencies in their
methodological design, standardized guidelines for their evaluation are also
currently lacking. In response, our objective is to devise standardized
assessment guidelines tailored for evaluating the performance of generative AI
systems in medical contexts. To this end, we conducted a thorough literature
review using the PubMed and Google Scholar databases, focusing on research that
tests generative AI capabilities in medicine. Our multidisciplinary team,
comprising experts in life sciences, clinical medicine, medical engineering,
and generative AI users, conducted several discussion sessions and developed a
checklist of 23 items. The checklist is designed to encompass the critical
evaluation aspects of generative AI in medical applications comprehensively.
This checklist, and the broader assessment framework it anchors, address
several key dimensions, including question collection, querying methodologies,
and assessment techniques. We aim to provide a holistic evaluation of AI
systems. The checklist delineates a clear pathway from question gathering to
result assessment, offering researchers guidance through potential challenges
and pitfalls. Our framework furnishes a standardized, systematic approach for
research involving the testing of generative AI's applicability in medicine. It
enhances the quality of research reporting and aids in the evolution of
generative AI in medicine and life sciences.
