Benchmarking Vision LLMs in fetal ultrasound interpretation: a five-point expert evaluation of standard vs. custom prompts.

Authors

Abstract

Fetal ultrasound imaging is critical for prenatal care, demanding accurate anatomical interpretation. This study evaluates the potential of Vision Large Language Models (LLMs) in interpreting fetal ultrasound images, exploring whether tailored prompts can enhance performance compared to standard prompts, and assessing their utility in clinical settings. Nine fetal ultrasound images were analyzed using six advanced Vision LLMs via the Chatbot Arena platform. Standard prompts were compared against expert-crafted tailored questions. Three expert sonographers assessed the models' outputs across five criteria-anatomical recognition, biometric potential, picture quality, normalcy assessment, and clinical recommendations-using a Likert scale (1-5). Standard prompts yielded limited interpretative accuracy. In contrast, custom prompts significantly improved performance, with Claude Sonnet 3.5 and ChatGPT4o achieving median scores of 19 and 18, respectively. Models excelled in analyzing fetal femur and trans-cerebellum images, with clinical advice being the easiest to identify. Challenges persisted in precise anatomical identification and image quality assessment, revealing limitations in visual recognition. Smaller models like pixtral-12b showed notable improvement with tailoring, suggesting fine-tuning potential, while larger models did not consistently outperform smaller ones, indicating factors beyond model size influence efficacy. Tailored prompts markedly enhance Vision LLMs' ability to interpret fetal ultrasound images, supporting their potential as aids in prenatal diagnosis and education. However, limitations in anatomical precision and image quality assessment persist. Future research should focus on refining models with specialized datasets, optimizing architectures, and advancing prompt engineering to maximize clinical utility.

Tags

Topics