Retrieval-augmented in-context learning for multimodal large language models in disease classification.
Authors
Affiliations (7)
Affiliations (7)
- Department of Electrical and Computer Engineering, University of Minnesota, 200 Union St SE, Minneapolis, 55455, MN, USA.
- Division of Computational Health Sciences, Department of Surgery, University of Minnesota, 516 Delaware St SE, Minneapolis, 55455, MN, USA.
- Department of Civil and Environmental Engineering, University of Michigan, 2350 Hayward Street, Ann Arbor, 48109, MI, USA.
- Institute for Health Informatics, University of Minnesota, 516 Delaware Street SE, Minneapolis, 55455, MN, USA.
- Department of Computer Science and Engineering, University of Minnesota, 200 Union St SE, Minneapolis, 55455, MN, USA.
- Department of Chemical Engineering and Materials Science, University of Minnesota, 421 Washington Ave. SE, Minneapolis, 55455, MN, USA.
- Division of Computational Health Sciences, Department of Surgery, University of Minnesota, 516 Delaware St SE, Minneapolis, 55455, MN, USA. Electronic address: [email protected].
Abstract
We aim to dynamically retrieve informative demonstrations, enhancing in-context learning in multimodal large language models (MLLMs) for disease classification. We propose a Retrieval-Augmented In-Context Learning (RAICL) framework, which integrates retrieval-augmented generation (RAG) and in-context learning (ICL) to adaptively select demonstrations with similar disease patterns, enabling more effective ICL in MLLMs. Specifically, RAICL examines embeddings from patch embeddings and diverse encoders, including ResNet, BERT, BioBERT, and ClinicalBERT, to retrieve appropriate demonstrations, and construct conversational prompts optimized for ICL. We evaluated the framework on two real-world multi-modal datasets (TCGA and IU Chest X-ray), assessing its performance across multiple MLLMs (Qwen, Llava, Gemma), embedding strategies, similarity metrics, and varying numbers of demonstrations. RAICL consistently outperformed non-retrieval baselines. Accuracy improved from 0.7857 to 0.8726 on TCGA and from 0.7924 to 0.8658 on IU Chest X-ray. Multimodal inputs surpassed single modalities, with text-only outperforming image-only, and few-shot retrieval further boosted performance. Across similarity metrics, Euclidean distance achieved the best results. Improvements were consistent across different MLLMs, demonstrating robustness and scalability. RAICL provides an efficient and scalable approach to enhance in-context learning in MLLMs for multimodal disease classification.