Exploring the Capabilities of Large Language Model Encoders for Image-Text Retrieval in Chest X-rays.
Authors
Abstract
Multimodal learning from paired medical images and clinical text is a central challenge in medical data driven informatics, where effective cross-modal alignment is critical for scalable analysis and retrieval. In chest radiography, vision-language pretraining is constrained by heterogeneous radiology reports that contain abbrevia tions, impression-only notes, and institution-specific writing styles. Unlike general-domain settings, naively aggre gating large collections of noisy reports can plateau or even degrade multimodal learning when reporting styles differ substantially. We propose a domain-adapted bidirec tional large language model text encoder for chest radio graph reports, trained with masked token prediction and supervised contrastive learning on stylistically diverse but clinically equivalent report variants to produce robust, generalizable text embeddings. We then integrate this encoder into a dual-tower contrastive vision-language framework using parameter-efficient adaptation to improve image-text alignment. Across 1.6 million paired studies from public datasets and a de-identified hospital cohort, the proposed models improve bidirectional retrieval accuracy and external generalization, achieving GREEN scores of 0.308 on MIMIC-CXR and 0.618 on Open-I, while reducing the degradation observed when abbreviation-rich, impression-only hospital reports are added to training. Significance: Robust cross-modal embeddings enable scalable retrieval and multimodal representation learning from routine clinical data for biomedical and health informatics.