FreeMMIF: interactive multimodal medical image fusion via instruction-aware diffusion.
Authors
Affiliations (2)
Affiliations (2)
- The First Clinical Medical School, Beijing University of Chinese Medicine, Beijing, China.
- Department of Mechanical Engineering, Tsinghua University, Beijing, China.
Abstract
As a key technique in clinical diagnosis, multimodal medical image fusion (MMIF) integrates functional and metabolic information to assist diagnosis and enhance disease analysis reliability. However, existing methods typically rely on a single optimization objective, failing to meet clinical demands for flexible, on-the-fly result adjustments. To address this, we propose FreeMMIF, an interactive framework integrating a vision-language model (VLM)-based pseudo-labeling strategy and an instruction-aware diffusion model for task-guided, preference-adaptive fusion. During training, a VLM with specific prompts selects optimal candidates from existing methods as pseudo-ground truths to provide robust supervision without reference images. Subsequently, the diffusion process is modulated via an adaptive feature re-weighting branch, training the model to dynamically coordinate outputs by balancing weighted inputs from source images and pseudo-ground truths. Finally, we employ prompt engineering to construct a weight-generating VLM, allowing physicians to adjust source modality ratios via text instructions for direct control over fusion results. Experimental results demonstrate that FreeMMIF generates diagnostic-quality fusion results that precisely align with clinical intentions. Our code is available at: https://github.com/hzbbucm/FreeMMIF.