Anatomy-guided prompting with cross-modal self-alignment for whole-body PET-CT breast cancer segmentation.
Authors
Affiliations (9)
Affiliations (9)
- organization=Faculty of Applied Sciences, Macao Polytechnic University, city=Macao, postcode=999078, country=China. Electronic address: [email protected].
- organization=Department of Nuclear Medicine, The Fifth Affiliated Hospital, Sun Yat-sen University, city=Zhuhai, postcode=519000, country=China. Electronic address: [email protected].
- organization=Department of Radiology, The Netherlands Cancer Institute, city=Amsterdam, postcode=1066, country=Netherlands. Electronic address: [email protected].
- organization=Faculty of Applied Sciences, Macao Polytechnic University, city=Macao, postcode=999078, country=China. Electronic address: [email protected].
- organization=Faculty of Applied Sciences, Macao Polytechnic University, city=Macao, postcode=999078, country=China. Electronic address: [email protected].
- organization=Faculty of Science and Technology, University of Macau, city=Macao, postcode=999078, country=China. Electronic address: [email protected].
- organization=Department of Biomedical Engineering, Case Western Reserve University, city=Cleveland, postcode=44106, country=USA. Electronic address: [email protected].
- organization=Department of Nuclear Medicine, The Fifth Affiliated Hospital, Sun Yat-sen University, city=Zhuhai, postcode=519000, country=China. Electronic address: [email protected].
- organization=Faculty of Applied Sciences, Macao Polytechnic University, city=Macao, postcode=999078, country=China. Electronic address: [email protected].
Abstract
Accurate segmentation of breast cancer in PET-CT images is crucial for precise staging, monitoring treatment response, and guiding personalized therapy. However, the small size and dispersed nature of metastatic lesions, coupled with the scarcity of annotated data and heterogeneity between modalities that hinders effective information fusion, make this task challenging. This paper proposes a novel anatomy-guided cross-modal learning framework to address these issues. Our approach first generates organ pseudo-labels through a teacher-student learning paradigm, which serve as anatomical prompts to guide cancer segmentation. We then introduce a self-aligning cross-modal pre-training method that aligns PET and CT features in a shared latent space through masked 3D patch reconstruction, enabling effective cross-modal feature fusion. Finally, we initialize the segmentation network's encoder with the pre-trained encoder weights, and incorporate organ labels through a Mamba-based prompt encoder and Hypernet-Controlled Cross-Attention mechanism for dynamic anatomical feature extraction and fusion. Notably, our method outperforms eight state-of-the-art methods, including CNN-based, transformer-based, and Mamba-based approaches, on two datasets encompassing primary breast cancer, metastatic breast cancer, and other types of cancer segmentation tasks.