Transfer learning from 2D natural images to 4D fMRI brain images via geometric mapping.
Authors
Affiliations (30)
Affiliations (30)
- Center of Brain Sciences, Beijing Institute of Basic Medical Sciences, Beijing, China.
- State Key Laboratory of Cognitive Science and Mental Health, Institute of Psychology, Chinese Academy of Sciences, Beijing 100101, China; Department of Psychology, University of Chinese Academy of Sciences, Beijing 100049, China.
- The Third Affiliated Hospital of Zhengzhou University, China.
- Affiliated Brain Hospital of Guangzhou Medical University, Guangzhou 510370, China.
- The First Affiliated Hospital of Jinan University, Guangzhou, Guangdong 250024, China.
- Department of Radiology, The First Affiliated Hospital, College of Medicine, Zhejiang University, Hangzhou, Zhejiang 310058, China.
- Huaxi MR Research Center (HMRRC), Department of Radiology, West China Hospital of Sichuan University, Chengdu, Sichuan 610044, China; Research Unit of Psychoradiology, Chinese Academy of Medical Sciences, Chengdu, Sichuan 610052, China.
- Department of Psychiatry, First Affiliated Hospital of Kunming Medical University, Kunming, Yunnan 650032, China.
- Department of Psychiatry, and National Clinical Research Center for Mental Disorders, The Second Xiangya Hospital of Central South University, Changsha 410011 Hunan, China.
- Department of Neurology, Affiliated ZhongDa Hospital of Southeast University, Nanjing, Jiangsu 210009, China.
- Department of Psychiatry, The First Affiliated Hospital of Chongqing Medical University, Chongqing 400042, China.
- Department of Clinical Psychology, Suzhou Psychiatric Hospital, The Affiliated Guangji Hospital of Soochow University, Suzhou, Jiangsu 215003, China.
- Xijing Hospital of Air Force Military Medical University, Xi'an, Shaanxi 710032, China.
- Beijing Anding Hospital, Capital Medical University, Beijing 100120, China.
- Affiliated Mental Health Center & Hangzhou Seventh People's Hospital, Zhejiang University School of Medicine, Hangzhou, Zhejiang 310063, China; Mental Health Center and Psychiatric Laboratory, West China Hospital of Sichuan University, Chengdu, Sichuan 610044, China.
- Department of Psychosomatics and Psychiatry, Zhongda Hospital, School of Medicine, Southeast University, Nanjing, Jiangsu 210009, China.
- Shenzhen Kangning Hospital, Shenzhen, Guangzhou 518020, China.
- Faculty of Psychology, Southwest University, Chongqing 400715, China.
- National Clinical Research Center for Mental Disorders (Peking University Sixth Hospital) & Key Laboratory of Mental Health, Ministry of Health (Peking University), Beijing 100191, China.
- Xi'an Central Hospital, Xi'an, Shaanxi 710004, China.
- State Key Laboratory of Cognitive Science and Mental Health, Institute of Psychology, Chinese Academy of Sciences, Beijing 100101, China.
- Institute of Neuroscience, Chongqing Medical University, Chongqing 400016, China; Chongqing Key Laboratory of Neurobiology, Chongqing 400000, China; Department of Neurology, The First Affiliated Hospital of Chongqing Medical University, Chongqing 400042, China.
- Chongqing Key Laboratory of Neurobiology, Chongqing 400000, China.
- The First Affiliated Hospital of Anhui Medical University, Hefei 230022, China; Research Center of Clinical Medical Imaging, Anhui Province, Hefei 230032, China; Anhui Provincial Institute of Translational Medicine, Hefei 230032, China.
- West China Hospital of Sichuan University, Chengdu, Sichuan 610044, China.
- First Hospital of Shanxi Medical University, Taiyuan, Shanxi 030001, China.
- Developmental Population Neuroscience Research Center, IDG/McGovern Institute for Brain Research, Beijing Normal University, Beijing 100091, China; National Basic Science Data Center, Beijing 100038, China.
- State Key Laboratory of Cognitive Science and Mental Health, Institute of Psychology, Chinese Academy of Sciences, Beijing 100101, China; Department of Psychology, University of Chinese Academy of Sciences, Beijing 100049, China; Department of Psychological and Cognitive Sciences, Tsinghua University, Beijing 100084, China.
- Center for Cognition and Brain Disorders, The Affiliated Hospital of Hangzhou Normal University, Hangzhou, Zhejiang 310018, China; Zhejiang Key Laboratory for Research in Assessment of Cognitive Impairments, Hangzhou, Zhejiang 310000, China.
- College of Intelligence Science and Technology, National University of Defense Technology, Changsha, Hunan, China. Electronic address: [email protected].
Abstract
Functional magnetic resonance imaging (fMRI) allows real-time observation of brain activity through blood oxygen level-dependent (BOLD) signals and is extensively used in studies related to sex classification, age estimation, behavioral measurements prediction, and mental disorder diagnosis. However, the application of deep learning techniques to brain fMRI analysis is hindered by the small sample size of fMRI datasets. Transfer learning offers a solution to this problem, but most existing approaches are designed for large-scale 2D natural images. The heterogeneity between 4D fMRI data and 2D natural images makes direct model transfer infeasible. This study proposes a novel geometric mapping-based fMRI transfer learning method that enables transfer learning from 2D natural images to 4D fMRI brain images, bridging the transfer learning gap between fMRI data and natural images. The proposed Multi-scale Multi-domain Feature Aggregation (MMFA) module extracts effective aggregated features and reduces the dimensionality of fMRI data to 3D space. By treating the cerebral cortex as a folded Riemannian manifold in 3D space and mapping it into 2D space using surface geometric mapping, we make the transfer learning from 2D natural images to 4D brain images possible. Moreover, the topological relationships of the cerebral cortex are maintained with our method, and calculations are performed along the Riemannian manifold of the brain, effectively addressing signal interference problems. The experimental results based on the Human Connectome Project (HCP) dataset demonstrate the effectiveness of the proposed method. Our method achieved state-of-the-art performance in sex classification, age estimation, and behavioral measurement prediction tasks. Moreover, we propose a cascaded transfer learning approach for depression diagnosis, and proved its effectiveness on 23 depression datasets. In summary, the proposed fMRI transfer learning method, which accounts for the structural characteristics of the brain, is promising for applying transfer learning from natural images to brain fMRI images, significantly enhancing the performance in various fMRI analysis tasks.