Knowledge distillation and teacher-student learning in medical imaging: Comprehensive overview, pivotal role, and future directions.
Authors
Affiliations (5)
Affiliations (5)
- College of Information Science and Engineering, Northeastern University, Shenyang 110819, China; Hebei Key Laboratory of Micro-Nano Precision Optical Sensing and Measurement Technology, Qinhuangdao, 066004, China.
- Department of Control Science and Engineering, Harbin Institute of Technology, Harbin 150001, China.
- Department of Control Science and Engineering, Harbin Institute of Technology, Harbin 150001, China. Electronic address: [email protected].
- College of Information Science and Engineering, Northeastern University, Shenyang 110819, China; Hebei Key Laboratory of Micro-Nano Precision Optical Sensing and Measurement Technology, Qinhuangdao, 066004, China; State Key Laboratory of Synthetical Automation for Process Industries, Shenyang, 110819, China. Electronic address: [email protected].
- Department of Mechanical and Industrial Engineering, Faculty of Engineering, Norwegian University of Science and Technology, 7034, Trondheim, Norway. Electronic address: [email protected].
Abstract
Knowledge Distillation (KD) is a technique to transfer the knowledge from a complex model to a simplified model. It has been widely used in natural language processing and computer vision and has achieved advanced results. Recently, the research of KD in medical image analysis has grown rapidly. The definition of knowledge has been further expanded by combining with the medical field, and its role is not limited to simplifying the model. This paper attempts to comprehensively review the development and application of KD in the medical imaging field. Specifically, we first introduce the basic principles, explain the definition of knowledge and the classical teacher-student network framework. Then, the research progress in medical image classification, segmentation, detection, reconstruction, registration, radiology report generation, privacy protection and other application scenarios is presented. In particular, the introduction of application scenarios is based on the role of KD. We summarize eight main roles of KD techniques in medical image analysis, including model compression, semi-supervised method, weakly supervised method, class balancing, etc. The performance of these roles in all application scenarios is analyzed. Finally, we discuss the challenges in this field and propose potential solutions. KD is still in a rapid development stage in the medical imaging field, we give five potential development directions and research hotspots. A comprehensive literature list of this survey is available at https://github.com/XiangQA-Q/KD-in-MIA.