Development of an expert-annotated chest X-ray dataset to support AI validation in tuberculosis diagnosis.
Authors
Affiliations (12)
Affiliations (12)
- Division of Diagnostic Imaging, Department of Radiology, Faculty of Medicine, Prince of Songkla University, Hat Yai, Songkhla, Thailand. [email protected].
- Department of Radio-Diagnosis and Imaging, Samtse General Hospital, Samtse, 22001, Bhutan.
- Department of Family and Preventive Medicine, Department of Clinical Research and Medical Data Science, Faculty of Medicine, Prince of Songkla University, Hat Yai, Songkhla, Thailand.
- Department of Biomedical Sciences and Biomedical Engineering, Faculty of Medicine, Prince of Songkla University, Hat Yai, Songkhla, Thailand.
- Department of Internal Medicine, Faculty of Medicine, Mahasarakham University, Maha Sarakham, Thailand.
- Radiology Department, Udonthani Hospital, Udon Thani, Thailand.
- Radiology Department, Chiangrai Prachanukroh Hospital, Chiang Rai, Thailand.
- Tuberculosis Division, Department of Disease Control, Ministry of Public Health, Bangkok, Thailand.
- Division of Diagnostic Imaging, Department of Radiology, Faculty of Medicine, Prince of Songkla University, Hat Yai, Songkhla, Thailand.
- Central Chest Institute of Thailand, Department of Medical Services, Ministry of Public Health, Nonthaburi, Thailand.
- Rajavithi Hospital, Department of Medical Services, Ministry of Public Health, Department of Radiology, College of Medicine, Rangsit University, Bangkok, Thailand.
- Department of Diagnostic and Therapeutic Radiology, Faculty of Medicine Ramathibodhi Hospital, Mahidol University, Bangkok, Thailand.
Abstract
To assess inter-rater agreement and diagnostic performance of six United States National Institute for Occupational Safety and Health-certified B readers (physicians certified in standardized chest X-ray (CXR) classification for pneumoconiosis) in diagnosing tuberculosis (TB) on CXR, with the goal of developing a reliable dataset to support external validation of artificial intelligence (AI) models for TB detection. 1097 CXRs from five institutions were analyzed by six B readers. Patients aged ≥ 15 years were included, excluding those with HIV or opportunistic infections. CXRs were classified as unremarkable or abnormal. Abnormalities were categorized using a modified International Labor Organization classification and classified as consistent or inconsistent with TB. Microbiological references included sputum smears, cultures, or molecular tests. Descriptive statistics summarized the CXR and microbiological findings. Inter-rater agreement was assessed with Fleiss' kappa. Diagnostic performance was evaluated by comparing CXR findings to microbiological references. Of the 3117 readings, 69% of CXRs were abnormal, and 31% were unremarkable. Microbiological results confirmed that 87% of abnormal CXRs were TB cases, while 83% of unremarkable CXRs were non-TB. Inter-rater agreements were κ = 0.83 for all findings, κ = 0.67 for findings consistent with TB, and κ = 0.76 for active TB findings. For findings consistent with TB, the sensitivity, specificity, and accuracy ranged from 77.2% to 91.1%, 87.4% to 98.6%, and 84.1% to 90.1%, respectively. B readers exhibited strong agreement and high accuracy in diagnosing TB on CXR, providing a robust dataset that could be used for external validation of AI models in TB diagnosis. With substantial inter-rater agreement and strong diagnostic performance, B readers' interpretations of CXRs provide a reliable dataset to support external validation of AI models for TB diagnosis. B readers can provide a reliable reference for validating AI models in TB diagnosis. Less variability and high accuracy were observed among B readers in CXR-based TB diagnosis.