This study aimed to develop an open-source multimodal large language model (CXR-LLaVA) for interpreting chest X-ray images (CXRs), leveraging recent advances in large language models (LLMs) to potentially replicate the image interpretation skills of human radiologists. For training, we collected 592,580 publicly available CXRs, of which 374,881 had labels for certain radiographic abnormalities (Dataset 1) and 217,699 provided free-text radiology reports (Dataset 2). After pre-training a vision transformer with Dataset 1, we integrated it with an LLM influenced by the LLaVA network. Then, the model was fine-tuned, primarily using Dataset 2. The model's diagnostic performance for major pathological findings was evaluated, along with the acceptability of radiologic reports by human radiologists, to gauge its potential for autonomous reporting. The model demonstrated impressive performance in test sets, achieving an average F1 score of 0.81 for six major pathological findings in the MIMIC internal test set and 0.56 for six major pathological findings in the external test set. The model's F1 scores surpassed those of GPT-4-vision and Gemini-Pro-Vision in both test sets. In human radiologist evaluations of the external test set, the model achieved a 72.7% success rate in autonomous reporting, slightly below the 84.0% rate of ground truth reports. This study highlights the significant potential of multimodal LLMs for CXR interpretation, while also acknowledging the performance limitations. Despite these challenges, we believe that making our model open-source will catalyze further research, expanding its effectiveness and applicability in various clinical contexts. Question How can a multimodal large language model be adapted to interpret chest X-rays and generate radiologic reports? Findings The developed CXR-LLaVA model effectively detects major pathological findings in chest X-rays and generates radiologic reports with a higher accuracy compared to general-purpose models. Clinical relevance This study demonstrates the potential of multimodal large language models to support radiologists by autonomously generating chest X-ray reports, potentially reducing diagnostic workloads and improving radiologist efficiency.

HER2 expression is crucial for the application of HER2-targeted antibody-drug conjugates. This study aims to construct a predictive model by integrating multiparametric magnetic resonance imaging (mpMRI) based multimodal radiomics and the Vesical Imaging-Reporting and Data System (VI-RADS) score for noninvasive identification of HER2 status in bladder urothelial carcinoma (BUC). A total of 197 patients were retrospectively enrolled and randomly divided into a training cohort (n = 145) and a testing cohort (n = 52). The multimodal radiomics features were derived from mpMRI, which were also utilized for VI-RADS score evaluation. LASSO algorithm and six machine learning methods were applied for radiomics feature screening and model construction. The optimal radiomics model was selected to integrate with VI-RADS score to predict HER2 status, which was determined by immunohistochemistry. The performance of predictive model was evaluated by receiver operating characteristic curve with area under the curve (AUC). Among the enrolled patients, 110 (55.8%) patients were demonstrated with HER2-positive and 87 (44.2%) patients were HER2-negative. Eight features were selected to establish radiomics signature. The optimal radiomics signature achieved the AUC values of 0.841 (95% CI 0.779-0.904) in the training cohort and 0.794 (95%CI 0.650-0.938) in the testing cohort, respectively. The KNN model was selected to evaluate the significance of radiomics signature and VI-RADS score, which were integrated as a predictive nomogram. The AUC values for the nomogram in the training and testing cohorts were 0.889 (95%CI 0.840-0.938) and 0.826 (95%CI 0.702-0.950), respectively. Our study indicated the predictive model based on the integration of mpMRI-based radiomics and VI-RADS score could accurately predict HER2 status in BUC. The model might aid clinicians in tailoring individualized therapeutic strategies.

Pancreatic ductal adenocarcinoma (PDAC) is the third leading cause of cancer-related deaths in the United States, largely due to its poor five-year survival rate and frequent late-stage diagnosis. A significant barrier to early detection even in high-risk cohorts is that the pancreas often appears morphologically normal during the pre-diagnostic phase. Yet, the disease can progress rapidly from subclinical stages to widespread metastasis, undermining the effectiveness of screening. Recently, artificial intelligence (AI) applied to cross-sectional imaging has shown significant potential in identifying subtle, early-stage changes in pancreatic tissue that are often imperceptible to the human eye. Moreover, AI-driven imaging also aids in the discovery of prognostic and predictive biomarkers, essential for personalized treatment planning. This article uniquely integrates a critical discussion on AI's role in detecting visually occult PDAC on pre-diagnostic imaging, addresses challenges of model generalizability, and emphasizes solutions like standardized datasets and clinical workflows. By focusing on both technical advancements and practical implementation, this article provides a forward-thinking conceptual framework that bridges current gaps in AI-driven PDAC research.

Definitive chemoradiation is the primary treatment for locally advanced head and neck carcinoma (LAHNSCC). Optimising outcome predictions requires validated biomarkers, since TNM8 and HPV could have limitations. Radiomics may enhance risk stratification. This single-centre observational study collected clinical data and baseline CT scans from 171 LAHNSCC patients treated with chemoradiation. The dataset was divided into training (80%) and test (20%) sets, with a 5-fold cross-validation on the training set. Researchers extracted 108 radiomics features from each primary tumour and applied survival analysis and classification models to predict progression-free survival (PFS) and 5-year progression, respectively. Performance was evaluated using inverse probability of censoring weights and c-index for the PFS model and AUC, sensitivity, specificity, and accuracy for the 5-year progression model. Feature importance was measured by the SHapley Additive exPlanations (SHAP) method and patient stratification was assessed through Kaplan-Meier curves. The final dataset included 171 LAHNSCC patients, with 53% experiencing disease progression at 5 years. The random survival forest model best predicted PFS, with an AUC of 0.64 and CI of 0.66 on the test set, highlighting 4 radiomics features and TNM8 as significant contributors. It successfully stratified patients into low and high-risk groups (log-rank p < 0.005). The extreme gradient boosting model most effectively predicted a 5-year progression, incorporating 12 radiomics features and four clinical variables, achieving an AUC of 0.74, sensitivity of 0.53, specificity of 0.81, and accuracy of 0.66 on the test set. The combined clinical-radiomics model improved the standard TNM8 and clinical variables in predicting 5-year progression though further validation is necessary. Question There is an unmet need for non-invasive biomarkers to guide treatment in locally advanced head and neck cancer. Findings Clinical data (TNM8 staging, primary tumour site, age, and smoking) plus radiomics improved 5-year progression prediction compared with the clinical comprehensive model or TNM staging alone. Clinical relevance SHAP simplifies complex machine learning radiomics models for clinicians by using easy-to-understand graphical representations, promoting explainability.

To develop and evaluate a physics-driven, saturation contrast-aware, deep-learning-based framework for motion artifact correction in CEST MRI. A neural network was designed to correct motion artifacts directly from a Z-spectrum frequency (Ω) domain rather than an image spatial domain. Motion artifacts were simulated by modeling 3D rigid-body motion and readout-related motion during k-space sampling. A saturation-contrast-specific loss function was added to preserve amide proton transfer (APT) contrast, as well as enforce image alignment between motion-corrected and ground-truth images. The proposed neural network was evaluated on simulation data and demonstrated in healthy volunteers and brain tumor patients. The experimental results showed the effectiveness of motion artifact correction in the Z-spectrum frequency domain (MOCO_Ω) compared to in the image spatial domain. In addition, a temporal convolution applied to a dynamic saturation image series was able to leverage motion artifacts to improve reconstruction results as a denoising process. The MOCO_Ω outperformed existing techniques for motion correction in terms of image quality and computational efficiency. At 3 T, human experiments showed that the root mean squared error (RMSE) of APT images decreased from 4.7% to 2.1% at 1 μT and from 6.2% to 3.5% at 1.5 μT in case of "moderate" motion and from 8.7% to 2.8% at 1 μT and from 12.7% to 4.5% at 1.5 μT in case of "severe" motion, after motion artifact correction. The MOCO_Ω could effectively correct motion artifacts in CEST MRI without compromising saturation transfer contrast.

Osteoarthritis (OA) is heterogeneous and involves structural changes in the whole joint, such as cartilage, meniscus/labrum, ligaments, and tendons, mainly with short T2 relaxation times. Detecting OA before the onset of irreversible changes is crucial for early proactive management and limit growing disease burden. The more recent advanced quantitative imaging techniques and deep learning (DL) algorithms in musculoskeletal imaging have shown great potential for visualizing "pre-OA." In this review, we first focus on ultrashort echo time-based magnetic resonance imaging (MRI) techniques for direct visualization as well as quantitative morphological and compositional assessment of both short- and long-T2 musculoskeletal tissues, and second explore how DL revolutionize the way of MRI analysis (eg, automatic tissue segmentation and extraction of quantitative image biomarkers) and the classification, prediction, and management of OA. PLAIN LANGUAGE SUMMARY: Detecting osteoarthritis (OA) before the onset of irreversible changes is crucial for early proactive management. OA is heterogeneous and involves structural changes in the whole joint, such as cartilage, meniscus/labrum, ligaments, and tendons, mainly with short T2 relaxation times. Ultrashort echo time-based magnetic resonance imaging (MRI), in particular, enables direct visualization and quantitative compositional assessment of short-T2 tissues. Deep learning is revolutionizing the way of MRI analysis (eg, automatic tissue segmentation and extraction of quantitative image biomarkers) and the detection, classification, and prediction of disease. They together have made further advances toward identification of imaging biomarkers/features for pre-OA. LEVEL OF EVIDENCE: 5 TECHNICAL EFFICACY: Stage 2.

Pancreatic ductal adenocarcinoma (PDAC) is the deadliest malignant tumor, with a grim 5-year overall survival rate of about 12%. As its incidence and mortality rates rise, it is likely to become the second-leading cause of cancer-related death. The radiological assessment determined the stage and management of PDAC. However, it is a highly heterogeneous disease with the complexity of the tumor microenvironment, and it is challenging to adequately reflect the biological aggressiveness and prognosis accurately through morphological evaluation alone. With the dramatic development of artificial intelligence (AI), multiparametric magnetic resonance imaging (mpMRI) using specific contrast media and special techniques can provide morphological and functional information with high image quality and become a powerful tool in quantifying intratumor characteristics. Besides, AI has been widespread in the field of medical imaging analysis. Radiomics is the high-throughput mining of quantitative image features from medical imaging that enables data to be extracted and applied for better decision support. Deep learning is a subset of artificial neural network algorithms that can automatically learn feature representations from data. AI-enabled imaging biomarkers of mpMRI have enormous promise to bridge the gap between medical imaging and personalized medicine and demonstrate huge advantages in predicting biological characteristics and the prognosis of PDAC. However, current AI-based models of PDAC operate mainly in the realm of a single modality with a relatively small sample size, and the technical reproducibility and biological interpretation present a barrage of new potential challenges. In the future, the integration of multi-omics data, such as radiomics and genomics, alongside the establishment of standardized analytical frameworks will provide opportunities to increase the robustness and interpretability of AI-enabled image biomarkers and bring these biomarkers closer to clinical practice. EVIDENCE LEVEL: 3 TECHNICAL EFFICACY: Stage 4.

To evaluate the repeatability of AI-based automatic measurement of vertebral and cardiovascular markers on low-dose chest CT. We included participants of the population-based Imaging in Lifelines (ImaLife) study with low-dose chest CT at baseline and 3-4 month follow-up. An AI system (AI-Rad Companion chest CT prototype) performed automatic segmentation and quantification of vertebral height and density, aortic diameters, heart volume (cardiac chambers plus pericardial fat), and coronary artery calcium volume (CACV). A trained researcher visually checked segmentation accuracy. We evaluated the repeatability of adequate AI-based measurements at baseline and repeat scan using Intraclass Correlation Coefficient (ICC), relative differences, and change in CACV risk categorization, assuming no physiological change. Overall, 632 participants (63 ± 11 years; 56.6% men) underwent short-term repeat CT (mean interval, 3.9 ± 1.8 months). Visual assessment showed adequate segmentation in both baseline and repeat scan for 98.7% of vertebral measurements, 80.1-99.4% of aortic measurements (except for the sinotubular junction (65.2%)), and 86.0% of CACV. For heart volume, 53.5% of segmentations were adequate at baseline and repeat scans. ICC for adequately segmented cases showed excellent agreement for all biomarkers (ICC > 0.9). Relative difference between baseline and repeat measurements was < 4% for vertebral and aortic measurements, 7.5% for heart volume, and 28.5% for CACV. There was high concordance in CACV risk categorization (81.2%). In low-dose chest CT, segmentation accuracy of AI-based software was high for vertebral, aortic, and CACV evaluation and relatively low for heart volume. There was excellent repeatability of vertebral and aortic measurements and high concordance in overall CACV risk categorization. Question Can AI algorithms for opportunistic screening in chest CT obtain an accurate and repeatable result when applied to multiple CT scans of the same participant? Findings Vertebral and aortic analysis showed accurate segmentation and excellent repeatability; coronary calcium segmentation was generally accurate but showed modest repeatability due to a non-electrocardiogram-triggered protocol. Clinical relevance Opportunistic screening for diseases outside the primary purpose of the CT scan is time-consuming. AI allows automated vertebral, aortic, and coronary artery calcium (CAC) assessment, with highly repeatable outcomes of vertebral and aortic biomarkers and high concordance in overall CAC categorization.

Accurate preoperative evaluation of myometrial invasion (MI) is essential for treatment decisions in endometrial cancer (EC). However, the diagnostic accuracy of commonly utilized magnetic resonance imaging (MRI) techniques for this assessment exhibits considerable variability. This study aims to enhance preoperative discrimination of absence or presence of MI by developing and validating a multimodal deep learning radiomics (MDLR) model based on MRI. During March 2010 and February 2023, 1139 EC patients (age 54.771 ± 8.465 years; range 24-89 years) from five independent centers were enrolled retrospectively. We utilized ResNet18 to extract multi-scale deep learning features from T2-weighted imaging followed by feature selection via Mann-Whitney U test. Subsequently, a Deep Learning Signature (DLS) was formulated using Integrated Sparse Bayesian Extreme Learning Machine. Furthermore, we developed Clinical Model (CM) based on clinical characteristics and MDLR model by integrating clinical characteristics with DLS. The area under the curve (AUC) was used for evaluating diagnostic performance of the models. Decision curve analysis (DCA) and integrated discrimination index (IDI) were used to assess the clinical benefit and compare the predictive performance of models. The MDLR model comprised of age, histopathologic grade, subjective MR findings (TMD and Reading for MI status) and DLS demonstrated the best predictive performance. The AUC values for MDLR in training set, internal validation set, external validation set 1, and external validation set 2 were 0.899 (95% CI, 0.866-0.926), 0.874 (95% CI, 0.829-0.912), 0.862 (95% CI, 0.817-0.899) and 0.867 (95% CI, 0.806-0.914) respectively. The IDI and DCA showed higher diagnostic performance and clinical net benefits for the MDLR than for CM or DLS, which revealed MDLR may enhance decision-making support. The MDLR which incorporated clinical characteristics and DLS could improve preoperative accuracy in discriminating absence or presence of MI. This improvement may facilitate individualized treatment decision-making for EC.

To develop a deep subspace learning network that can function across different pulse sequences. A contrast-invariant component-by-component (CBC) network structure was developed and compared against previously reported spatiotemporal multicomponent (MC) structure for reconstructing MR Multitasking images. A total of 130, 167, and 16 subjects were imaged using T₁, T₁-T₂, and T₁-T₂- <math xmlns="http://www.w3.org/1998/Math/MathML"> <semantics> <mrow><msubsup><mi>T</mi> <mn>2</mn> <mo>*</mo></msubsup> </mrow> <annotation>$$ {\mathrm{T}}_2^{\ast } $$</annotation></semantics> </math> -fat fraction (FF) mapping sequences, respectively. We compared CBC and MC networks in matched-sequence experiments (same sequence for training and testing), then examined their cross-sequence performance and generalizability by unmatched-sequence experiments (different sequences for training and testing). A "universal" CBC network was also evaluated using mixed-sequence training (combining data from all three sequences). Evaluation metrics included image normalized root mean squared error and Bland-Altman analyses of end-diastolic maps, both versus iteratively reconstructed references. The proposed CBC showed significantly better normalized root mean squared error than MC in both matched-sequence and unmatched-sequence experiments (p < 0.001), fewer structural details in quantitative error maps, and tighter limits of agreement. CBC was more generalizable than MC (smaller performance loss; p = 0.006 in T₁ and p < 0.001 in T₁-T₂ from matched-sequence testing to unmatched-sequence testing) and additionally allowed training of a single universal network to reconstruct images from any of the three pulse sequences. The mixed-sequence CBC network performed similarly to matched-sequence CBC in T₁ (p = 0.178) and T₁-T₂ (p = 0121), where training data were plentiful, and performed better in T₁-T₂- <math xmlns="http://www.w3.org/1998/Math/MathML"> <semantics> <mrow><msubsup><mi>T</mi> <mn>2</mn> <mo>*</mo></msubsup> </mrow> <annotation>$$ {\mathrm{T}}_2^{\ast } $$</annotation></semantics> </math> -FF (p < 0.001) where training data were scarce. Contrast-invariant learning of spatial features rather than spatiotemporal features improves performance and generalizability, addresses data scarcity, and offers a pathway to universal supervised deep subspace learning.