Frame forecasting in cine MRI using the PCA respiratory motion model: comparing recurrent neural networks trained online and transformers.
Authors
Affiliations (4)
Affiliations (4)
- The University of Tokyo, 113-8654 Tokyo, Japan. Electronic address: [email protected].
- Japan Atomic Energy Commission, 100-8914 Tokyo, Japan.
- The University of Tokyo, 113-8654 Tokyo, Japan.
- National Institutes for Quantum and Radiological Science and Technology, 263-8555 Chiba, Japan.
Abstract
Respiratory-induced motion complicates accurate irradiation of thoraco-abdominal tumors during radiotherapy, as treatment-system latency entails target-location uncertainties. This work addresses frame forecasting in dynamic chest and liver MRI to compensate for such delays. We investigate RNNs trained with online learning algorithms, enabling adaptation to changing respiratory patterns via on-the-fly parameter updates, and transformers, increasingly common in time-series forecasting for their ability to capture long-term dependencies. Experiments were conducted using twelve sagittal thoracic and upper-abdominal cine-MRI sequences from ETH Zürich and Otto-von-Guericke University Magdeburg (OvGU); the OvGU data exhibited higher motion variability, noise, and lower contrast. PCA decomposes the Lucas-Kanade optical-flow field into static deformation modes and low-dimensional, time-dependent weights. We compare various methods for forecasting these weights: linear filters, population and sequence-specific encoder-only transformers, and RNNs trained with real-time recurrent learning (RTRL), unbiased online recurrent optimization, decoupled neural interfaces, and sparse one-step approximation (SnAp-1). Predicted displacements were used to warp the reference frame and generate future images. Prediction accuracy decreased with the horizon h. Linear regression performed best at short horizons (1.3 mm geometrical error at h=0.32s, ETH Zürich dataset), while RTRL and SnAp-1 outperformed the other algorithms at medium-to-long horizons, with geometrical errors below 1.4 mm and 2.8 mm on the sequences from ETH Zürich and OvGU, respectively. The sequence-specific transformer was competitive for low-to-medium horizons, but transformers remained overall limited by data scarcity and domain shift between datasets. Predicted frames visually resembled the ground truth, with notable errors occurring near the diaphragm at end-inspiration and regions affected by out-of-plane motion.