A speech-to-video synthesis approach using spatio-temporal diffusion for vocal tract MRI.
Authors
Affiliations (12)
Affiliations (12)
- Harvard Medical School/Massachusetts General Hospital, Boston, 02114, Massachusetts, USA; Pattern Recognition Lab, Department of Computer Science, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, 91058, Bayern, Germany; GITA Lab, Faculty of Engineering, Universidad de Antioquia, Medellín, 050010, Antioquia, Colombia. Electronic address: [email protected].
- Harvard Medical School/Massachusetts General Hospital, Boston, 02114, Massachusetts, USA; Pattern Recognition Lab, Department of Computer Science, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, 91058, Bayern, Germany; GITA Lab, Faculty of Engineering, Universidad de Antioquia, Medellín, 050010, Antioquia, Colombia. Electronic address: [email protected].
- Harvard Medical School/Massachusetts General Hospital, Boston, 02114, Massachusetts, USA. Electronic address: [email protected].
- Department of Radiology & Biomedical Imaging and Biomedical Informatics & Data Science, Yale University, New Heaven, 06510, Connecticut, USA. Electronic address: [email protected].
- Department of Neural and Pain Sciences and Department of Orthodontics and Pediatrics, University of Maryland School of Dentistry, Baltimore, 21210, Maryland, USA; Department of Orthodontics and Pediatrics, University of Maryland School of Dentistry, Baltimore, 21201, Maryland, USA. Electronic address: [email protected].
- Department of Neural and Pain Sciences and Department of Orthodontics and Pediatrics, University of Maryland School of Dentistry, Baltimore, 21210, Maryland, USA. Electronic address: [email protected].
- Pattern Recognition Lab, Department of Computer Science, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, 91058, Bayern, Germany; GITA Lab, Faculty of Engineering, Universidad de Antioquia, Medellín, 050010, Antioquia, Colombia. Electronic address: [email protected].
- Pattern Recognition Lab, Department of Computer Science, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, 91058, Bayern, Germany. Electronic address: [email protected].
- Smart Imaging Lab, Radiological Institute, University Hospital Erlangen, Erlangen, 91052, Bayern, Germany. Electronic address: [email protected].
- Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, 21218, Maryland, USA. Electronic address: [email protected].
- Pattern Recognition Lab, Department of Computer Science, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, 91058, Bayern, Germany. Electronic address: [email protected].
- Harvard Medical School/Massachusetts General Hospital, Boston, 02114, Massachusetts, USA. Electronic address: [email protected].
Abstract
Understanding the relationship between vocal tract motion during speech and the resulting acoustic signal is crucial for aided clinical assessment and developing personalized treatment and rehabilitation strategies. Toward this goal, we introduce an audio-to-video generation framework for creating Real Time/cine-Magnetic Resonance Imaging (RT-/cine-MRI) visuals of the vocal tract from speech signals. Our framework first preprocesses RT-/cine-MRI sequences and speech samples to achieve temporal alignment, ensuring synchronization between visual and audio data. We then employ a modified stable diffusion model, integrating structural and temporal blocks, to effectively capture movement characteristics and temporal dynamics in the synchronized data. This process enables the generation of MRI sequences from new speech inputs, improving the conversion of audio into visual data. We evaluated our framework on healthy controls and tongue cancer patients by analyzing and comparing the vocal tract movements in synthesized videos. Our framework demonstrated adaptability to new speech inputs and effective generalization. In addition, positive human evaluations confirmed its effectiveness, with realistic and accurate visualizations, suggesting its potential for outpatient therapy and personalized simulation of vocal tract visualizations.