DiffStyleTTS: Diffusion-based Hierarchical Prosody Modeling for Text-to-Speech with Diverse and Controllable Styles

Abstract

Human speech exhibits rich and flexible prosodic variations. To address the one-to-many mapping problem from text to prosody in a reasonable and flexible manner, we propose DiffStyleTTS, a multi-speaker acoustic model based on a conditional diffusion module and an improved classifier-free guidance strategy. The model hierarchically models speech prosodic features and steers different prosodic styles to guide prosody prediction. Experiments show that DiffStyleTTS outperforms representative baselines in naturalness and achieves superior synthesis speed compared to three diffusion-based baselines. By adjusting the guiding scale, DiffStyleTTS effectively controls the guidance intensity over synthesized prosody — combining diversity and controllability of speech prosody.

Jiaxuan Liu

M.Eng. Student in Information & Communication Engineering · Speech AI Researcher

My research focuses on text-to-speech, expressive & emotional speech synthesis, multimodal movie dubbing, and speech foundation models.