How Do Instructions Shape Speech? Cross-Attention Attribution for Style-Captioned Text-to-Speech
Researchers have developed a new method to understand how natural language instructions influence the output of style-captioned text-to-speech (TTS) systems. By adapting the DAAM framework to speech diffusion models, the study analyzes how specific words in style captions shape the generated waveforms. The findings indicate that style tokens have a lower temporal variance than content tokens and that their influence peaks in the early stages of generation and deeper layers of the model. AI
IMPACT Provides a deeper understanding of controllability in expressive TTS systems, potentially leading to improved voice generation.