SoftSpeech: Unsupervised Duration Model In FastSpeech 2

Microsoft, China

{yuanhyi,helei,peterpan,xwang,yuchao.zhang}@microsoft.com

In this paper, we propose a neural Text-To-Speech (TTS) system SoftSpeech, which employs a novel soft length regulated duration attention based decoder. It learns the encoder output mapping to decoder output simultaneously from an unsupervised duration model (Soft-LengthRegulator) without the requirement of external duration information. The Soft-LengthRegulator is consist of a Feed-Forward Transformer (FFT) block with Conditional Layer Normalization (CLN), following a learned upsampling layer with multi-head attention and guided multi-head attention constraint, and it is integrated in each decoder layer and achieves accelerated training convergence and better naturalness within FastSpeech 2 framework. Soft Dynamic Time Warping (Soft-DTW) is adopted to align the mismatch spectrogram loss. Moreover, a Fine-Grained style Variational AutoEncoder (VAE) is designed to further improve the naturalness of synthesized speech. The experiments show SoftSpeech outperforms FastSpeech 2 in subjective tests, and can be successfully applied to minority languages with low resources.

Multi-lingual Multi-speaker TTS task:

Voice	Recording	FastSpeech 2	SoftSpeech
Madarin Male	这世界上有千百种各式各样的死法,可你知道最恐怖的死法是哪一种吗? 咕噜噜……一个黑影滚到了我的脚下,虽然此刻周围的灯光昏暗阴森,我却依然能清楚的看到那是什么。	看着江开张大着嘴，死不瞑目的面容，我先是觉得脖子后面，凉风嗖嗖直冒。 就在几个小时之前，这个来自东北的大汉，还跟我谈笑风生，约定了回去之后，一定要去我们俩常去的那家烧烤摊好好搓上一顿。	看着江开张大着嘴，死不瞑目的面容，我先是觉得脖子后面，凉风嗖嗖直冒。 就在几个小时之前，这个来自东北的大汉，还跟我谈笑风生，约定了回去之后，一定要去我们俩常去的那家烧烤摊好好搓上一顿。
Madarin Female	李克强强调，发展的根本是为了民生。 同时还要树立正确的投资理念，基金产品能够提供长期的保值增值机会，但不会导致短期的暴富。	4000光年外发现超高密度行星：质量为木星4倍。 解析1690年-2011年美国报业发展史。	4000光年外发现超高密度行星：质量为木星4倍。 解析1690年-2011年美国报业发展史。
English Male	Got my girlfriend a Nicolas Cage themed Valentines Day card, since that's her least favorite actor. The Game Coordinator will be down for about an hour starting at four p.m. pacific standard time tomorrow.	It’s simply the most fun part about teaching. Netanyahu has taken a more cautious approach to Gaza, saying he wants to avoid war if it is unnecessary.	It’s simply the most fun part about teaching. Netanyahu has taken a more cautious approach to Gaza, saying he wants to avoid war if it is unnecessary.
English Female	But thousands of acres are still smoldering , and the evacuees have not been allowed back home . It means everything to Becky , Stephanie , Marcus and Emily , and students like them .	There are strengths at all three levels of the defense. When what you've lost is so profound?	There are strengths at all three levels of the defense. When what you've lost is so profound?

Minority low resource TTS task:

Voice	Recording	FastSpeech 2	SoftSpeech
Estonian Male	Inspiratsiooniallikaks nimetab Nerva looduses esinevat värvikooslust ja kompositsioonilisi lahendusi ning suurimaks eeskujuks peab Loojat. Seega tuleb ka olukorras, kus Te neljandal kuul tööd ei tee, maksta Teile kokkulepitud kuutasu.	Ameerika meedia vahendab, kuidas maruvihased rahvamassid süütavad ja rüüstavad hooneid ning loobivad korrakaitsjaid kividega. Mängiti kaheksas alagrupis, edasi pääsesid iga grupi kolm esimest.	Ameerika meedia vahendab, kuidas maruvihased rahvamassid süütavad ja rüüstavad hooneid ning loobivad korrakaitsjaid kividega. Mängiti kaheksas alagrupis, edasi pääsesid iga grupi kolm esimest.
Estonian Female	Võidud jagunesid võrdselt, kusjuures esimesed kolm kohtumist Kehras võitis alati kodumeeskond. 29-aastane Bertans mängis viimased poolteist hooaega Euroliiga klubis Milano Armani.	Suurema osa oma elust veedab inimene paratamatult siseruumides, mistõttu on äärmiselt oluline ruumide õhu kvaliteet. India koroonapatsiendid on sunnitud haiglavoodeid jagama.	Suurema osa oma elust veedab inimene paratamatult siseruumides, mistõttu on äärmiselt oluline ruumide õhu kvaliteet. India koroonapatsiendid on sunnitud haiglavoodeid jagama.
Maltese Male	Se jinfetħu l-kumplament tal-ħwienet u ċ-ċentri tal-ħarsien tat-tfal. Il-Mexxej Laburista qal li Dr Delia moħħu f'elezzjoni ġenerali bikrija, f'mument meta l-pajjiż għandu bżonn stabbiltà.	L-ismijiet tal-ilwien fil-Malti jinqasmu fi tlieta. Anzjan weġġa' gravi wara li ttajjar f'Ħaż-Żabbar	L-ismijiet tal-ilwien fil-Malti jinqasmu fi tlieta. Anzjan weġġa' gravi wara li ttajjar f'Ħaż-Żabbar
Maltese Female	B’ideat u kontribuzzjonijiet miż-żgħażagħ, ir-riżultati jistgħu jkunu ħafna iktar effettivi. Fil-ħames snin tal-amministrazzjoni preċedenti, il-kontijiet tad-dawl għolew b’68% f’pajjiżna.	Minn għada l-iskeda televiżiva ta' Television Malta se tkompli tingħata l-ħajja permezz ta’ għadd ta' xandiriet ġodda. Il-baritonu Joseph Lia u martu, ir-Russa Natalia Rakjmatulina, ħassru diversi kunċerti fosthom fir-Russja.	Minn għada l-iskeda televiżiva ta' Television Malta se tkompli tingħata l-ħajja permezz ta’ għadd ta' xandiriet ġodda. Il-baritonu Joseph Lia u martu, ir-Russa Natalia Rakjmatulina, ħassru diversi kunċerti fosthom fir-Russja.