- Handong Global University’s Deep Learning Lab Developed the World's First Emotion-Expressing Song Voice Synthesis AI 'MuseSVS'
%20MuseSVS%EC%9D%98%20%EA%B5%AC%EC%A1%B0(1).jpg)
(Photo 1) Structure of MuseSVS
The Deep Learning Research Lab of Handong Global University (President Dosoung Choi) has developed the world's first emotion-expressing multi-speaker song voice synthesis AI called 'MuseSVS'.
Singing Voice Synthesis (SVS) is an AI technology that synthesizes singing voices from musical scores. While previous AI systems could only synthesize voices singing the lyrics, there were no AIs capable of expressing emotions through voice synthesis until now. The recently developed MuseSVS by Handong Global University takes musical scores and additional inputs, including the speaker's emotional types and intensities, to produce singing voices with varying tones, vibrato, and volume. Notably, the statistical pitch predictor and context-based note duration predictor, applied for the first time in this singing voice synthesis AI, effectively generate subtle changes in pitch and rhythm required for expressing emotions. MuseSVS has been published in the prestigious IEEE/ACM Transactions on Audio, Speech, and Language Processing (TASLP), the world's top journal in the field of voice processing.
%20MuseSVS%EA%B0%80%20%ED%95%A9%EC%84%B1%ED%95%9C%20%EB%85%B8%EB%9E%98%EC%9D%8C%EC%84%B1%EC%9D%98%20%EC%9D%8C%EC%A0%95%EB%B3%80%ED%99%94%20(%EC%8A%AC%ED%94%94%EC%9D%B4%20%EA%B0%95%ED%95%B4%EC%A7%88%EC%88%98%EB%A1%9D%20%EA%B0%95%ED%95%9C%20%EB%B9%84%EB%B8%8C%EB%9D%BC%ED%86%A0%EB%A5%BC%20%ED%95%A9%EC%84%B1%ED%95%9C%EB%8B%A4)(1).jpg)
(Photo 2) The pitch variation of the song voice synthesized by MuseSVS
(Stronger vibrato is synthesized as sadness intensifies).
Professor In Jung Kim, the guiding professor at Handong Global University's Deep Learning Research Lab, stated, "When people sing the same song, they express emotions through subtle changes in pitch, rhythm, and volume. MuseSVS effectively learns these fine variations from the data." He further explained, "MuseSVS follows the macro flow of pitch and rhythm according to the musical score, while synthesizing micro-level variations differently based on the specified emotion type and intensity." The development of MuseSVS was led by Sungjae Kim (Handong Global University Ph.D. candidate), with the participation of Handong Global University students Yewon Kim (Master's candidate) and Jewoo Jeon (undergraduate student).
