Yandex released new solution for voice-over real-time translation for videos on YouTube and Vimeo and several other platforms.

The volume level of the original soundtrack is reduced, while the translation comes to the fore. For voice acting of the translated text, you can choose male and female voices. In the second case, Yandex' voice assistant Alice is used. The voice-over function is available via Yandex Browser and in the Yandex mobile app.

According to the company, the translation takes place in several stages. First comes speech recognition, turning it into text and breaking it down into sentences using neural networks. Then the gender of the speaker is determined using biometric algorithms, the sentences are translated into Russian, and the voice is synthesized. Finally, the translation is combined with the video sequence, for which neural networks are also used, in particular, to synchronize Russian sentences with shorter English ones.

Connect with

Andrew Zakonov, CPO Smart Devices in Yandex

Product cards

Alice

Yandex first announced its system for automatic translation of video clips in July this year. The prototype was available only for English and Russian, but the company plans to further extend the service for several other languages.

Original (in Russian)