Microsoft Edge will dub streamed video with AI-translated audio

Microsoft is planning to either add subtitles or even dub video produced by major video sites, using AI to translate the audio into foreign languages within Microsoft Edge in real time.

At its Microsoft Build developer conference, Microsoft named several sites that would benefit from the new real-time translation capabilities within Edge, including Reuters, CNBC News, Bloomberg, and Coursera, plus Microsoft’s own LinkedIn.

Interestingly, Microsoft also named Google’s YouTube as a beneficiary of the translation capabilities. (That’s slightly odd, because YouTube offers AI closed-captioning, shown above, which can be auto-translated by adjusting the video’s settings. YouTube does not offer dubbed audio.)

Microsoft plans to translate the video from Spanish to English and from English to German, Hindi, Italian, Russian, and Spanish. There are plans to add additional languages and video platforms in the future, Microsoft said.

AI translation has been a staple of Microsoft apps for some time — Microsoft Teams, for example, announced inline message translation in 2018. In 2022, Microsoft announced live translation for captions for Teams Premium subscribers. Microsoft Edge itself has supported translation of a web page or selected text since 2020. Live Captions, added as part of the Windows 11 2022 Update, adds AI-generated captions to streamed and even pre-recorded video, such as a family reunion. But it doesn’t perform translation.

But what Teams and other Microsoft apps haven’t done is perform dubbing, which would replace the spoken audio with an AI-generated voice, translated into the user’s native language.

And doing so can be expensive. There are a number of companies that perform AI-powered dubbing. The “personal” plan of synthesys.io offers just 15 minutes of audio/video generation per month, for $20. (A “Creator Unlimited” plan costing $41 per month offers unlimited minutes.) ElevenLabs.io will do it for free, provided you give the service the URL of the YouTube video. But translating a 3-minute video apparently stalled out over more than a half hour, after which I gave up.

Automatic translation of web pages is one of the most useful features of a web browser. Will translated or dubbed video be as pervasive? Most people, regardless of what language they speak, know where to go for sources of news in their local language. But this will certainly be a differentiator for Edge, as well as a useful tool in case you absolutely need it.

Personal Software, Windows