Artem (00:07)
And so it was always some kind of dream to be able to speak any language.

Klemen (00:12)
Yeah, yeah people switch languages like mid sentence or you know just quite often.

Lily (00:18)
First is we home the voice and then we translate the language and then we make sure it fits a video lens and with a portal one hundred fifty plus different language. If you train one model for every language, like not a very good model for for these use cases.

Hermes (00:32)
Hey everyone, welcome to the Convoy I World Podcast. Today, we have a very special episode focused on real-time translation. We're speaking with the researchers, developers, and founders helping people have conversations across languages. We'll explore what it takes to translate speech as people talk, preserve meaning and tone, and keep a conversation flowing without making anyone wait. Let's dive in.

There's constantly points in my daily life where I'm either on a video call, watching a live stream of some sort, and you're not always like the native language, even as English, right? And so that really breaks down the barriers of knowledge sharing. So you you mention having rolled this out in in different virtual events, live events, that sort of thing. What kind of feedback are you getting from people? Like how is this impacting the end user? Because that that that's the real magic, right? Like the impact it has on the person actually getting to use it.

Ivan (01:28)
Well, as I told you before, now it feels like magic. Like I mean you you live in in the b publica era when the the tower is still building and you can speak all the languages. Yeah. and that's the case. It's like I was having a sales pitch to one LSP in Argentina and the girls on the other side they couldn't speak good English, so they were like shy, couldn't they couldn't convey like correctly what they wanted to buy and then

I just switched on palabra, like with a push of a button, they changed like three s like one eighty. They're they're like different personalities. I could read their personality, they could read mine, even though we were speaking different languages. I I even stopped speaking stopped speaking English. I switched to Hebrew, but

Hermes (02:12)
It's fine. It it's like that comfort zone, right? It puts you in a more comfortable spot. It gets people more open to engaging and again breaks down those barriers. Like you were saying, they came in not so easy to communicate, right?

Ivan (02:26)
Transfer is meaning what's important.

Hermes (02:28)
That's the huge piece. The meaning, the feeling

Ivan (02:31)
Right.

Feeling the the context. You never get out of context. And that's not just a technology. We're speaking about live magic like coming through behind a just here behind a

Hermes (02:42)
Yeah, it's amazing. Like the the technology's moving so fast. It feels like we're living in the future.

So the AI does its thing, right? It it recognizes patterns, it recognizes similarities. And that that's super interesting because language, I guess, all has similar roots, right? Like there was different languages that spurred off and created new language. So it's finding those similarities and it's able to to learn from that, right?

Klemen (03:05)
Yeah, exactly. And you know, it's kind of hard for us to comprehend this because none of us, I guess, speaks like, you know, more than a dozen languages, let's say. I think that's pretty fair to say. Most of us who speak maybe two, three languages fluently. So imagine that you have a person, a human, that speaks sixty languages with with native actors fluently. So this is basically our AMO. And and so I think what's happening inside it's hard.

hard to say precisely like that the space of languages on the voice and speech side and also on the output space becomes very like kind of continuous. It's not so many discrete, this is English, this is Spanish, this is French. You know, they they all get kind of like in a spectrum, like nice nicely connected in some ways. They a large amount of overlap.

Hermes (03:50)
It it blends it together. So no, that totally makes sense. Even as a like a Greek speaker, you can see certain words when you hear a native American speaker or English speaker, they're not pronounced the right way. Versus some native speakers will pronounce them the right way. And so capturing that in a data set, that's super cool. What what would you say some of the biggest challenges are when you're talking about like this transcription translation and real time?

Klemen (04:15)
So there's a difference between transcription and translation. Yes. Yeah. So maybe we can separate that one out. And there's also like a problem of you know, offline batch and real time. So these are like kind of two two completely different things. We can take them separately, if that makes sense. So maybe let's start first with with the way with the offline and real so streaming, low latency is the real time part. So so this is another another another thing that Sonyox has been, you know, the best, you know, at

from very early, from me when we started basically the company, we we immediately started working on real time speech to text, okay? Which really means that you are streaming the audio into the neural network and you have to with some very low latency, very quickly, very fast, generate some work, parts of the words. And while the audio keeps coming in, you have to be generating words. So this is much different than what, for example, ChatGPTs or or or other LLMs, text LLMs are.

Because what you do with others is like you provide all the context, you ask, hey, how are you? You ask the entire thing and then it generates the entire response. So you don't have this concept of bidirectional streaming. Okay. You provide entire input and the entire output is generated. And then you again provide entire input and output is generated. So for streaming with low latency, it's really much, much more difficult. Because you have this, you know, you have to basically make decisions in low latency setting in real fun. So basically

Speech shoot X AI system is is in some way, I say it sometimes to the team, like a robot. It's basically a robot. It gets in a a stream of raw audio and it needs to make decisions in real time. It doesn't steer the car, but it does produce tokens, it does produce words. And how do you know this robot, this speech shoot X AI system is accurate? It's good if it's producing the right words with low latency. So as soon as possible. So so the streaming problem.

is the real time aspect of this is really it's a lot more challenging how to do it well, especially with these modern architectures with transformers and LLMs like architectures. It is really difficult to do to do this with with accurately well. So we we have to do a bunch of innovations here just to to train transformers to become streaming transformers. So that's that's that's the files, the offline part and the real time. The transcription and translation are really two quite different problems.

So

in our speech to text AI model, speech to text API, we don't offer just transcription, real-time transcription. We also la offer speech translation, real-time speech translation. And we we saw a big opportunity here to improve upon what everything has been done so far. So if you use Google Translate or other kind of real-time speech translation services, you will see they're basically translating with a relatively large delay. So basically, you know, you first see a transcription.

And then it's kind of wait until the end of the sentence and only then the translation occurs. You know, this really breaks the conversation flow because you know, you miss I don't know, ten, fifteen seconds before you get anything translated. So we said, I mean, we can probably do translation much, much, much faster. We have to somehow basically do it mid sentence, think of it like you know, in chunks. As I'm you know, as I'm talking, I can also do the translation of that. With some with some a little bit delay, but I can do it. And maybe it's

one second, two seconds delay. So we train the model in the same way. So the model has been trained to basically do translation as as the person speaks. So you'll get a transcription and you'll get immediately a translation. But not that not wait until the end of that. So a little bit of words and then translated words. A little bit more words and translated words. So and then you can r then it's really interactive, like for the first time. So you can just fluently talk to another person in another language back and forth. And there's there's almost no there's basically no delay on our second

Hermes (08:05)
So it's taking the real time versus we'll we'll leave batch to the side. Just thinking of it in real time. You're making almost like two predictions, right? First is the prediction what's the next token, what's the next word, but then you have to reevaluate as more context comes in because different languages obviously they move around the words, they move around the bits. So it's doing two predictions that as it does the translation?

Klemen (08:28)
Yeah, so i kind of, kind of. So I mean this is exactly what's happening like a psychronous. So because it's a streaming model, it's a real time model, as it gets more context, more input, more audio, like when I'm saying beautiful, right? So as I'm getting more more of as I'm giving more of the signal, more of the input, the model is better at recognizing this word. So I and when I have kind of fully finished speaking the world, you know, the model will predict that word much, much more accurately in in between.

And this holds for active speech. So that's how transcription is done. And then translation is done after the transcription of a few words is completed and it's assessed with high accuracy. The model does this on its own. It's been trained like this. now this transcription is very accurate and it's the right time to do translation, because it's not always the right time to do translation, right? You cannot do translation every three, four, five words. For some languages like German or Korean, you would completely break the translation meaning. Because

the words completely shuffle when you do translation. So basically the model's been trained to to listen and and wait until it actually can create a high quality translation. And so that's built into the model and that's that's you know that's how you get real time transcription with high accuracy as well as really high quality translation.

Hermes (09:44)
I'm just trying to like process it and and get an understanding of like where in this like multilingual conversion and like the the code switching, how how does like AI play this role in in making that happen?

Artem (09:56)
So in terms of code switching for AI, it's in some sense it's much easier for an algorithm to switch between different languages than for human. Because imagine if you have a conference with ten different languages, then you will need at least ten different human interpreters and you will have a lot of different language pairs. And human interpreters will need to translate f via intermediate language.

like English. So one interpreter translates from Finnish to English and second one translates from English to Japanese, for example. But for AI algorithm it knows all different languages at the same time. So it can translate from any language to any language simultaneously. So and it simplifies the overall system a lot.

Hermes (10:45)
It's awesome. Like how it can just know everything, right? So so it's a seamless translation from one to the other. There's no intermediary. Loses a lot of the the context and and the feeling of it. So having that one to one is is huge.

Klemen (10:57)
Mm-hmm.

So tell me that, you know, with with Noor and and your telecom GPT Arabic universe colliding, what what unique challenges arise when when you are creating natural language or natural real time dialogue for Arabic speakers? Particularly, I mean, this would be fascinating for the listeners to understand that this region has some unique challenges but also opportunities because you gotta think about diversity.

code switching and and voice accessibility. So that'll be some interesting notes that if you can share Baron.

Merouane (11:28)
So definitely before talking about the challenges, maybe the reason why we and people are building basically more regional large language models. And I think this is quite important for people to understand because many people come to me and tell me, look, you know, I have a ChatGPT in English and then

As of today, or at least was trained on English data. And what happens is basically it's quite good in translation, which obviously is the case also translates quite good. And so if I want to talk or or have something in Arabic, it's very simple. What I do is take my Arabic text, translate it in English, put it basically on ChatGPT, get the answer or my report in English, and then from the English translate it back to Arabic. And that would work. It effectively works. And the translation, by the way.

today is quite good in terms of level. But if you do that, you did not understand at all what LM are and how they work. Why? If you do that, you will get an answer, but most probably you'll get an answer which is basically an English guy speaking in Arabic. Okay? What you want from an LM is to feel as if you're talking basically to the person who is an Arabic. Because the kind of references he'll give you in the report

will be more about Mark Twain. Mark Twain, there's no connection between Mark Twain and my culture because it's not linked to that. Typically, if you ask for an image generation of an image of a marriage, for example, there's more probability that there is basically a church which appears on the thing, which is also deconnected from Bertal environment. So you want a large language model which conserves that kind of culture and that kind of interaction. And so at the end you need to train it on the data set.

Which corresponds to your culture, and that's quite important. And I think this is quite important under people to understand why you need basically some LMs which are trained on the data set of that region of that culture and not do this translation mechanism to work out. Now to do that, there are some challenges. And one of the biggest challenges is data. Data either basically written or spoken. Spoken being the most complicated one. On the written one,

There is a natural bias in internet, which is basically most of the data in internet is English based. Okay. And also the images. And it's not because it's done like that. It's just historically there's a cost to upload data for people. To upload any kind of text or images on internet, it requires that you have an internet connection. And we know that internet is not a ubiquitous on earth. Many countries in Africa will cost you money to get that. And so we have this challenge of basically getting

datasets which are basically on that language. Spoken on which you guys are working on is even more complicated. And the reason is basically that spoken language is something that basically we don't have that access to quite easily because you need to record it. It's a second step.

And recording is another step. And and a good example is India, by the way. India has the smartest people, as we all know. I mean, all the key CEOs in the world, many are are Indians, things like that. And India has been very late in coming up in the L L market in terms of building its models. And one of the reasons is exactly the digitalization. You need basically to have a lot of data which is already digitalized in the country to start working. And and so it requires some kind of effort in these things to gather.

And so one of the, as you said, one of the challenges is acquisition of data. And it's in various form. And of course, India is also an example here also. It's called dialects. Dialects makes makes your head even worse because the amount of dialect the the kind of of diversity in dialects is huge. And second, the amount of dialects that we can get in terms of data is quite few. And so this requires some smart techniques on how you build these models in the sense that you're not gonna do foundational models in Arabic.

Or or other languages, you're gonna make sure that you use a foundational models, do the right fine-tuning, work it out, and all these things. Conversation is another step, meaning having the interaction with voice, which is a modality. I'm talking here more about text. But once you go into and we can talk about it into voice, then of course this incurs other kind of challenges because the way the intonation is made, all these things adds up on top of the real time.

Some countries speak slowly. I have a lot of Finnish friends. You can't know how much they speak very slowly. And some cultures they're very speedy in how they speak. And so this also creates a lot of challenges.

Hermes (15:57)
Aside from the variations in in tone and and regionality and things like that, you talked about no one has like support for Arabic or certain type of like Catalan Spanish. Like how what kind of kind of interest are you seeing from the enterprise customers around multilingual support? Really what are the challenges aside from data collection? Is it all data collection or is it other challenges as well? It's

Lily (16:21)
demand is honestly overwhelming, totally overwhelming. It's a good idea. It's why we need it's why we need to grow the team. It's why l but you look at like web traffic to our website and like there's more web traffic and we've done no paid marketing anywhere, let alone in India. And like three times as many people in India are looking at information about how to build voice agents than there are in the United States and Canada and Europe combined. And like the data is a huge part of it and it comes with an understanding of like what

People want to send to the model at the end of the day. Cause like it's one thing to train a Hindi text speech model. It's another thing to train a Hindi text speech model for customer support where like every other word is like literally in English account number, inquiry, billing, right? And so you have to build for those use cases. I mean, data collection is probably the biggest part. I would also say like the linguistic awareness of what it takes to build in some of these languages is non-trivial. Arabic. And by the way, like we're gonna have like Rhyme is gonna have extremely high quality Saudi Arabic.

Text speech models that are running self hosted so that like the demand in Saudi Arabia can be met because by the way, they have really strict data residency requirements. It's like they can't even use a cloud provider. Anyway, I digress. No, everything

Hermes (17:26)
has

to be in in con

Lily (17:27)
But the point being like we're gonna train out the highest, like a usable, not only usable, like amazing Saudi Arabic text speech model before we even have language models that can generate text in Arabic that sounds like it's someone from Saudi Arabia. It's wild. Sad but true.

Hermes (17:40)
And when you talk about that Hindi model and like the Indian call centers, you brought up a very interesting topic around like like swapping between languages, which in certain cultures and certain languages, that's just how people speak, right? And that's a certain nuance to even their own voice that varies from like family to family. So yeah, they could be speaking Hindi and then it's English account numbers, it's English words, it's like that weird mix that

It can't just be one language model. The one language model wouldn't be able to sustain that kind of weird switching because it would have to it it it would always have to be bilingual at least.

Lily (18:17)
And that's like the design decisions that we often have to make is like if you train one model for every language, like sad but true, like not a very good model for for these use cases. And again, there are a lot of reasons that have to have controllability, right? Like you don't want it to randomly start sounding Spanish when you don't want it to, all this stuff. So like the dec design decisions that we have to often make are like, okay, we we know that shipping monolingual models results in higher quality, but like what are the like languages that we can package together to make something work, right? Like how can we have a English model? How can we have a

Spanglish model, right? And like of course over time, I'm not saying like the end to end approach isn't gonna bear out with a lot of fruits, but like today you just have to be very strategic about it.

Hermes (18:56)
So it's about making that consciousness.

Lily (18:58)
'Cause you don't necessarily need a you don't necessarily need a Hindi model that like can also speak Mandarin Chinese.

Hermes (19:02)
It's not there yet.

Lily (19:03)
But you definitely need a Hindi model that can say customer support in English, like a hundred percent of the time.

Hermes (19:08)
One hundred percent.

you walk us through a little bit about what makes the the video translation more advanced than like your typical dubbing or like lip sync style technology?

Lily (19:19)
Yeah, yeah. So there are several things that we do in the video translation. First is we clone the voice and we translate the language and we make sure it fits a video lens and with support 150 plus different language. And the second is we do the whole face reanimation rather than just the mouse region. So we do a lot of reanimations of the multi-pod face and so on.

And also we have a very well designed workflow that allow people to easily get input a video and get everything clear and then output the results and so on. So I also have lots of freedom to allow the users to post adding their results and to make sure the translation or the things are accurate and the kind of stuff. And what's more, we are the only one that's able to put the whole pipeline to live real time.

Which means real time interactions with the translations and then that happens in all the different places, including I think even webinar and meetings in the world in different places. So definitely that's a lot.

Hermes (20:27)
It's huge for like those massive global town hall. Now you can have your senior leadership speaking in their native languages, but having it come to the audience in their own native language with all the proper reanimation of the face to make it feel realistic and not just like it's a cheap dub.

Lily (20:46)
Yeah. Yeah, that's right.

Hermes (20:47)
I think our producers for this show are gonna want to dive into that to see how we can bring this even our show just

to different audiences in their own languages, right? Because you can reanimate and people could be listening and watching along in their own native language without having to listen to the original in English, but still get a first class experience,

right?

the ability to speak to the AI i in your most natural form is is exploding. And then healthcare, I can totally see that. There's so many instances where there needs to be a transcription, a record. How did it happen? What happened? What are the notes? It's huge if that helps there because there's so much time lost to just

Klemen (21:26)
Yeah, it really boosts the productivity of the doctors, of the staff, medical staff, and speech is at the very core of this you know, automization, optimization process.

Hermes (21:36)
So when we're talking about because your models are multilingual, right, how does this real time translation change the experience for multilingual teams or audiences that are not native to the speaker's language?

Klemen (21:50)
Yeah, I mean with our real time translation, we really broke this language barrier. So now you can have a a conference, okay, where you have many where you have basically attendees from many different languages that might not be, you know, so fluent, let's say, in the language that the speaker speaks. It could be even English, you know. People from all kinds of works, you know, they are not necessarily fluent or really comfortable precisely understanding the the speaker, even if it's in a popular language like like England.

And so we see now that some conferences are you know are now being translated into many different other languages with Sonyox. And this is live, completely live. As the pr you know, as a person speaks, it's giving a presentation, it could be event, conference, whatever. There would be TVs or other other kind of projections, you know, to to to to do multiple translations in parallel. So that that's that's something that we've seen. Another thing that we've seen is we we have a we've built a mobile app.

Sonycs and people often just use it and there they might be at some event or with a family, right? Multiling like another, yeah, maybe I should mention this. Multilinguality, family with multilingual members is a is another another another big use case. I didn't know about this. I guess I wasn't exposed to it. How how prevalent it is, but it's actually huge. So people might get married, you know, the partners might get married with with two different families with two different languages.

And there's no communication between them because they they don't necessarily speak this the the the the common language. And now we've seen so many examples where they use the Sonyx app, the mobile app, and they just converse because real time translation, bidirectional real time translation is right there from one language to another, etc. And so this this is like really breaking the the the language barrier experience. Like it's dear to me because my sist my sister has married into a French family and she's being she's using it and she's just had out something I don't know, great expensive.

Hermes (23:41)
No, that's huge. we'll have to use that next time I I take my family to visit in in Greece.

Klemen (23:46)
Yeah, you

should.

Hermes (23:47)
Like in terms of live translations, the voice is coming through, how do you keep the the data private? What do you guys do?

Artem (23:54)
privacy is a a very important question nowadays and it's getting more and more important every day. And that's why we do like standard things like best kind of encryption and so on and so forth. But what's more important, we process everything in memory so we don't store anything on the servers. So once it's translated, it's deleted. So we don't have any problems with storing data and or other privacy concerns. Nice.

Hermes (24:23)
That's huge. Right. And on the enterprise side. So w when when you're moving to to talking to bigger clients and they have this concerns, we'll say, around ownership and data sovereignty, like how d how does that exactly work with you guys and how do you

Artem (24:39)
Yeah, some some big clients they want everything working inside the security contour and so that they have zero probability that anything will go outside. And for such cases we have different kind of deployments in private clouds or even on premise on the servers inside the security contour. And in terms of API that we support, it looks the same, but the only question is on which servers

works our solution. And in most cases our clients can try and test our solution in our cloud first, and when they're happy with the latency, with the quality of translation, they can deploy it on the server.

Hermes (25:21)
That's awesome. So you give a place where they can take that baby step, right? Get started, get working with the product, and then once they're ready, if they have that need, move into like a self hosted.

Artem (25:31)
Yeah, of course because you don't want to build a data center and to install a computer just to test the technology. You want to test it with a couple of lines of code and then if you are happy you can then think about all the deployment stuff

and so on.

Hermes (25:46)
Makes total sense. In terms of once you get this set up, what kind of customization o options do you guys offer? And how do you tailor translation outputs for specific industries or use cases?

Ivan (25:57)
Well, say we work with Agora on a case. we would want your business to grow as fast as possible so we would show total flexibility across our product product stack. So we'll try to adapt to your product needs, the to to your customers' needs. I know that's like a B2B to C product there, so it's kind of a challenging thing to do, but we aim to do this business wise. As we do have control over the pipeline, we can use machine learning for the machine.

To better understand the industry needs and for it to be flexible. We could like learn it a little bit more on some cases, on some topics, we can do vocabulary uploads, we can adapt API methods, switch between technologies for for customers to be happy, for the business to grow. We have our customers' business in mind here and that's the main case. We want our customers to expand to new areas where they can speak to anyone, and that's that's our mission here.

Hermes (26:54)
It's awesome to hear having that customer focused approach is is huge, right?

Ivan (26:57)
That's the only way company can survive now actually, so

Hermes (27:00)
That's the Yeah, you have to you have to focus fullheartedly on the customer very very much. Speaking of that, what tools or metrics do you provide to your customers to be able to like evaluate the translation quality within production?

Artem (27:13)
Yeah, so how we see it is that we test and benchmark everything inside and so that we provide the best quality that we can. But that said, our customers can use our API because it's freely available for benchmarking, and they can benchmark on the internal benchmarks as they they like. And we also have a couple of no code solutions that you can download, for example, and benchmark and try and test.

them yourself w without even without the need of writing any code.

Hermes (27:47)
No code is huge. And nowadays that's like one of the main drivers, the ability to just get in there without much setup and put something together that works. In terms of overall advancements in the field, like what do you see as being like a couple of the key pieces that need to be made? Like the key advancements that need to be made over the next I think five years is probably too much. So over the next two years. So let's let's keep it quick because the the windows are fast and the AI is

Ivan (28:12)
Yeah, yeah. If we are speaking about five years, I guess we'll be seeing really great advancements. All the translations will be seamlessly integrated into all the phone calls, all all the communication on the OS level. When you start a the stream like we have now, it will easily and seamlessly be translated into any language. So that's how the barriers would be broken. We are moving humanely like past past this.

biblical issue where everyone will be able to speak to anyone. But the speech prediction will be on almost magical rate. I think in two years already, in five years it'll be crazy. But I think we'll we'll see like seamless translation of all the content you have on your phone, like in two years already. In five years it will be AI created, then AI translated.

Hermes (28:55)
That's gonna be the day, right? And in terms of like this AI to AI world, what role do you see, you know, unsupervised learning really playing into this future development of the speech models?

Artem (29:07)
Yeah, so I think that unsupervised learning already plays a huge maybe the biggest role right now because all the large models like GPT like models are pre trained with unsupervised learning and it's true for text models, it's true for speech models. So it's I'd say it's the main technique right now to train large models. There are other techniques that are applied on top of unsupervised pre training.

But it's unsupervised pre-training gives a huge, huge gain in performance and in accuracy of algorithms. And probably it will be even it will play even bigger role in the future because the amount of data is growing exponentially and you can't label this data. So you have to have techniques that could train algorithms without data label

Hermes (29:59)
Yeah.

Artem (30:00)
yeah.

But what's very important is data pipelines because you can't just take any data and upload it to the AI. You need to pre process it and you have to clean it. You need to clean it without human laborers because they are time limited, but you need to train Eugrips and build pipelines so that you have a clean data for unsupervised pre training and it's a very big part of current AI production systems.