Hermes Frangoudis (00:07)
Welcome everyone to the Convo AI World podcast where we the teams building the technology powering today's industry leading conversational AI technologies. Today have product lead from Google focused on AI content safety. Thanks for joining us today.

Ninny Wan (00:23)
Thanks for having me. I'm excited to be here.

Hermes Frangoudis (00:26)
let's right into it. Can you share a little bit around the origin story of the AI moderation efforts and Google's initiatives and like how they kind of evolved?

Ninny Wan (00:35)
Yeah, content moderation is not a new thing to Google. I'm sure that's not a surprise to anybody. My team, actually, when I first started at Google, was really focused on the non-GenAI stuff. And so anything that was what we consider user-generated content, so like YouTube comments or things like that, where you don't want bad things like hate speech or particularly violent content or things like that. That's what kind of my team took care of. And then when the GenAI boom happened and a lot of our products started pivoting to be AI enabled,

it was a very natural and organic growth to cover that space as well. I will say that, know, GenAI obviously offers a whole host of different and new challenges for us, but our general ethos is the same, right? We want to make sure that if users are going to use our products, they are able to have like a safe, productive experience and be able to have some protections against the things that they probably don't want to see on a regular basis.

Hermes Frangoudis (01:26)
Makes total sense. And you mentioned like the evolution kind of over time and now with large language models and this sort of like AI boom, how has that really changed the, maybe not so much the focus, but what you guys do and the mission?

Ninny Wan (01:42)
Yeah, think the mission is still the same. What I see is that we're actually very focused now on being more nuanced on how we enforce safety. So back again, when I first joined, we had a generalized user-generated content group, and we focused on all the modalities and any product that Google could use us. Now that we're in the GenAI space, I noticed that what we're doing a lot is we're trying to figure out, hey, what specific things are special? So for example, I have a whole team that I work with

that is the on-device team. That's one of my products and we build specific GenAI safety for on-device. Separately, we have one wholly for server side. We also divide between our 1P clients which are you Google products and our 3P partners. And so that type of nuanced approach I think has really served us in the sense that we are able to better meet our customers where they are, for all the different types of customers we serve, which has actually been a really cool thing to see. It's interesting to see that type of

delineation because now we're able to better service our customers and better tailor our strategies for those customers. It's a little better than the one-size-fits-all. We're really trying to meet our customers where they are.

Hermes Frangoudis (02:46)
That's really cool to see a company like Google that has such a great amount of teams, right? Each one is tailored and focused on their own individual areas and really can probably shine, shine bright. Right. That's awesome. So speaking of shining bright, like what were some of the maybe initial challenges you faced and like, how did you overcome them with the work you're doing in moderation?

Ninny Wan (02:55)
Completely. Yeah. Yeah.

Yeah, I mean, when we first started, GenAI was the Wild West, right? There wasn't really a lot. We didn't know what the regulators were expecting. We didn't know even the definitions of abuse types. You'd think like, the definition of hate speech is the same across any type of abuse, right? GenAI or not. And that's generally true, but there are nuances, right? Because the fundamental process of content moderation is different. If you're looking at user-generated content, you're really trying to make sure that, you know,

Meany, for example, isn't posting really terrible stuff and then subjecting everyone on the internet to it. When you think about GenAI, you're really trying to make sure that the products that Google is producing isn't suggesting, generating, echoing bad content that users might not want to see. And so it's a little bit different. The definitions of abuse type have come a long way. I think we really have to thank our close partners in the global policy and standards team for that one. They've worked tirelessly to make sure that we have policies that actually make sense.

That are enforceable, that are specific enough, that can cover all the use cases that our products need. And that has been pretty great to see. So I think that that's the first one, right? It's just the definition of what we consider abuse. The second thing that I've... Yeah, just setting some boundaries, right? Some things, you know, honestly haven't changed that much. Like hate speech is a good example of that. Hate speech is pretty consistent. But there are some things that we see just manifest differently. For example, like sexually explicit content

Hermes Frangoudis (04:14)
Like setting some boundaries, right?

Ninny Wan (04:29)
is different when a bot is, like when a GenAI feature is making it versus when someone is trying to upload content that is potentially like themselves or someone that they know. So I think that's the first thing. I think the second thing that we see is really just around data. Google, I think really prides itself in making sure that we're aligning with all the privacy and compliance requirements that we have. And we are very sensitive about user data. Like we really want to make sure that we're not using user data unless we absolutely have to.

That we're making sure that we are completely in alignment with desires of whether or not their data is used at all. And this is very pertinent, particularly on device. On device is a very customized, personalized space for a lot of our end users. And we want to make sure that they know, hey, you can use a Pixel device, and we will completely abide by privacy requirements. We're not going to see anything that you don't want us to see. And so,

Hermes Frangoudis (05:18)
That's huge.

Ninny Wan (05:19)
a huge part of what my team has done, because I remember before this you were asking, what else is included in content moderation? A lot of what my team has done is actually make synthetic data so that we don't ever have to use user data. And that's come a really long way. Back in the day, it was very manual. It was just teams of people trying to make up seeds. And then your pipeline might make a few at a time, not as much as you would want.

And now we're able to scale in a matter of hours, which is a huge step forward across all modalities. So we can do synthetic data for text or image, which is really great.

Hermes Frangoudis (05:50)
Well, that's huge. Cause like when it comes to training, there's only so much data on the internet at this point. Right? And that's like a big, big issue. So how do you, how do you kind of adapt and prepare for all that? Super cool.

Ninny Wan (06:01)
Totally, yep.

And we also hear, you know, like a lot of the data that's on the internet, even if you were to use it, is sometimes an echo chamber. So in order to be able to have good coverage for all the abuse types that you wanna have coverage for, you probably want to do synthetic data, at least a little bit, to make sure that you're getting those edge cases. Because, you know, just because they're edge cases doesn't mean that they're less egregious in real life, right? Yeah, and so, yeah, we really benefit in that way.

Hermes Frangoudis (06:23)
If anything, they're just even harder to find, right?

That totally makes sense. It sounds like, speaking of edge cases, maybe can you talk a little bit about maybe some lessons learned from deploying some semantic moderation across like all these diverse product ecosystems?

Ninny Wan (06:42)
Yeah, think my org specifically is working on generalized classifiers, right? So we fully expect that we will provide kind of a baseline for all of our teams to work off of, and then they can tune and tweak them as they want. I think what we've learned is people like our clients are more similar than we originally were fearful that they would be. We were worried about the amount of time our future clients would have to put in to make sure that the solutions would work for them.

And what we found out is because, again, thanks to our policy team, our policy is pretty consistent. We're able to get a lot of efficiencies that way because our clients are more likely to use pretty much the same classifiers than to make customized things. There are, of course, going to be edge cases. There are special use cases within Google that are going to be special, right? So like the Play Store and Search, for example, have customized policies that they publish on their respective websites.

And so they have to tweak a little bit more and they have to make some custom stuff. But generally it's been a really good efficiency story, I think, in terms of being able to offer these generalized classifiers. I think the other thing that we've learned is that there will always be room for improvement. Like abuse is just a really fast moving space. We knew that even from the UGC days, the user-generated content days, but with GenAI we see that it's so frequently used that the abuse vectors change almost daily.

And so a large part of our jobs is to make sure that we can catch, like keep up with that and catch all the new things that might be coming through.

Hermes Frangoudis (08:07)
Wow. Staying on top of that must be its own, like, full-time job.

Ninny Wan (08:12)
It's its own full-time

fun thing that we do. Yeah.

Hermes Frangoudis (08:16)
So kind of digging down into the technologies behind this, can you talk about some of the technical components that enable this semantic understanding in terms of content moderation?

Ninny Wan (08:27)
Google has always been at the forefront of thinking about machine learning. I think the advent of transformer-based models has been huge for us really because there's this concept of like self-attention, right? With self-attention, you can get better contextual relationships between seemingly disparate types of data, which allows us to really understand like the context of it. That's the thing that I personally have been the most intrigued by in the field that I work in right now.

Which is just that when you think about the types of "badness," quote unquote, that's on the internet. A lot of it is really nuanced, right? Given different contexts, something that can seem very benign, like, "Cool, I'll see you there," could be really bad, depending on what the context of that statement is made in. And so I think that this type of understanding of the contextual background of what content is coming from has been really helpful for us.

And like transformer-based models also have a whole host of other benefits, right? So you get a lot more resource efficiency from using parallelization. We can also have high scalability that way, which allows us to scale a lot faster and scale cheaper than we have been able to in the past. We're also better able, again, through self-attention, to be able to capture these kind of like long-term dependencies between concepts and abuse types, which is also very cool.

That means that any model that you make is never really fully thrown away, right? Like the model learns and understands and grows and evolves as you go, which is very helpful. And it's kind of related to the concept of like transfer learning, which is a big thing for us. The better that we can have our heads understand each other, the better it is for our future models so that we're not like throwing away and starting over every version we have, right? We can build on what we've built before.

Hermes Frangoudis (10:13)
That's huge, being able to set that foundation and kind of grow on top of it. So in terms of growing that foundation, English isn't the only language in the world, right? So how do the moderation systems handle more of the multilingual content, like some of those cultural nuances?

Ninny Wan (10:28)
We train, just like for English. We try to always be very fair in the way that we train across all languages. That's another big Google tenant is to make sure that we're building for all of our users, which means that at minimum, not on average, at minimum, my models need to be able to cover at least the largest 36 languages in the world. Most of my models cover over 140 languages.

And that's kind of the baseline, right? We wanna make sure that regardless of what language you're speaking, we're able to provide coverage for you and that that coverage is not going to be different just because it's not English.

Hermes Frangoudis (11:02)
That's huge, you know, helping give everyone that same priority, right? Because it is really a globalized world and the internet's connected so much of that together that it's...

Ninny Wan (11:10)
Completely, yeah.

And this is also just another testament, honestly, to our amazing teams who have built synthetic data pipelines. Because without synthetic data pipelines, where we have kind of a natural language understanding, down to even the proper slang that they're going to use in Brazil, for example, versus the United States. And thanks to our teams of being able to do that, we get very good synthetic data now in multiple languages, which is really great.

Hermes Frangoudis (11:26)
Okay.

That's epic. That's huge. So in terms of self-evolving systems, these things are always building on top of each other. How do you ensure that these systems can kind of adapt to more of the emerging harmful content trends and threats?

Ninny Wan (11:51)
Yeah, this kind of goes back to what we were talking about earlier, right? We realize that abuse is just evolving so much faster. And to be fair, in safety, abuse has always been evolving very quickly. So it's always kind of a cat and mouse game. You're always trying to catch up with the bad guys, because the bad guys, remember, this is their full-time job. And so it's one of those things where I think we've invested a lot, particularly over the last two and a half years, I would say.

In making sure that we have very strong, what we call continuous learning pipelines, that allows us to make sure that model velocity, so like the speed at which you can build your next model, is decreased version over version. So like maybe your first version of the model takes you, I don't know, make up a number. The next version is going to be that number minus... at least halved, right? So your goal is to really make sure that you're being faster in your model velocity.

Hermes Frangoudis (12:36)
Wow.

Ninny Wan (12:40)
I also think that the continuous learning pipelines have helped because it allows us to identify what emerging trends we're seeing in abuse and really target those types of abuse. So for example, if something new comes up tomorrow, let's say like a new version of Tide Pod Challenge happens and that's kind of a dangerous thing that we want to make sure that we're looking at, then we can understand what that is, we can detect it accurately and then make sure that we are flagging appropriately. Because that's the other thing too.

Hermes Frangoudis (12:56)
⁓

Keep it from spreading.

Ninny Wan (13:06)
Yeah, exactly, because that's the other thing too, is like a lot of content moderation is just to make sure that our products know what's happening on their surfaces. And so really it's about providing a verdict, an accurate verdict, and then allowing the product teams to decide what they want to do with that verdict.

Hermes Frangoudis (13:21)
That's huge. Yeah. Put it right back into like the product teams' hands to kind of tailor it. Right. And speaking of that, are you able to talk about which products or services have integrated some of these systems?

Ninny Wan (13:26)
Yeah. Yeah.

I can't go into specifics, but I will say that we have very good coverage across Google right now. I will say that there is a larger process at Google. I don't think this surprises anyone, where any of the products that launch, particularly if they're GenAI, are required to go through a safety evaluation. And so every product that goes out is undergoing the safety evaluation and is required to hit a threshold of safety before they're even allowed to launch. And this is very consistent in the sense that there is a singular review process.

So everybody goes through the same process. ⁓ And the thresholds are set and reviewed upon and it allows a lot more interaction too between the client products, I guess, as well as the review board. Just to explain like, hey, this is what our product does, this is kind of some risks that we identify, is there anything else that we're missing? Because I think that the board that reviews all of our launches

Hermes Frangoudis (14:05)
Nice.

Ninny Wan (14:27)
is just so knowledgeable about what the gotchas are that it's a huge benefit. It's like a huge step forward to have them because they can point out stuff that you might not think about, vectors of attack that you might not have even thought of. And because they're looking at so many launches, they also have the benefit of saying like, hey, there's this emerging abuse vector. Have you thought about that? And so that really helps make all of our products a lot better.

Hermes Frangoudis (14:50)
That's huge. It's like having the experts right there to kind of battle harden it before it even sees the light of day, like public light of day, right?

Ninny Wan (14:54)
Yeah.

Yes, exactly.

Hermes Frangoudis (15:00)
Speaking around like the products and, how it affects people, can you talk about some examples maybe where AI moderation has improved this user experience?

Ninny Wan (15:12)
I think that when we think about the safety of any feature, it's really about a balance, right? We want users to have the freedom to do whatever creative thing that they want to do. But we also want to make sure that if you are a more sensitive audience member, so maybe someone who's under the age of 18 or considered a minor in your country, or if you're just someone who wants to make sure that you're not going to see bad stuff, that Google is not making you look at bad stuff, right?

And so I think it's a balancing game, but I will say one of the coolest things that I've seen happen, and we can talk about this a little bit more too, is one of our newest features is sensitive content warnings. This is an on-device feature that we announced in October of 2024. It's now in production and getting pretty good feedback. And it's basically like reinventing what I would consider like a core use case of Google, right? I think when you think about Google, you're thinking about your Google Messages app.

And of course, when you're thinking about Google messages, you're thinking about, you know, if someone sends me pictures and they're inappropriate pictures, so in this case, nudity-related pictures, maybe I don't want to see them. And so the sensitive content warnings were really exciting to us because it's not really a blocking, you know, like a lot of people think of content moderation as just like a hard no, like they're just not going to show you whatever that content is. And this was kind of a new approach. This is a speed bump, so it basically blurs any nudity-related images.

And then the user gets to decide whether or not they want to unblur the image and see it, or if they want to leave that blur on because it's something that they don't want to see. And this is a very cool feature, in my opinion, because not only is it a speed bump, we call it a speed bump, but it's also a cool feature because we're able to differentiate the user experience down to the end user level. So for right now, sensitive content warnings is default opt-in for anyone under the age of 18.

And then default opt-out for any adults. So anyone who wants to use it can, but our most sensitive users will be protected.

Hermes Frangoudis (17:05)
That's huge. It's great to hear. Because that's like peace of mind, right? At the end of the day, you're given that peace of mind where people don't have to worry about inappropriate stuff.

Ninny Wan (17:10)
Exactly.

Exactly, yeah. And it's really cool too because it's on device, which is one of my teams. And I'm just very proud of how we were able to do it in a privacy-compliant way. So the entire classifier, the entire model is on your device. Nothing ever leaves your device. So we never see the pictures. They're not being pinged to a server somewhere. It's perfectly privacy-compliant, which I think is very cool. Yep, we never see anything.

Hermes Frangoudis (17:36)
Just goes through the algorithm on device, right? Huge.

Speaking of never seeing anything, let's take the reverse end of it. How does your team collaborate with maybe like human moderators and handling like those sort of edge cases?

Ninny Wan (17:53)
human-in-the-loop is the concept of even if you're going to be building AI models, there needs to be a human-in-the-loop to be able to either confirm, label, give a ground truth, basically a golden set of truth. And so our human-in-the-loop process is a very critical one to our machine learning and model development process.

We are heavily dependent and we very much celebrate our folks that are on the front lines looking at this content for us and making sure that we have good ground truth. And I don't see that going away. I know that there's been a lot of discussions about, well, one day there will be no humans. But ultimately, like, human-in-the-loop is a pretty fundamental part of machine learning. And so we expect that that will continue. We do think that there can be efficiencies there, right? So it could be instead of

looking at X number, you're looking at X minus one, X minus 50 samples to get a good sense of what the ground truth will be, which is honestly good for everybody, right? That's just a more efficient way to work. And so I foresee that continuing. I think that our reviewers are amazing and they're highly specialized. So we're very lucky for that. They usually are very strong subject matter experts in the space that they are labeling. And we differentiate them based on the type of content that they're looking at.

So that's also very helpful because then they can really focus on the things that they're looking at and not have to worry about spreading themselves too thin across like multiple abuse types and multiple modalities of content. And that's been huge for us. We think of them as kind of our engine of the machine learning space.

Hermes Frangoudis (19:22)
They're the ones really powering that ground truth, right?

Ninny Wan (19:25)
Yeah, definitely. Yeah. And also, we have a lot of learnings from them too, right? So going back to how awesome our policy team is, oftentimes when we see things that are hard to label, that's usually where we get a of policy feedback too, because our labelers, our human-in-the-loop can say, hey, we were looking at these samples and these samples don't really like,

exactly match to the language that we have, or like, hey, we see these things that are maybe an emerging vector of abuse, would we want to add this to our policy? And that's been really helpful, because that means that we're having this live time feedback loop, even for policy... ⁓ Exactly, yes, which is super cool, because that's ultimately what all of our models are built on, right? That they're aligning to policy. And so the faster we can get the policy updated, the faster we can get our models updated, and the faster we can respond to

Hermes Frangoudis (19:46)
They don't fit. Yeah.

So it's like a living policy almost.

Ninny Wan (20:14)
real time emerging events.

Hermes Frangoudis (20:17)
Huge. The real time aspect. Because the space just moves so fast. So the fact that you can constantly adapt with it, is also very, goes back into that whole piece of mind at the end of the day. ⁓ So thinking a little bit more broadly around the landscape, what really sets Google's semantic understanding capabilities apart from maybe other

Ninny Wan (20:21)
Yes.

Yeah, yeah, yeah.

Hermes Frangoudis (20:41)
other products or solutions that are out on the market.

Ninny Wan (20:45)
I mean, Google has always been a pioneer in this space, and we continue to invest heavily in the research that we do in this space. I don't see that stopping. We are starting to push the boundaries of what is feasible, both in an efficiency perspective, as well as a responsible AI perspective. I've been really proud to work at Google over the last couple of years, because I think we really are pioneers in establishing what responsible AI means and what those principles should be.

I think that really is what sets us apart is so much of what we do is kind of forward-looking, that we are usually being asked to define, you know, what is responsibility? What is safety? What should a user expect? And the things that we have defined for ourselves and the policies that we have written and the abuse types that we're looking for, I think really have helped set the stage. Not to say that Google is everything, right? I know there's many other players in this space and we're all contributing kind of

as a community to make sure that this AI boom is something that is as safely implemented as possible. But I will say that I think that sets us apart. I think that when we think about our technology, that's obviously there, right? We as Google invest heavily into our research capabilities to make sure we're always on the cutting edge. I think that from a business process, business development perspective, we are also on the cutting edge. And I think that's really what sets us apart.

Hermes Frangoudis (22:04)
You guys have always been pioneers in this space. It makes sense. You're so far ahead, you're really pushing the boundaries forward.

Ninny Wan (22:12)
Since that first white paper, right?

Hermes Frangoudis (22:15)
So we talked a little bit about how the team monitors and adapts to these new emerging threats and you have the board you mentioned and that's like I'm gonna assume a cross-functional team right and that's who's focused on coming through with these policies and battle hardening everything. Is that also kind of like a living thing so as depending on

who's coming forward, like is that board also, it's not just like the same five people, right? It's like different people for different.

Ninny Wan (22:41)
Right. Yes.

Yes. The review process has actually gotten a lot more robust as we formalize it. Initially, when the GenAI products first started rolling out, there weren't that many of them. So you could kind of white glove each one through the process. And now as we're seeing more features, we want to make things more consistent. That whole process is formalized a lot more now than it was two, one year ago. And that has been

paying off in huge dividends, right? Now we're able to scale this type of, they're honestly very in depth review and a comprehensive review for our products across so many products. I actually don't know if we thought at the time that we'd be able to scale this fast. But we have been able to scale and it's been so impressive to see because it really is, like you said, like this cross-functional, like communal thing, right? It's not a singular team that owns everything.

It's kind of a bunch of teams sending their best people into this like very large process, yeah, where every step you're gonna get the subject matter experts for that step. So you're gonna get policy experts to weigh in, you're going to get human reviewers to weigh in, you're going to have a red team team that's just gonna be specifically helping your product go through this process. And on and on throughout the entire process to make sure that we are hardening every surface that we can, making sure that we're testing all the potential vectors and that.

We're trying to look, honestly, for any gotchas that we might have missed before we launched. And so it's been kind of heartwarming to see, honestly. I know that sounds corny, but it's been kind of heartwarming to see because it's been a huge effort that's across Google. And everyone that I've met that's in that process currently or has been in that process is just so dedicated, right? Like that is their main goal is to make sure that we're making safe products for our users. And I think that's just very admirable.

Hermes Frangoudis (24:29)
I think that it's very important that people are passionate about this sort of thing because that's really what brings out the best side of the product and the safety side of it. You feel safe when someone actually cares. You feel less safe when they don't. That's awesome.

Ninny Wan (24:33)
Yes.

Yes. Yes. And we have no shortage of passion. Let me tell you that. Yeah, We have

We have a lot of passion for what we do. I also am honestly, I feel very lucky to be in the space that I am just because, time and space could have brought me somewhere else. But I think that this is a very cool space to be in right now because things are changing so quickly. And I also think that

rare, honestly, to be able to work with so many people that are so equally passionate about the same thing that you are. So it's very cool place to be.

Hermes Frangoudis (25:14)
That's awesome. And it sounds like the scale of Google is really fed into that. You have so many diverse groups and people that they're actually able to do that, break apart, and bring those people together. That's really cool to Shifting a little bit more towards the product side of it. Is AI moderation technology available for external developers?

Can they use it through APIs and stuff like that to bring it to their apps?

Ninny Wan (25:39)
Yeah, so we are starting to work in that space. For many of our 3P partners, there's private previews and things like that that are happening. We expect that we'll probably continue to grow in that space over time. We know that there's a large appetite for that, but we want to make sure that we're adjusting our expectations of how we set up our safety, particularly by what our 3P partners expect.

Because one of the things we've learned, going back to the learnings, is that we as Google said, XYZ is abusive, and this is how we define how that is abusive. And then we take them to our partners and they're like, nope, that's not what I agreed to be abusive. I think it's one, two, three. Why would you think it's XYZ? That's crazy. And so there's lots of discussions about what exactly is abusive, what the definitions for that is, while trying to maintain some sense of cohesiveness. We don't want to make completely random things that are not in alignment with what Google wants.

And so this has been a long process. Do have existing 3P, or I guess externally available is what I should say, options. So like Shield Gemma is a big external option that I know a lot of folks play with. We also are experimenting on on-device as well with select partners to understand what that space could look like for them. We will continue to do that. have very exciting times ahead, I am sure.

And so I'm looking forward to seeing what's going to happen in the near future.

Hermes Frangoudis (27:02)
It sounds like you're working on something that's going to be more customizable and fit like all the different platforms and content types. And it's really cool to hear that you're taking a more moderated approach. No pun intended, right? Like working with these three partners to really understand the space and the nuance of it.

Ninny Wan (27:14)
Mm-hmm, yeah. Yeah, yeah.

Yeah. And all of this also is just like on top of what we already natively offer through Cloud right? So like Cloud is obviously the first one that would offer 3P availability of safety. And they have also continued to be the pioneers of championing what 3P partners want, how they want it, how we should present that to them, how it would make their lives easier, what types of, even down to like user experience, right? Like what type of toggles should we share and how would they like to see those toggles? And so,

That has been a really cool thing to see as well. Cloud is another team that I work with and it is, yeah, they have really pushed the boundaries of could be available for our 3Ps and it's been awesome. Like I have seen so many good stories come out of that. The Cloud clients love the Cloud team because they're an amazing team to work with and they are also very passionate about the 3P experience. So again, that passion thing, right? ⁓

Hermes Frangoudis (28:11)
Huge.

Ninny Wan (28:12)
And so it's been really cool to see on that side as well. But yes, we see other 3P opportunities too.

Hermes Frangoudis (28:18)
So when you're working with these 3P partners, I guess there's got to be feedback and that goes back into improving the models. And can you talk a little bit about maybe how some of that user feedback or product insights are going into like that evolution? Is it going back to the policies and the boards or is it even like broader than that.

Ninny Wan (28:39)
Yeah, for 3Ps, actually, we don't always directly put their data back into the models. It really is dependent on what our partner wants us to use or doesn't want us to use. If our partner does not want us to use data, we will not use their data. If they offer data in terms of feedback to say, like, hey, here are some false positives or false negatives that I saw, then we're happy to use them if they agree for us to be able to use them. Typically, though, what we're going to see is that we'll get feedback through, like, let's just use the Cloud example. We'll get feedback through the Cloud account managers.

Because they work very closely with all their clients. And it can be very specific, right? Like, hey, this specific piece of content is a false positive or a false negative. Can you look at it and confirm that it's a false positive or false negative? At which point, we'll bump it up against policy. Our policy leads will weigh in and say whether or not we actually think it's false positive or false negative, and we'll go from there. Other times, the feedback is very generalized. It can be...

anything from, hey this interface that I use to set up my policy configuration or my safety configuration is confusing. I need you to update the UX. Or, hey, I think that this thing generally, like this abuse type is generally wrong. Like, I would expect this to happen, but it's not happening. At which point, you know, anything that's related to UX, obviously we would punt to the right people in that space. We would punt to our UXers. But if it's anything safety related,

I think our first step is really just to confirm with our policy whether or not it is a valid piece that we would want to integrate into our model or if it's something that the policy team should take on to improve. So the Cloud account managers again can broker those types of conversations between us and our clients. And ultimately we can make policy changes, we can make model changes or pretty frequently we can also help our clients with configurability changes. Cause oftentimes what they want to do is feasible.

It's often just like a matter of trying to figure out what the right configuration for their settings is to make sure they can get that thing. So all of those things are options. We use all of them. I would say that for me, the most exciting thing is when we get to update our model, because I'm biased. ⁓ But yeah, when we do update our model, we work really closely with our partners to identify exactly what they want us to use. If anything, sometimes they'll say, hey, we actually don't have any specific data for you, but this is generally kind of what we were thinking about.

Hermes Frangoudis (30:33)
Thank you.

Ninny Wan (30:46)
And again, we would rely on our synthetic data pipelines to be able to make that data.

Hermes Frangoudis (30:52)
Makes sense. You bring it back internally, figure out how to like create enough of that content to really train on and, and make sure that you're hitting those marks. In terms of safety and hitting those marks, what role do you see AI playing in fostering healthier online communities? And like, how can that moderation play into making it just a more friendly environment?

Ninny Wan (31:01)
Yep, completely.

I mean, you really hit on the bread and butter of what we're doing here. Because I do think there's a balance, right? I think people want to have freedom to say things, post things, make things generative AI. But I also think that there's a boundary. it's a very hard boundary to pin down. There's multiple articles, there's multiple books, there's whole seminars talking about this. But it's really just about what

one person thinks is incredibly egregious, very violative, so abusive, might not be the same as what someone else thinks. And so that's kind of the struggle with moderation in general, is you want people to generally feel like we are catching the things that they would expect without over flagging on things and making it so unusable that it's never gonna answer anything and therefore this feature is not actually helpful to your life. And simultaneously you don't want it to be so lax,

that the person feels like, is kind of a toxic environment, I don't really wanna be in this space. And so it's a really fine balance. I think it's an ever-changing line also, like what is the line today or the line that we're trying to hold today might not be the line that is happening tomorrow. And that's largely dependent on things outside of our control, right? So if there's wars, conflicts, news events, anything that could change the communal mentality,

that's going to affect the way people see the safety that we're doing. And so I think that this is like the bread and butter. You hit on the hardest question that we have to answer every day, and we try to revise and review our answer every day to make sure that it's still applicable. But that is the bread and butter of what we do, right? We want to foster these healthy communities where people feel like they can be productive or creative without feeling like it's verging in toxicity, but maintaining freedom.

Hermes Frangoudis (32:41)
you

It's like replicating real life, right? Because you can put yourself in a lot of these situations that you know are always going to be positive and healthy, right? But sometimes you can't physically go there. So it's nice to know that that same version can exist in the digital world.

Ninny Wan (33:04)
Yeah, yeah.

Exactly, yeah. And it's also one of those things where, you know, because you and I might see what we consider abusive differently, it's also trying to make sure that we have a happy medium, right? Like we're not viewing too much to one side and too much to the other side. We see this a lot cross-culturally too. So again, going back to just making sure that we're making products for everybody, this is something that we have to talk about, right? There's going to be different cultural expectations on certain things like sexually explicit content between different parts of the world.

and we need to account for that.

Hermes Frangoudis (33:50)
Yeah, it's a very nuanced, but we appreciate you kind of fighting the good fight for everyone and really putting the safety first and that passion. We appreciate that. ⁓ So I do appreciate your time. And one of the things that I have at the end of all my interviews that we've been doing is kind of this wild card question. So if you weren't working in moderation or content safety,

Ninny Wan (33:55)
We're trying.

Yeah, we're trying.

Cool.

Hermes Frangoudis (34:17)
what other AI researcher or application are you passionate about exploring? Or would you be working?

Ninny Wan (34:25)
Okay, this is gonna be very controversial because I think all my engineering teams will be screaming when they watch this podcast later. But I actually am very passionate about the human-in-the-loop. I don't know if that was very obvious when I answered that question, but I think I'd probably work somewhere in the human-in-the-loop space. I think that that type of QA process is very important. So I'd probably work on one of the teams that enforce or provide human-in-the-loop services.

I thought it was gonna be a wild card about like, oh, what would you do if you couldn't work in AI? And I was like, oh no, I don't know, that's too hard. Don't ask me that question.

Hermes Frangoudis (34:57)
No, I think

That one's just too out there. But if you couldn't do what you're doing now, right? Like what else in that space are you passionate about? And that's really interesting that you're the human-in-the-loop. I could hear it when you were answering the question earlier, so I'm not surprised.

Ninny Wan (35:09)
Yes.

Yeah. It's just such an interesting space, right? Like you're on the front lines. You get to see all the stuff as it comes in. You get to work directly with all, with everybody, honestly, engineers, engineers making new machine learning models. You get to talk with policy. You're talking with regulatory affairs, legal. You get to understand cultural nuance. It just sounds really cool, honestly.

Hermes Frangoudis (35:30)
Being on the front lines of pushing the boundaries forward, understanding what boundaries are being pushed forward and what directions. It sounds very interesting.

Ninny Wan (35:37)
Yes.

Yeah, and I feel like it would also just be an interesting place to ask existential questions like, you know, how much sexual is sexual? You know, like, is this sexual in all countries or just some countries? You know, so that's kind of an interesting way to, yeah, yeah.

Hermes Frangoudis (35:51)
Tapping into your inner philosophy philosopher.

He got me all wanting to I want to be in that industry.

Ninny Wan (36:01)
⁓ Yeah,

Yeah, I mean, hey, this, that's the other cool thing. Anyone can be here, right? Some of my closest engineers that I work with are professional flute players in their past lives. They have a side band on the side. You don't have to be musical. Some people used to be mathematicians, you know, there's a, there's room for everybody in this space for sure.

Hermes Frangoudis (36:24)
Thank you so much, Ninny, for joining me today. And I want to thank everyone that's following along and watching live. This is the Convo AI World podcast where we're interviewing and speaking with the teams that are building the really coolest AI technologies in the conversational AI space. Thanks again to Ninny Wan from Google and the content safety team. Please remember to like, subscribe, retweet, do the whole social media thing. We'll be here and we'll see you on the next one.

Ninny Wan (36:50)
Thank you so much for having me. This was great.