back to indexCristos Goodrow: YouTube Algorithm | Lex Fridman Podcast #68
link |
The following is a conversation with Christos Goudreau,
link |
Vice President of Engineering at Google and Head of Search and Discovery at YouTube,
link |
also known as the YouTube Algorithm.
link |
YouTube has approximately 1.9 billion users,
link |
and every day people watch over 1 billion hours of YouTube video.
link |
It is the second most popular search engine behind Google itself.
link |
For many people, it is not only a source of entertainment,
link |
but also how we learn new ideas from math and physics videos to podcasts to debates, opinions,
link |
ideas from out of the box thinkers and activists on some of the most tense,
link |
challenging, and impactful topics in the world today.
link |
YouTube and other content platforms receive criticism from both viewers and creators,
link |
as they should, because the engineering task before them is hard, and they don't always
link |
succeed, and the impact of their work is truly world changing.
link |
To me, YouTube has been an incredible wellspring of knowledge.
link |
I've watched hundreds, if not thousands, of lectures that changed the way I see
link |
many fundamental ideas in math, science, engineering, and philosophy.
link |
But it does put a mirror to ourselves, and keeps the responsibility of the steps we take
link |
in each of our online educational journeys into the hands of each of us.
link |
The YouTube algorithm has an important role in that journey of helping us find new,
link |
exciting ideas to learn about.
link |
That's a difficult and an exciting problem for an artificial intelligence system.
link |
As I've said in lectures and other forums, recommendation systems will be one of the
link |
most impactful areas of AI in the 21st century, and YouTube is one of the biggest
link |
recommendation systems in the world.
link |
This is the Artificial Intelligence Podcast.
link |
If you enjoy it, subscribe on YouTube, give it five stars on Apple Podcast, follow on
link |
Spotify, support it on Patreon, or simply connect with me on Twitter, at Lex Friedman,
link |
spelled F R I D M A N.
link |
I recently started doing ads at the end of the introduction.
link |
I'll do one or two minutes after introducing the episode, and never any ads in the middle
link |
that can break the flow of the conversation.
link |
I hope that works for you and doesn't hurt the listening experience.
link |
This show is presented by Cash App, the number one finance app in the App Store.
link |
I personally use Cash App to send money to friends, but you can also use it to buy,
link |
sell, and deposit Bitcoin in just seconds.
link |
Cash App also has a new investing feature.
link |
You can buy fractions of a stock, say, $1 worth, no matter what the stock price is.
link |
Broker services are provided by Cash App Investing, a subsidiary of Square and Member
link |
I'm excited to be working with Cash App to support one of my favorite organizations
link |
called FIRST, best known for their FIRST Robotics and Lego competitions.
link |
They educate and inspire hundreds of thousands of students in over 110 countries and have
link |
a perfect rating and charity navigator, which means that donated money is used to maximum
link |
When you get Cash App from the App Store or Google Play and use code LEXPODCAST, you'll
link |
get $10, and Cash App will also donate $10 to FIRST, which again is an organization that
link |
I've personally seen inspire girls and boys to dream of engineering a better world.
link |
And now, here's my conversation with Christos Goudreau.
link |
YouTube is the world's second most popular search engine, behind Google, of course.
link |
We watch more than 1 billion hours of YouTube videos a day, more than Netflix and Facebook
link |
YouTube creators upload over 500,000 hours of video every day.
link |
Average lifespan of a human being, just for comparison, is about 700,000 hours.
link |
So, what's uploaded every single day is just enough for a human to watch in a lifetime.
link |
So, let me ask an absurd philosophical question.
link |
If from birth, when I was born, and there's many people born today with the internet,
link |
I watched YouTube videos nonstop, do you think there are trajectories through YouTube video
link |
space that can maximize my average happiness, or maybe education, or my growth as a human
link |
I think there are some great trajectories through YouTube videos, but I wouldn't recommend
link |
that anyone spend all of their waking hours or all of their hours watching YouTube.
link |
I mean, I think about the fact that YouTube has been really great for my kids, for instance.
link |
My oldest daughter, she's been watching YouTube for several years.
link |
She watches Tyler Oakley and the Vlogbrothers, and I know that it's had a very profound and
link |
positive impact on her character.
link |
And my younger daughter, she's a ballerina, and her teachers tell her that YouTube is
link |
a huge advantage for her because she can practice a routine and watch professional dancers do
link |
that same routine and stop it and back it up and rewind and all that stuff, right?
link |
So, it's been really good for them.
link |
And then even my son is a sophomore in college.
link |
He got through his linear algebra class because of a channel called Three Blue, One Brown,
link |
which helps you understand linear algebra, but in a way that would be very hard for anyone
link |
to do on a whiteboard or a chalkboard.
link |
And so, I think that those experiences, from my point of view, were very good.
link |
And so, I can imagine really good trajectories through YouTube, yes.
link |
Have you looked at, do you think of broadly about that trajectory over a period?
link |
Because YouTube has grown up now.
link |
So, over a period of years, you just kind of gave a few anecdotal examples, but I used
link |
to watch certain shows on YouTube.
link |
I've moved on to other shows.
link |
Ultimately, you want people to, from YouTube's perspective, to stay on YouTube, to grow as
link |
human beings on YouTube.
link |
So, you have to think not just what makes them engage today or this month, but what
link |
makes them engage today or this month, but also for a period of years.
link |
I mean, if YouTube is going to continue to enrich people's lives, then it has to grow
link |
with them, and people's interests change over time.
link |
And so, I think we've been working on this problem, and I'll just say it broadly as
link |
like how to introduce diversity and introduce people who are watching one thing to something
link |
else they might like.
link |
We've been working on that problem all the eight years I've been at YouTube.
link |
It's a hard problem because, I mean, of course, it's trivial to introduce diversity
link |
that doesn't help.
link |
Yeah, just add a random video.
link |
I could just randomly select a video from the billions that we have.
link |
It's likely not to even be in your language.
link |
So, the likelihood that you would watch it and develop a new interest is very, very low.
link |
And so, what you want to do when you're trying to increase diversity is find something that
link |
is not too similar to the things that you've watched, but also something that you might
link |
be likely to watch.
link |
And that balance, finding that spot between those two things is quite challenging.
link |
So, the diversity of content, diversity of ideas, it's a really difficult, it's a thing
link |
like that's almost impossible to define, right?
link |
Like, what's different?
link |
So, how do you think about that?
link |
So, two examples is I'm a huge fan of Three Blue One Brown, say, and then one diversity.
link |
I wasn't even aware of a channel called Veritasium, which is a great science, physics, whatever
link |
So, one version of diversity is showing me Derek's Veritasium channel, which I was really
link |
excited to discover.
link |
I actually now watch a lot of his videos.
link |
Okay, so you're a person who's watching some math channels and you might be interested
link |
in some other science or math channels.
link |
So, like you mentioned, the first kind of diversity is just show you some things from
link |
other channels that are related, but not just, you know, not all the Three Blue One Brown
link |
channel, throw in a couple others.
link |
So, that's maybe the first kind of diversity that we started with many, many years ago.
link |
Taking a bigger leap is about, I mean, the mechanisms we use for that is we basically
link |
cluster videos and channels together, mostly videos.
link |
We do almost everything at the video level.
link |
And so, we'll make some kind of a cluster via some embedding process and then measure
link |
what is the likelihood that users who watch one cluster might also watch another cluster
link |
that's very distinct.
link |
So, we may come to find that people who watch science videos also like jazz.
link |
This is possible, right?
link |
And so, because of that relationship that we've identified through the embeddings and
link |
then the measurement of the people who watch both, we might recommend a jazz video once
link |
So, there's this cluster in the embedding space of jazz videos and science videos.
link |
And so, you kind of try to look at aggregate statistics where if a lot of people that jump
link |
from science cluster to the jazz cluster tend to remain as engaged or become more engaged,
link |
then that means those two, they should hop back and forth and they'll be happy.
link |
There's a higher likelihood that a person who's watching science would like jazz than
link |
the person watching science would like, I don't know, backyard railroads or something
link |
And so, we can try to measure these likelihoods and use that to make the best recommendation
link |
So, we'll talk about the machine learning of that, but I have to linger on things that
link |
neither you or anyone have an answer to.
link |
There's gray areas of truth, which is, for example, now I can't believe I'm going there,
link |
It happens so that certain people believe certain things and they're very certain about
link |
Let's move outside the red versus blue politics of today's world, but there's different ideologies.
link |
For example, in college, I read quite a lot of Ayn Rand I studied, and that's a particular
link |
philosophical ideology I found interesting to explore.
link |
So, that was that kind of space.
link |
I've kind of moved on from that cluster intellectually, but it nevertheless is an interesting cluster.
link |
I was born in the Soviet Union.
link |
Socialism, communism is a certain kind of political ideology that's really interesting
link |
Again, objectively, there's a set of beliefs about how the economy should work and so on.
link |
And so, it's hard to know what's true or not in terms of people within those communities
link |
are often advocating that this is how we achieve utopia in this world, and they're pretty
link |
So, how do you try to manage politics in this chaotic, divisive world?
link |
Not positive or any kind of ideas in terms of filtering what people should watch next
link |
and in terms of also not letting certain things be on YouTube.
link |
This is an exceptionally difficult responsibility.
link |
Well, the responsibility to get this right is our top priority.
link |
And the first comes down to making sure that we have good, clear rules of the road, right?
link |
Like, just because we have freedom of speech doesn't mean that you can literally say anything,
link |
Like, we as a society have accepted certain restrictions on our freedom of speech.
link |
There are things like libel laws and things like that.
link |
And so, where we can draw a clear line, we do, and that's what we do.
link |
We draw a clear line, we do, and we continue to evolve that line over time.
link |
However, as you pointed out, wherever you draw the line, there's going to be a border
link |
And in that border line area, we are going to maybe not remove videos, but we will try
link |
to reduce the recommendations of them or the proliferation of them by demoting them.
link |
Alternatively, in those situations, try to raise what we would call authoritative or
link |
credible sources of information.
link |
So, we're not trying to, I mean, you mentioned Ayn Rand and communism.
link |
Those are two valid points of view that people are going to debate and discuss.
link |
And of course, people who believe in one or the other of those things are going to try
link |
to persuade other people to their point of view.
link |
And so, we're not trying to settle that or choose a side or anything like that.
link |
What we're trying to do is make sure that the people who are expressing those point
link |
of view and offering those positions are authoritative and credible.
link |
So, let me ask a question about people I don't like personally.
link |
I don't care if you leave comments on this.
link |
But sometimes, they're brilliantly funny, which is trolls.
link |
So, people who kind of mock, I mean, the internet is full, Reddit of mock style
link |
comedy where people just kind of make fun of, point out that the emperor has no clothes.
link |
And there's brilliant comedy in that, but sometimes it can get cruel and mean.
link |
So, on that, on the mean point, and sorry to look at the comments, but I'm going to
link |
and sorry to linger on these things that have no good answers.
link |
But actually, I totally hear you that this is really important that you're trying to
link |
But how do you reduce the meanness of people on YouTube?
link |
I understand that anyone who uploads YouTube videos has to become resilient to a certain
link |
amount of meanness.
link |
Like I've heard that from many creators.
link |
And we are trying in various ways, comment ranking, allowing certain features to block
link |
people, to reduce or make that meanness or that trolling behavior less effective on YouTube.
link |
And so, I mean, it's very important, but it's something that we're going to keep having
link |
to work on and as we improve it, like maybe we'll get to a point where people don't have
link |
to suffer this sort of meanness when they upload YouTube videos.
link |
I hope we do, but it just does seem to be something that you have to be able to deal
link |
with as a YouTube creator nowadays.
link |
Do you have a hope that, so you mentioned two things that I kind of agree with.
link |
So there's like a machine learning approach of ranking comments based on whatever, based
link |
on how much they contribute to the healthy conversation.
link |
Let's put it that way.
link |
Then the other is almost an interface question of how do you, how does the creator filter?
link |
So block or how does, how do humans themselves, the users of YouTube manage their own conversation?
link |
Do you have hope that these two tools will create a better society without limiting freedom
link |
of speech too much, without sort of attacking, even like saying that people, what do you
link |
mean limiting, sort of curating speech?
link |
I mean, I think that that overall is our whole project here at YouTube.
link |
Like we fundamentally believe and I personally believe very much that YouTube can be great.
link |
It's been great for my kids.
link |
I think it can be great for society.
link |
But it's absolutely critical that we get this responsibility part right.
link |
And that's why it's our top priority.
link |
Susan Wojcicki, who's the CEO of YouTube, she says something that I personally find
link |
very inspiring, which is that we want to do our jobs today in a manner so that people
link |
20 and 30 years from now will look back and say, YouTube, they really figured this out.
link |
They really found a way to strike the right balance between the openness and the value
link |
that the openness has and also making sure that we are meeting our responsibility to
link |
So the burden on YouTube actually is quite incredible.
link |
And the one thing that people don't give enough credit to the seriousness and the magnitude
link |
of the problem, I think.
link |
So I personally hope that you do solve it because a lot is in your hand, a lot is riding
link |
on your success or failure.
link |
So it's besides, of course, running a successful company, you're also curating the content
link |
of the internet and the conversation on the internet.
link |
That's a powerful thing.
link |
So one thing that people wonder about is how much of it can be solved with pure machine
link |
So looking at the data, studying the data and creating algorithms that curate the comments,
link |
curate the content, and how much of it needs human intervention, meaning people here at
link |
YouTube in a room sitting and thinking about what is the nature of truth, what are the
link |
ideals that we should be promoting, that kind of thing.
link |
So algorithm versus human input, what's your sense?
link |
I mean, my own experience has demonstrated that you need both of those things.
link |
Algorithms, I mean, you're familiar with machine learning algorithms and the thing
link |
they need most is data and the data is generated by humans.
link |
And so, for instance, when we're building a system to try to figure out which are the
link |
videos that are misinformation or borderline policy violations, well, the first thing we
link |
need to do is get human beings to make decisions about which of those videos are in which category.
link |
And then we use that data and basically take that information that's determined and governed
link |
by humans and extrapolate it or apply it to the entire set of billions of YouTube videos.
link |
And we couldn't get to all the videos on YouTube well without the humans, and we couldn't use
link |
the humans to get to all the videos of YouTube.
link |
So there's no world in which you have only one or the other of these things.
link |
And just as you said, a lot of it comes down to people at YouTube spending a lot of time
link |
trying to figure out what are the right policies, what are the outcomes based on those policies,
link |
are they the kinds of things we want to see?
link |
And then once we kind of get an agreement or build some consensus around what the policies
link |
are, well, then we've got to find a way to implement those policies across all of YouTube.
link |
And that's where both the human beings, we call them evaluators or reviewers, come into
link |
play to help us with that.
link |
And then once we get a lot of training data from them, then we apply the machine learning
link |
techniques to take it even further.
link |
Do you have a sense that these human beings have a bias in some kind of direction?
link |
I mean, that's an interesting question.
link |
We do sort of in autonomous vehicles and computer vision in general, a lot of annotation, and
link |
we rarely ask what bias do the annotators have.
link |
Even in the sense that they're better at annotating certain things than others.
link |
For example, people are much better at, for example, at writing, they're much better at
link |
or much better at annotating segmentation at segmenting cars in a scene versus segmenting
link |
There's specific mechanical reasons for that, but also because it's semantic gray area.
link |
And just for a lot of reasons, people are just terrible at annotating trees.
link |
Okay, so in the same kind of sense, do you think of, in terms of people reviewing videos
link |
or annotating the content of videos, is there some kind of bias that you're aware of or
link |
seek out in that human input?
link |
Well, we take steps to try to overcome these kinds of biases or biases that we think would
link |
So for instance, like we ask people to have a bias towards scientific consensus.
link |
That's something that we instruct them to do.
link |
We ask them to have a bias towards demonstration of expertise or credibility or authoritativeness.
link |
But there are other biases that we want to make sure to try to remove.
link |
And there's many techniques for doing this.
link |
One of them is you send the same thing to be reviewed to many people.
link |
And so, that's one technique.
link |
Another is that you make sure that the people that are doing these sorts of tasks, that
link |
these sorts of tasks are from different backgrounds and different areas of the United States or
link |
But then, even with all of that, it's possible for certain kinds of what we would call unfair
link |
biases to creep into machine learning systems, primarily, as you said, because maybe the
link |
training data itself comes in in a biased way.
link |
So, we also have worked very hard on improving the machine learning systems to remove and
link |
reduce unfair biases when it goes against or involves some protected class, for instance.
link |
Thank you for exploring with me some of the more challenging things.
link |
I'm sure there's a few more that we'll jump back to.
link |
But let me jump into the fun part, which is maybe the basics of the quote, unquote, YouTube
link |
What does the YouTube algorithm look at to make recommendation for what to watch next?
link |
And it's from a machine learning perspective.
link |
Or when you search for a particular term, how does it know what to show you next?
link |
Because it seems to, at least for me, do an incredible job of both.
link |
Well, that's kind of you to say.
link |
It didn't used to do a very good job, but it's gotten better over the years.
link |
Even I observed that it's improved quite a bit.
link |
Those are two different situations.
link |
Like when you search for something, YouTube uses the best technology we can get from Google
link |
to make sure that the YouTube search system finds what someone's looking for.
link |
And of course, the very first things that one thinks about is, okay, well, does the
link |
word occur in the title, for instance?
link |
But there are much more sophisticated things where we're mostly trying to do some syntactic
link |
match or maybe a semantic match based on words that we can add to the document itself.
link |
For instance, maybe is this video watched a lot after this query?
link |
That's something that we can observe and then as a result, make sure that that document
link |
would be retrieved for that query.
link |
Now, when you talk about what kind of videos would be recommended to watch next, that's
link |
something, again, we've been working on for many years and probably the first real attempt
link |
to do that well was to use collaborative filtering.
link |
Can you describe what collaborative filtering is?
link |
It's just basically what we do is we observe which videos get watched close together by
link |
And if you observe that and if you can imagine creating a graph where the videos that get
link |
watched close together by the most people are very close to one another in this graph
link |
and videos that don't frequently get watched close together by the same person or the same
link |
people are far apart, then you end up with this graph that we call the related graph
link |
that basically represents videos that are very similar or related in some way.
link |
And what's amazing about that is that it puts all the videos that are in the same
link |
language together, for instance, and we didn't even have to think about language.
link |
It just does it, right?
link |
And it puts all the videos that are about sports together and it puts most of the music
link |
videos together and it puts all of these sorts of videos together just because that's sort
link |
of the way the people using YouTube behave.
link |
So that already cleans up a lot of the problem.
link |
It takes care of the lowest hanging fruit, which happens to be a huge one of just managing
link |
these millions of videos.
link |
I remember a few years ago I was talking to someone who was trying to propose that we
link |
do a research project concerning people who are bilingual, and this person was making
link |
this proposal based on the idea that YouTube could not possibly be good at recommending
link |
videos well to people who are bilingual.
link |
And so she was telling me about this and I said, well, can you give me an example of
link |
what problem do you think we have on YouTube with the recommendations?
link |
And so she said, well, I'm a researcher in the US and when I'm looking for academic
link |
topics, I want to see them in English.
link |
And so she searched for one, found a video, and then looked at the watch next suggestions
link |
and they were all in English.
link |
And so she said, oh, I see.
link |
YouTube must think that I speak only English.
link |
And so she said, now I'm actually originally from Turkey and sometimes when I'm cooking,
link |
let's say I want to make some baklava, I really like to watch videos that are in Turkish.
link |
And so she searched for a video about making the baklava and then selected it and it was
link |
in Turkish and the watch next recommendations were in Turkish.
link |
And she just couldn't believe how this was possible and how is it that you know that
link |
I speak both these two languages and put all the videos together?
link |
And it's just as a sort of an outcome of this related graph that's created through
link |
collaborative filtering.
link |
So for me, one of my huge interests is just human psychology, right?
link |
And that's such a powerful platform on which to utilize human psychology to discover what
link |
people, individual people want to watch next.
link |
But it's also be just fascinating to me.
link |
You know, I've, Google search has ability to look at your own history and I've done
link |
that before, just, just what I've searched three years for many, many years.
link |
And it's fascinating picture of who I am actually.
link |
And I don't think anyone's ever summarized.
link |
I personally would love that.
link |
A summary of who I am as a person on the internet to me, because I didn't get a reply
link |
of who I am as a person on the internet to me, because I think it reveals, I think it
link |
puts a mirror to me or to others.
link |
You know, that's actually quite revealing and interesting, you know, just the, maybe
link |
in the number of, it's a joke, but not really is the number of cat videos I've watched or
link |
videos of people falling, you know, stuff that's absurd, that kind of stuff.
link |
It's really interesting.
link |
And of course it's really good for the machine learning aspect to, to show, to figure out
link |
what to show next.
link |
But it's interesting.
link |
Have you just as a tangent played around with the idea of giving a map to people sort of,
link |
as opposed to just using this information to show what's next, showing them here are
link |
the clusters you've loved over the years kind of thing?
link |
Well, we do provide the history of all the videos that you've watched.
link |
So you can definitely search through that and look through it and search through it
link |
to see what it is that you've been watching on YouTube.
link |
We have actually in various times experimented with this sort of cluster idea, finding ways
link |
to demonstrate or show people what topics they've been interested in or what clusters
link |
they've watched from.
link |
It's interesting that you bring this up because in some sense, the way the recommendation
link |
system of YouTube sees a user is exactly as the history of all the videos they've
link |
watched on YouTube.
link |
And so you can think of yourself or any user on YouTube as kind of like a DNA strand of
link |
all your videos, right?
link |
That sort of represents you, you can also think of it as maybe a vector in the space
link |
of all the videos on YouTube.
link |
And so now once you think of it as a vector in the space of all the videos on YouTube,
link |
then you can start to say, okay, well, which other vectors are close to me and to my vector?
link |
And that's one of the ways that we generate some diverse recommendations is because you're
link |
like, okay, well, these people seem to be close with respect to the videos they've
link |
watched on YouTube, but here's a topic or a video that one of them has watched and
link |
enjoyed, but the other one hasn't, that could be an opportunity to make a good recommendation.
link |
I got to tell you, I mean, I know I'm going to ask for things that are impossible, but
link |
I would love to cluster than human beings.
link |
I would love to know who has similar trajectories as me, because you probably would want to
link |
There's a social aspect there, like actually finding some of the most fascinating people
link |
I find on YouTube, but have like no followers and I start following them and they create
link |
incredible content and on that topic, I just love to ask, there's some videos that just
link |
blow my mind in terms of quality and depth and just in every regard are amazing videos
link |
and they have like 57 views, okay?
link |
How do you get videos of quality to be seen by many eyes?
link |
So the measure of quality, is it just something, yeah, how do you know that something is good?
link |
Well, I mean, I think it depends initially on what sort of video we're talking about.
link |
So in the realm of, let's say you mentioned politics and news, in that realm, you know,
link |
quality news or quality journalism relies on having a journalism department, right?
link |
Like you have to have actual journalists and fact checkers and people like that and so
link |
in that situation and in others, maybe science or in medicine, quality has a lot to do with
link |
the authoritativeness and the credibility and the expertise of the people who make the
link |
Now, if you think about the other end of the spectrum, you know, what is the highest quality
link |
prank video or what is the highest quality Minecraft video, right?
link |
That might be the one that people enjoy watching the most and watch to the end or it might
link |
be the one that when we ask people the next day after they watched it, were they satisfied
link |
And so we in, especially in the realm of entertainment, have been trying to get at better and better
link |
measures of quality or satisfaction or enrichment since I came to YouTube.
link |
And we started with, well, you know, the first approximation is the one that gets more views.
link |
But you know, we both know that things can get a lot of views and not really be that
link |
high quality, especially if people are clicking on something and then immediately realizing
link |
that it's not that great and abandoning it.
link |
And that's why we moved from views to thinking about the amount of time people spend watching
link |
it with the premise that like, you know, in some sense, the time that someone spends watching
link |
a video is related to the value that they get from that video.
link |
It may not be perfectly related, but it has something to say about how much value they
link |
But even that's not good enough, right?
link |
Because I myself have spent time clicking through channels on television late at night
link |
and ended up watching Under Siege 2 for some reason I don't know.
link |
And if you were to ask me the next day, are you glad that you watched that show on TV
link |
I'd say, yeah, I wish I would have gone to bed or read a book or almost anything else,
link |
And so that's why some people got the idea a few years ago to try to survey users afterwards.
link |
And so we get feedback data from those surveys and then use that in the machine learning
link |
system to try to not just predict what you're going to click on right now, what you might
link |
watch for a while, but what when we ask you tomorrow, you'll give four or five stars to.
link |
So just to summarize, what are the signals from a machine learning perspective that a
link |
So you mentioned just clicking on the video views, the time watched, maybe the relative
link |
time watched, the clicking like and dislike on the video, maybe commenting on the video.
link |
All of those things.
link |
All of those things.
link |
And then the one I wasn't actually quite aware of, even though I might have engaged in it
link |
is a survey afterwards, which is a brilliant idea.
link |
Is there other signals?
link |
I mean, that's already a really rich space of signals to learn from.
link |
Is there something else?
link |
Well, you mentioned commenting, also sharing the video.
link |
If you think it's worthy to be shared with someone else you know.
link |
Within YouTube or outside of YouTube as well?
link |
Let's see, you mentioned like, dislike.
link |
How important is that?
link |
It's very important, right?
link |
We want, it's predictive of satisfaction.
link |
But it's not perfectly predictive.
link |
If you subscribe to the channel of the person who made the video, then that also is a piece
link |
of information and it signals satisfaction.
link |
Although over the years, we've learned that people have a wide range of attitudes about
link |
what it means to subscribe.
link |
We would ask some users who didn't subscribe very much, but they watched a lot from a few
link |
We'd say, well, why didn't you subscribe?
link |
And they would say, well, I can't afford to pay for anything.
link |
We tried to let them understand like, actually it doesn't cost anything.
link |
It just helps us know that you are very interested in this creator.
link |
But then we've asked other people who subscribe to many things and don't really watch any
link |
of the videos from those channels.
link |
And we say, well, why did you subscribe to this if you weren't really interested in any
link |
more videos from that channel?
link |
And they might tell us, well, I just, you know, I thought the person did a great job
link |
and I just want to kind of give them a high five.
link |
That's where I sit.
link |
I go to channels where I just, this person is amazing.
link |
I like this person.
link |
But then I like this person and I really want to support them.
link |
That's how I click subscribe.
link |
Even though I mean never actually want to click on their videos when they're releasing
link |
I just love what they're doing.
link |
And it's maybe outside of my interest area and so on, which is probably the wrong way
link |
to use the subscribe button.
link |
But I just want to say congrats.
link |
This is great work.
link |
Well, so you have to deal with all the space of people that see the subscribe button is
link |
totally different.
link |
And so, you know, we can't just close our eyes and say, sorry, you're using it wrong.
link |
You know, we're not going to pay attention to what you've done.
link |
We need to embrace all the ways in which all the different people in the world use the
link |
subscribe button or the like and the dislike button.
link |
So in terms of signals of machine learning, using for the search and for the recommendation,
link |
you've mentioned title.
link |
So like metadata, like text data that people provide description and title and maybe keywords.
link |
Maybe you can speak to the value of those things in search and also this incredible
link |
fascinating area of the content itself.
link |
So the video content itself, trying to understand what's happening in the video.
link |
So YouTube released a data set that, you know, in the machine learning computer vision world,
link |
this is just an exciting space.
link |
How much is that currently?
link |
How much are you playing with that currently?
link |
How much is your hope for the future of being able to analyze the content of the video itself?
link |
Well, we have been working on that also since I came to YouTube.
link |
Analyzing the content.
link |
Analyzing the content of the video, right?
link |
And what I can tell you is that our ability to do it well is still somewhat crude.
link |
We can tell if it's a music video, we can tell if it's a sports video, we can probably
link |
tell you that people are playing soccer.
link |
We probably can't tell whether it's Manchester United or my daughter's soccer team.
link |
So these things are kind of difficult and using them, we can use them in some ways.
link |
So for instance, we use that kind of information to understand and inform these clusters that
link |
And also maybe to add some words like soccer, for instance, to the video, if it doesn't
link |
occur in the title or the description, which is remarkable that often it doesn't.
link |
One of the things that I ask creators to do is please help us out with the title and the
link |
For instance, we were a few years ago having a live stream of some competition for World
link |
of Warcraft on YouTube.
link |
And it was a very important competition, but if you typed World of Warcraft in search,
link |
you wouldn't find it.
link |
World of Warcraft wasn't in the title?
link |
World of Warcraft wasn't in the title.
link |
It was match 478, you know, A team versus B team and World of Warcraft wasn't in the
link |
I'm just like, come on, give me.
link |
Being literal on the internet is actually very uncool, which is the problem.
link |
Oh, is that right?
link |
Well, I mean, in some sense, well, some of the greatest videos, I mean, there's a humor
link |
to just being indirect, being witty and so on.
link |
And actually being, you know, machine learning algorithms want you to be, you know, literal,
link |
You just want to say what's in the thing, be very, very simple.
link |
And in some sense that gets away from wit and humor.
link |
So you have to play with both, right?
link |
But you're saying that for now, sort of the content of the title, the content of the description,
link |
the actual text is one of the best ways for the algorithm to find your video and put them
link |
in the right cluster.
link |
And I would go further and say that if you want people, human beings to select your video
link |
in search, then it helps to have, let's say World of Warcraft in the title.
link |
Because why would a person, you know, if they're looking at a bunch, they type World of Warcraft
link |
and they have a bunch of videos, all of whom say World of Warcraft, except the one that
link |
Well, even the person is going to think, well, maybe this isn't somehow search made a mistake.
link |
This isn't really about World of Warcraft.
link |
So it's important not just for the machine learning systems, but also for the people
link |
who might be looking for this sort of thing.
link |
They get a clue that it's what they're looking for by seeing that same thing prominently
link |
in the title of the video.
link |
Let me push back on that.
link |
So I think from the algorithm perspective, yes, but if they typed in World of Warcraft
link |
and saw a video that with the title simply winning and the thumbnail has like a sad orc
link |
or something, I don't know, right?
link |
Like I think that's much, it gets your curiosity up.
link |
And then if they could trust that the algorithm was smart enough to figure out somehow that
link |
this is indeed a World of Warcraft video, that would have created the most beautiful
link |
I think in terms of just the wit and the humor and the curiosity that we human beings naturally
link |
But you're saying, I mean, realistically speaking, it's really hard for the algorithm
link |
to figure out that the content of that video will be a World of Warcraft video.
link |
And you have to accept that some people are going to skip it.
link |
I mean, and so you're right.
link |
The people who don't skip it and select it are going to be delighted, but other people
link |
might say, yeah, this is not what I was looking for.
link |
And making stuff discoverable, I think is what you're really working on and hoping.
link |
So from your perspective, put stuff in the title description.
link |
And remember the collaborative filtering part of the system starts by the same user watching
link |
videos together, right?
link |
So the way that they're probably going to do that is by searching for them.
link |
That's a fascinating aspect of it.
link |
It's like ant colonies.
link |
That's how they find stuff.
link |
So I mean, what degree for collaborative filtering in general is one curious ant, one curious
link |
So just a person who is more willing to click on random videos and sort of explore these
link |
In your sense, how many people are just like watching the same thing over and over and
link |
And how many are just like the explorers and just kind of like click on stuff and then
link |
help the other ant in the ant's colony discover the cool stuff?
link |
Do you have a sense of that at all?
link |
I really don't think I have a sense for the relative sizes of those groups.
link |
But I would say that people come to YouTube with some certain amount of intent.
link |
And as long as they, to the extent to which they try to satisfy that intent, that certainly
link |
helps our systems, right?
link |
Because our systems rely on kind of a faithful amount of behavior, right?
link |
And there are people who try to trick us, right?
link |
There are people and machines that try to associate videos together that really don't
link |
belong together, but they're trying to get that association made because it's profitable
link |
And so we have to always be resilient to that sort of attempt at gaming the systems.
link |
So speaking to that, there's a lot of people that in a positive way, perhaps, I don't know,
link |
I don't like it, but like to want to try to game the system to get more attention.
link |
Everybody creators in a positive sense want to get attention, right?
link |
So how do you work in this space when people create more and more sort of click baity titles
link |
Sort of very to ask him, Derek has made a video where basically describes that it seems
link |
what works is to create a high quality video, really good video, where people would want
link |
to watch it once they click on it, but have click baity titles and thumbnails to get them
link |
to click on it in the first place.
link |
And he's saying, I'm embracing this fact, I'm just going to keep doing it.
link |
And I hope you forgive me for doing it and you will enjoy my videos once you click on
link |
So in what sense do you see this kind of click bait style attempt to manipulate, to get people
link |
in the door to manipulate the algorithm or play with the algorithm or game the algorithm?
link |
I think that you can look at it as an attempt to game the algorithm.
link |
But even if you were to take the algorithm out of it and just say, okay, well, all these
link |
videos happen to be lined up, which the algorithm didn't make any decision about which one to
link |
put at the top or the bottom, but they're all lined up there, which one are the people
link |
And I'll tell you the same thing that I told Derek is, I have a bookshelf and they have
link |
two kinds of books on them, science books.
link |
I have my math books from when I was a student and they all look identical except for the
link |
titles on the covers.
link |
They're all yellow, they're all from Springer and they're every single one of them.
link |
The cover is totally the same.
link |
On the other hand, I have other more pop science type books and they all have very interesting
link |
covers and they have provocative titles and things like that.
link |
I wouldn't say that they're click baity because they are indeed good books.
link |
And I don't think that they cross any line, but that's just a decision you have to make.
link |
Like the people who write classical recursion theory by Piero di Freddie, he was fine with
link |
the yellow title and nothing more.
link |
Whereas I think other people who wrote a more popular type book understand that they need
link |
to have a compelling cover and a compelling title.
link |
And I don't think there's anything really wrong with that.
link |
We do take steps to make sure that there is a line that you don't cross.
link |
And if you go too far, maybe your thumbnail is especially racy or it's all caps with too
link |
many exclamation points, we observe that users are sometimes offended by that.
link |
And so for the users who are offended by that, we will then depress or suppress those videos.
link |
And which reminds me, there's also another signal where users can say, I don't know if
link |
it was recently added, but I really enjoy it.
link |
Just saying, something like, I don't want to see this video anymore or something like,
link |
like this is a, like there's certain videos that just cut me the wrong way.
link |
Like just, just jump out at me, it's like, I don't want to, I don't want this.
link |
And it feels really good to clean that up, to be like, I don't, that's not, that's not
link |
I think that might've been recently added, but that's also a really strong signal.
link |
We don't want to make a recommendation that people are unhappy with.
link |
And that makes me, that particular one makes me feel good as a user in general and as a
link |
machine learning person.
link |
Cause I feel like I'm helping the algorithm.
link |
My interactions on YouTube don't always feel like I'm helping the algorithm.
link |
Like I'm not reminded of that fact.
link |
Like for example, Tesla and Autopilot and Elon Musk create a feeling for their customers,
link |
for people that own Teslas, that they're helping the algorithm of Tesla vehicles.
link |
Like they're all, like are really proud they're helping the fleet learn.
link |
I think YouTube doesn't always remind people that you're helping the algorithm get smarter.
link |
And for me, I love that idea.
link |
Like we're all collaboratively, like Wikipedia gives that sense that we're all together creating
link |
a beautiful thing.
link |
YouTube is a, doesn't always remind me of that.
link |
It's a, this conversation is reminding me of that, but.
link |
Well that's a good tip.
link |
We should keep that fact in mind when we design these features.
link |
I'm not sure I really thought about it that way, but that's a very interesting perspective.
link |
It's an interesting question of personalization that I feel like when I click like on a video,
link |
I'm just improving my experience.
link |
It would be great.
link |
It would make me personally, people are different, but make me feel great if I was helping also
link |
the YouTube algorithm broadly say something.
link |
You know what I'm saying?
link |
Like there's a, that I don't know if that's human nature, but you want the products you
link |
love, and I certainly love YouTube, like you want to help it get smarter, smarter, smarter
link |
because there's some kind of coupling between our lives together being better.
link |
If YouTube is better than I will, my life will be better.
link |
And there's that kind of reasoning.
link |
I'm not sure what that is and I'm not sure how many people share that feeling.
link |
That could be just a machine learning feeling.
link |
But on that point, how much personalization is there in terms of next video recommendations?
link |
So is it kind of all really boiling down to clustering?
link |
Like if I'm the nearest clusters to me and so on and that kind of thing, or how much
link |
is personalized to me, the individual completely?
link |
It's very, very personalized.
link |
So your experience will be quite a bit different from anybody else's who's watching that same
link |
video, at least when they're logged in.
link |
And the reason is that we found that users often want two different kinds of things when
link |
they're watching a video.
link |
Sometimes they want to keep watching more on that topic or more in that genre.
link |
And other times they just are done and they're ready to move on to something else.
link |
And so the question is, well, what is the something else?
link |
And one of the first things one can imagine is, well, maybe something else is the latest
link |
video from some channel to which you've subscribed.
link |
And that's going to be very different for you than it is for me.
link |
And even if it's not something that you subscribe to, it's something that you watch a lot.
link |
And again, that'll be very different on a person by person basis.
link |
And so even the Watch Next, as well as the homepage, of course, is quite personalized.
link |
So what, we mentioned some of the signals, but what does success look like?
link |
What does success look like in terms of the algorithm creating a great long term experience
link |
Or to put another way, if you look at the videos I've watched this month, how do you
link |
know the algorithm succeeded for me?
link |
I think, first of all, if you come back and watch more YouTube, then that's one indication
link |
that you found some value from it.
link |
So just the number of hours is a powerful indicator.
link |
Well, I mean, not the hours themselves, but the fact that you return on another day.
link |
So that's probably the most simple indicator.
link |
People don't come back to things that they don't find value in, right?
link |
There's a lot of other things that they could do.
link |
But like I said, ideally, we would like everybody to feel that YouTube enriches their lives
link |
and that every video they watched is the best one they've ever watched since they've started
link |
And so that's why we survey them and ask them, is this one to five stars?
link |
And so our version of success is every time someone takes that survey, they say it's five
link |
And if we ask them, is this the best video you've ever seen on YouTube?
link |
They say, yes, every single time.
link |
So it's hard to imagine that we would actually achieve that.
link |
Maybe asymptotically we would get there, but that would be what we think success is.
link |
I've recently said somewhere, I don't know, maybe tweeted, but that Ray Dalio has this
link |
video on the economic machine, I forget what it's called, but it's a 30 minute video.
link |
And I said it's the greatest video I've ever watched on YouTube.
link |
It's like I watched the whole thing and my mind was blown as a very crisp, clean description
link |
of how the, at least the American economic system works.
link |
It's a beautiful video.
link |
And I was just, I wanted to click on something to say this is the best thing.
link |
This is the best thing ever.
link |
Please let me, I can't believe I discovered it.
link |
I mean, the views and the likes reflect its quality, but I was almost upset that I haven't
link |
found it earlier and wanted to find other things like it.
link |
I don't think I've ever felt that this is the best video I've ever watched.
link |
And to me, the ultimate utopia, the best experiences were every single video.
link |
Where I don't see any of the videos I regret and every single video I watch is one that
link |
actually helps me grow, helps me enjoy life, be happy and so on.
link |
So that's a heck of a, that's one of the most beautiful and ambitious, I think, machine
link |
So when you look at a society as opposed to the individual user, do you think of how YouTube
link |
is changing society when you have these millions of people watching videos, growing, learning,
link |
changing, having debates?
link |
Do you have a sense of, yeah, what the big impact on society is?
link |
I think it's huge, but do you have a sense of what direction we're taking this world?
link |
Well, I mean, I think openness has had an impact on society already.
link |
There's a lot of...
link |
What do you mean by openness?
link |
Well, the fact that unlike other mediums, there's not someone sitting at YouTube who
link |
decides before you can upload your video, whether it's worth having you upload it or
link |
worth anybody seeing it really, right?
link |
And so there are some creators who say, like, I wouldn't have this opportunity to reach
link |
Tyler Oakley often said that he wouldn't have had this opportunity to reach this audience
link |
if it weren't for YouTube.
link |
And so I think that's one way in which YouTube has changed society.
link |
I know that there are people that I work with from outside the United States, especially
link |
from places where literacy is low, and they think that YouTube can help in those places
link |
because you don't need to be able to read and write in order to learn something important
link |
for your life, maybe how to do some job or how to fix something.
link |
And so that's another way in which I think YouTube is possibly changing society.
link |
So I've worked at YouTube for eight, almost nine years now.
link |
And it's fun because I meet people and you tell them where you work, you say you work
link |
on YouTube and they immediately say, I love YouTube, right?
link |
Which is great, makes me feel great.
link |
But then of course, when I ask them, well, what is it that you love about YouTube?
link |
Not one time ever has anybody said that the search works outstanding or that the recommendations
link |
What they always say when I ask them, what do you love about YouTube is they immediately
link |
start talking about some channel or some creator or some topic or some community that they
link |
found on YouTube and that they just love.
link |
And so that has made me realize that YouTube is really about the video and connecting the
link |
people with the videos.
link |
And then everything else kind of gets out of the way.
link |
So beyond the video, it's an interesting, because you kind of mentioned creator.
link |
What about the connection with just the individual creators as opposed to just individual video?
link |
So like I gave the example of Ray Dalio video that the video itself is incredible, but there's
link |
some people who are just creators that I love.
link |
One of the cool things about people who call themselves YouTubers or whatever is they have
link |
They usually, almost all of them, they suck horribly in the beginning and then they kind
link |
of grow and then there's that genuineness in their growth.
link |
So YouTube clearly wants to help creators connect with their audience in this kind of
link |
So how do you think about that process of helping creators grow, helping them connect
link |
with their audience, develop not just individual videos, but the entirety of a creator's life
link |
Well, I mean, we're trying to help creators find the biggest audience that they can find.
link |
And the reason why that's, you brought up creator versus video, the reason why creator
link |
channel is so important is because if we have a hope of people coming back to YouTube, well,
link |
they have to have in their minds some sense of what they're going to find when they come
link |
If YouTube were just the next viral video and I have no concept of what the next viral
link |
video could be, one time it's a cat playing a piano and the next day it's some children
link |
interrupting a reporter and the next day it's some other thing happening, then it's hard
link |
for me to, when I'm not watching YouTube, say, gosh, I really would like to see something
link |
from someone or about something, right?
link |
And so that's why I think this connection between fans and creators is so important
link |
for both, because it's a way of sort of fostering a relationship that can play out into the
link |
Let me talk about kind of a dark and interesting question in general, and again, a topic that
link |
you or nobody has an answer to.
link |
But social media has a sense of, it gives us highs and it gives us lows in the sense
link |
that sort of creators often speak about having sort of burnout and having psychological ups
link |
and downs and challenges mentally in terms of continuing the creation process.
link |
There's a momentum, there's a huge excited audience that makes creators feel great.
link |
And I think it's more than just financial.
link |
I think it's literally just, they love that sense of community.
link |
It's part of the reason I upload to YouTube.
link |
I don't care about money, never will.
link |
What I care about is the community, but some people feel like this momentum, and even when
link |
there's times in their life when they don't feel, you know, for some reason don't feel
link |
So how do you think about burnout, this mental exhaustion that some YouTube creators go through?
link |
Is that something we have an answer for?
link |
Is that something, how do we even think about that?
link |
Well, the first thing is we want to make sure that the YouTube systems are not contributing
link |
to this sense, right?
link |
And so we've done a fair amount of research to demonstrate that you can absolutely take
link |
If you are a creator and you've been uploading a lot, we have just as many examples of people
link |
who took a break and came back more popular than they were before as we have examples
link |
of going the other way.
link |
Can we pause on that for a second?
link |
So the feeling that people have, I think, is if I take a break, everybody, the party
link |
will leave, right?
link |
So if you could just linger on that.
link |
So in your sense that taking a break is okay.
link |
Yes, taking a break is absolutely okay.
link |
And the reason I say that is because we have, we can observe many examples of being, of
link |
creators coming back very strong and even stronger after they have taken some sort of
link |
And so I just want to dispel the myth that this somehow necessarily means that your channel
link |
is going to go down or lose views.
link |
That is not the case.
link |
We know for sure that this is not a necessary outcome.
link |
And so we want to encourage people to make sure that they take care of themselves.
link |
That is job one, right?
link |
You have to look after yourself and your mental health.
link |
And I think that it probably, in some of these cases, contributes to better videos once they
link |
Because a lot of people, I mean, I know myself, if I burn out on something, then I'm probably
link |
not doing my best work, even though I can keep working until I pass out.
link |
And so I think that the taking a break may even improve the creative ideas that someone
link |
I think that's a really important thing to sort of dispel.
link |
I think that applies to all of social media, like literally I've taken a break for a day
link |
every once in a while.
link |
Sorry if that sounds like a short time, but even like, sorry, email, just taking a break
link |
from email, or only checking email once a day, especially when you're going through
link |
something psychologically in your personal life or so on, or really not sleeping much
link |
because of work deadlines, it can refresh you in a way that's profound.
link |
And so the same applies.
link |
It was there when you came back, right?
link |
And it looks different, actually, when you come back.
link |
You're sort of brighter eyed with some coffee, everything, the world looks better.
link |
So it's important to take a break when you need it.
link |
So you've mentioned kind of the YouTube algorithm that isn't E equals MC squared, it's not the
link |
single equation, it's potentially sort of more than a million lines of code.
link |
Is it more akin to what successful autonomous vehicles today are, which is they're just
link |
basically patches on top of patches of heuristics and human experts really tuning the algorithm
link |
and have some machine learning modules?
link |
Or is it becoming more and more a giant machine learning system with humans just doing a little
link |
bit of tweaking here and there?
link |
What's your sense?
link |
First of all, do you even have a sense of what is the YouTube algorithm at this point?
link |
And however much you do have a sense, what does it look like?
link |
Well, we don't usually think about it as the algorithm because it's a bunch of systems
link |
that work on different services.
link |
The other thing that I think people don't understand is that what you might refer to
link |
as the YouTube algorithm from outside of YouTube is actually a bunch of code and machine learning
link |
systems and heuristics, but that's married with the behavior of all the people who come
link |
to YouTube every day.
link |
So the people part of the code, essentially.
link |
If there were no people who came to YouTube tomorrow, then the algorithm wouldn't work
link |
That's the whole part of the algorithm.
link |
And so when people talk about, well, the algorithm does this, the algorithm does that, it's sometimes
link |
hard to understand, well, it could be the viewers are doing that.
link |
And the algorithm is mostly just keeping track of what the viewers do and then reacting to
link |
those things in sort of more fine grain situations.
link |
And I think that this is the way that the recommendation system and the search system
link |
and probably many machine learning systems evolve is you start trying to solve a problem
link |
and the first way to solve a problem is often with a simple heuristic.
link |
And you want to say, what are the videos we're going to recommend?
link |
Well, how about the most popular ones?
link |
That's where you start.
link |
And over time, you collect some data and you refine your situation so that you're making
link |
less heuristics and you're building a system that can actually learn what to do in different
link |
situations based on some observations of those situations in the past.
link |
And you keep chipping away at these heuristics over time.
link |
And so I think that just like with diversity, I think the first diversity measure we took
link |
was, okay, not more than three videos in a row from the same channel.
link |
It's a pretty simple heuristic to encourage diversity, but it worked, right?
link |
Who needs to see four, five, six videos in a row from the same channel?
link |
And over time, we try to chip away at that and make it more fine grain and basically
link |
have it remove the heuristics in favor of something that can react to individuals and
link |
individual situations.
link |
So how do you, you mentioned, you know, we know that something worked.
link |
How do you get a sense when decisions are kind of A, B testing that this idea was a
link |
good one, this was not so good?
link |
How do you measure that and across which time scale, across how many users, that kind of
link |
Well, you mentioned the A, B experiments.
link |
And so just about every single change we make to YouTube, we do it only after we've run
link |
a A, B experiment.
link |
And so in those experiments, which run from one week to months, we measure hundreds, literally
link |
hundreds of different variables and measure changes with confidence intervals in all of
link |
them, because we really are trying to get a sense for ultimately, does this improve
link |
the experience for viewers?
link |
That's the question we're trying to answer.
link |
And an experiment is one way because we can see certain things go up and down.
link |
So for instance, if we noticed in the experiment, people are dismissing videos less frequently,
link |
or they're saying that they're more satisfied, they're giving more videos five stars after
link |
they watch them, then those would be indications that the experiment is successful, that it's
link |
improving the situation for viewers.
link |
But we can also look at other things, like we might do user studies, where we invite
link |
some people in and ask them, like, what do you think about this?
link |
What do you think about that?
link |
How do you feel about this?
link |
And other various kinds of user research.
link |
But ultimately, before we launch something, we're going to want to run an experiment.
link |
So we get a sense for what the impact is going to be, not just to the viewers, but also to
link |
the different channels and all of that.
link |
An absurd question.
link |
Well, actually, it's interesting.
link |
Maybe there's an answer.
link |
But if I want to make a viral video, how do I do it?
link |
I don't know how you make a viral video.
link |
I know that we have in the past tried to figure out if we could detect when a video was going
link |
And those were, you take the first and second derivatives of the view count and maybe use
link |
that to do some prediction.
link |
But I can't say we ever got very good at that.
link |
Oftentimes we look at where the traffic was coming from.
link |
If a lot of the viewership is coming from something like Twitter, then maybe it has
link |
a higher chance of becoming viral than if it were coming from search or something.
link |
But that was just trying to detect a video that might be viral.
link |
How to make one, I have no idea.
link |
You get your kids to interrupt you while you're on the news or something.
link |
But after the fact, on one individual video, sort of ahead of time predicting is a really
link |
But after the video went viral, in analysis, can you sometimes understand why it went viral?
link |
From the perspective of YouTube broadly, first of all, is it even interesting for YouTube
link |
that a particular video is viral or does that not matter for the individual, for the experience
link |
Well, I think people expect that if a video is going viral and it's something they would
link |
be interested in, then I think they would expect YouTube to recommend it to them.
link |
So if something's going viral, it's good to just let the wave, let people ride the wave
link |
Well, I mean, we want to meet people's expectations in that way, of course.
link |
So like I mentioned, I hung out with Derek Mueller a while ago, a couple of months back.
link |
He's actually the person who suggested I talk to you on this podcast.
link |
Well, thank you, Derek.
link |
At that time, he just recently posted an awesome science video titled, why are 96 million black
link |
balls on this reservoir?
link |
And in a matter of, I don't know how long, but like a few days, he got 38 million views
link |
and it's still growing.
link |
Is this something you can analyze and understand why it happened, this video and you want a
link |
particular video like it?
link |
I mean, we can surely see where it was recommended, where it was found, who watched it and those
link |
So it's actually, sorry to interrupt, it is the video which helped me discover who Derek
link |
I didn't know who he is before.
link |
So I remember, you know, usually I just have all of these technical, boring MIT Stanford
link |
talks in my recommendation because that's how I watch.
link |
And then all of a sudden there's this black balls and reservoir video with like an excited
link |
nerd with like just, why is this being recommended to me?
link |
So I clicked on it and watched the whole thing and it was awesome.
link |
And then a lot of people had that experience, like why was I recommended this?
link |
But they all of course watched it and enjoyed it, which is, what's your sense of this just
link |
wave of recommendation that comes with this viral video that ultimately people get enjoy
link |
after they click on it?
link |
Well, I think it's the system, you know, basically doing what anybody who's recommending something
link |
would do, which is you show it to some people and if they like it, you say, okay, well,
link |
can I find some more people who are a little bit like them?
link |
Okay, I'm going to try it with them.
link |
Oh, they like it too.
link |
Let me expand the circle some more, find some more people.
link |
Oh, it turns out they like it too.
link |
And you just keep going until you get some feedback that says that, no, now you've gone
link |
These people don't like it anymore.
link |
And so I think that's basically what happened.
link |
And you asked me about how to make a video go viral or make a viral video.
link |
I don't think that if you or I decided to make a video about 96 million balls that it
link |
would also go viral.
link |
It's possible that Derek made like the canonical video about those black balls in the lake.
link |
And I don't know whether or not just following along is the secret.
link |
But it's fascinating.
link |
I mean, just like you said, the algorithm sort of expanding that circle and then figuring
link |
out that more and more people did enjoy it and that sort of phase shift of just a huge
link |
number of people enjoying it and the algorithm quickly, automatically, I assume, figuring
link |
I don't know, the dynamics of psychology of that is a beautiful thing.
link |
So what do you think about the idea of clipping?
link |
Too many people annoyed me into doing it, which is they were requesting it.
link |
They said it would be very beneficial to add clips in like the coolest points and actually
link |
have explicit videos.
link |
Like I'm re uploading a video, like a short clip, which is what the podcasts are doing.
link |
Do you see as opposed to, like I also add timestamps for the topics, do you want the
link |
Do you see YouTube somehow helping creators with that process or helping connect clips
link |
to the original videos or is that just on a long list of amazing features to work towards?
link |
I mean, it's not something that I think we've done yet, but I can tell you that I think
link |
clipping is great and I think it's actually great for you as a creator.
link |
And here's the reason.
link |
If you think about, I mean, let's say the NBA is uploading videos of its games.
link |
Well, people might search for warriors versus rockets or they might search for Steph Curry.
link |
And so a highlight from the game in which Steph Curry makes an amazing shot is an opportunity
link |
for someone to find a portion of that video.
link |
And so I think that you never know how people are going to search for something that you've
link |
And so you want to, I would say you want to make clips and add titles and things like
link |
that so that they can find it as easily as possible.
link |
Do you have a dream of a future, perhaps a distant future when the YouTube algorithm
link |
Sort of automatically detects the parts of the video that are really interesting, exciting,
link |
potentially exciting for people and sort of clip them out in this incredibly rich space.
link |
Cause if you talk about, if you talk, even just this conversation, we probably covered
link |
30, 40 little topics and there's a huge space of users that would find, you know, 30% of
link |
those topics really interesting.
link |
And that space is very different.
link |
It's something that's beyond my ability to clip out, right?
link |
But the algorithm might be able to figure all that out, sort of expand into clips.
link |
Do you have a, do you think about this kind of thing?
link |
Do you have a hope or dream that one day the algorithm will be able to do that kind of
link |
deep content analysis?
link |
Well, we've actually had projects that attempt to achieve this, but it really does depend
link |
on understanding the video well and our understanding of the video right now is quite crude.
link |
And so I think it would be especially hard to do it with a conversation like this.
link |
One might be able to do it with, let's say a soccer match more easily, right?
link |
You could probably find out where the goals were scored.
link |
And then of course you, you need to figure out who it was that scored the goal and, and
link |
that might require a human to do some annotation.
link |
But I think that trying to identify coherent topics in a transcript, like, like the one
link |
of our conversation is, is not something that we're going to be very good at right away.
link |
And I was speaking more to the general problem actually of being able to do both a soccer
link |
match and our conversation without explicit sort of almost my, my hope was that there
link |
exists an algorithm that's able to find exciting things in video.
link |
So Google now on Google search will help you find the segment of the video that you're
link |
So if you search for something like how to change the filter in my dishwasher, then if
link |
there's a long video about your dishwasher and this is the part where the person shows
link |
you how to change the filter, then, then it will highlight that area.
link |
And provide a link directly to it.
link |
And do you know if, from your recollection, do you know if the thumbnail reflects, like,
link |
what's the difference between showing the full video and the shorter clip?
link |
Do you know how it's presented in search results?
link |
I don't remember how it's presented.
link |
And the other thing I would say is that right now it's based on creator annotations.
link |
So it's not the thing we're talking about.
link |
But folks are working on the more automatic version.
link |
It's interesting, people might not imagine this, but a lot of our systems start by using
link |
almost entirely the audience behavior.
link |
And then as they get better, the refinement comes from using the content.
link |
And I wish, I know there's privacy concerns, but I wish YouTube explored the space, which
link |
is sort of putting a camera on the users if they allowed it, right, to study their, like,
link |
I did a lot of emotion recognition work and so on, to study actual sort of richer signal.
link |
One of the cool things when you upload 360 like VR video to YouTube, and I've done this
link |
a few times, so I've uploaded myself, it's a horrible idea.
link |
Some people enjoyed it, but whatever.
link |
The video of me giving a lecture in 360 with a 360 camera, and it's cool because YouTube
link |
allows you to then watch where did people look at?
link |
There's a heat map of where, you know, of where the center of the VR experience was.
link |
And it's interesting because that reveals to you, like, what people looked at.
link |
It's not always what you were expecting.
link |
In the case of the lecture, it's pretty boring, it is what we were expecting, but we did a
link |
few funny videos where there's a bunch of people doing things, and everybody tracks
link |
You know, in the beginning, they all look at the main person and they start spreading
link |
around and looking at the other people.
link |
So that kind of, that's a really strong signal of what people found exciting in the video.
link |
I don't know how you get that from people just watching, except they tuned out at this
link |
Like, it's hard to measure this moment was super exciting for people.
link |
I don't know how you get that signal.
link |
Maybe comment, is there a way to get that signal where this was like, this is when their
link |
eyes opened up and they're like, like for me with the Ray Dalio video, right?
link |
Like at first I was like, okay, this is another one of these like dumb it down for you videos.
link |
And then you like start watching, it's like, okay, there's really crisp, clean, deep explanation
link |
of how the economy works.
link |
That's where I like set up and started watching, right?
link |
That moment, is there a way to detect that moment?
link |
The only way I can think of is by asking people to label it.
link |
You mentioned that we're quite far away in terms of doing video analysis, deep video
link |
Of course, Google, YouTube, you know, we're quite far away from solving autonomous driving
link |
I think we're closer to that.
link |
Well, the, you know, you never know.
link |
And the Wright brothers thought they're never, they're not going to fly for 50 years, three
link |
years before they flew.
link |
So what are the biggest challenges would you say?
link |
Is it the broad challenge of understanding video, understanding natural language, understanding
link |
the challenge before the entire machine learning community or just being able to understand
link |
Is there something specific to video that's even more challenging than understanding natural
link |
language understanding?
link |
What's your sense of what the biggest challenge is?
link |
Video is just so much information.
link |
And so precision becomes a real problem.
link |
It's like, you know, you're trying to classify something and you've got a million classes
link |
and the distinctions among them, at least from a machine learning perspective are often
link |
pretty small, right?
link |
Like, you know, you need to see this person's number in order to know which player it is.
link |
And there's a lot of players or you need to see, you know, the logo on their chest in
link |
order to know like which team they play for.
link |
And so, and that's just figuring out who's who, right?
link |
And then you go further and saying, okay, well, you know, was that a goal?
link |
Was it not a goal?
link |
Like, is that an interesting moment as you said, or is that not an interesting moment?
link |
These things can be pretty hard.
link |
So Yann LeCun, I'm not sure if you're familiar sort of with his current thinking and work.
link |
So he believes that self, what he's referring to as self supervised learning will be the
link |
solution sort of to achieving this kind of greater level of intelligence.
link |
In fact, the thing he's focusing on is watching video and predicting the next frame.
link |
So predicting the future of video, right?
link |
So for now we're very far from that, but his thought is because it's unsupervised or as
link |
he refers to as self supervised, you know, if you watch enough video, essentially if
link |
you watch YouTube, you'll be able to learn about the nature of reality, the physics,
link |
the common sense reasoning required by just teaching a system to predict the next frame.
link |
So he's confident this is the way to go.
link |
So for you, from the perspective of just working with this video, how do you think an algorithm
link |
that just watches all of YouTube, stays up all day and night watching YouTube would be
link |
able to understand enough of the physics of the world about the way this world works,
link |
be able to do common sense reasoning and so on?
link |
Well, I mean, we have systems that already watch all the videos on YouTube, right?
link |
But they're just looking for very specific things, right?
link |
They're supervised learning systems that are trying to identify something or classify something.
link |
And I don't know if, I don't know if predicting the next frame is really going to get there
link |
because I'm not an expert on compression algorithms, but I understand that that's kind of what
link |
compression video compression algorithms do is they basically try to predict the next
link |
frame and then fix up the places where they got it wrong.
link |
And that leads to higher compression than if you actually put all the bits for the next
link |
So I don't know if I believe that just being able to predict the next frame is going to
link |
be enough because there's so many frames and even a tiny bit of error on a per frame basis
link |
can lead to wildly different videos.
link |
So the thing is, the idea of compression is one way to do compression is to describe through
link |
text what's contained in the video.
link |
That's the ultimate high level of compression.
link |
So the idea is traditionally when you think of video image compression, you're trying
link |
to maintain the same visual quality while reducing the size.
link |
But if you think of deep learning from a bigger perspective of what compression is, is you're
link |
trying to summarize the video.
link |
And the idea there is if you have a big enough neural network, just by watching the next,
link |
trying to predict the next frame, you'll be able to form a compression of actually understanding
link |
what's going on in the scene.
link |
If there's two people talking, you can just reduce that entire video into the fact that
link |
two people are talking and maybe the content of what they're saying and so on.
link |
That's kind of the open ended dream.
link |
So I just wanted to sort of express that because it's interesting, compelling notion, but it
link |
is nevertheless true that video, our world is a lot more complicated than we get a credit
link |
I mean, in terms of search and discovery, we have been working on trying to summarize
link |
videos in text or with some kind of labels for eight years at least.
link |
And you know, and we're kind of so, so.
link |
So if you were to say the problem is a hundred percent solved and eight years ago was zero
link |
percent solved, where are we on that timeline would you say?
link |
To summarize a video well, maybe less than a quarter of the way.
link |
So on that topic, what does YouTube look like 10, 20, 30 years from now?
link |
I mean, I think that YouTube is evolving to take the place of TV.
link |
I grew up as a kid in the seventies and I watched a tremendous amount of television
link |
and I feel sorry for my poor mom because people told her at the time that it was going to
link |
rot my brain and that she should kill her television.
link |
But anyway, I mean, I think that YouTube is at least for my family, a better version of
link |
television, right?
link |
It's one that is on demand.
link |
It's more tailored to the things that my kids want to watch.
link |
And also they can find things that they would never have found on television.
link |
And so I think that at least from just observing my own family, that's where we're headed is
link |
that people watch YouTube kind of in the same way that I watched television when I was younger.
link |
So from a search and discovery perspective, what do you, what are you excited about in
link |
the five, 10, 20, 30 years?
link |
Like what kind of things?
link |
It's already really good.
link |
I think it's achieved a lot of, of course we don't know what's possible.
link |
So it's the task of search of typing in the text or discovering new videos by the next
link |
So I personally am really happy with the experience.
link |
I continuously, I rarely watch a video that's not awesome from my own perspective, but what's,
link |
what else is possible?
link |
What are you excited about?
link |
Well, I think introducing people to more of what's available on YouTube is not only very
link |
important to YouTube and to creators, but I think it will help enrich people's lives
link |
because there's a lot that I'm still finding out is available on YouTube that I didn't
link |
I've been working YouTube eight years and it wasn't until last year that I learned that,
link |
that I could watch USC football games from the 1970s.
link |
Like I didn't even know that was possible until last year and I've been working here
link |
So, you know, what was broken about, about that?
link |
That it took me seven years to learn that this stuff was already on YouTube even when
link |
So I think there's a big opportunity there.
link |
And then as I said before, you know, we want to make sure that YouTube finds a way to ensure
link |
that it's acting responsibly with respect to society and enriching people's lives.
link |
So we want to take all of the great things that it does and make sure that we are eliminating
link |
the negative consequences that might happen.
link |
And then lastly, if we could get to a point where all the videos people watch are the
link |
best ones they've ever watched, that'd be outstanding too.
link |
Do you see in many senses becoming a window into the world for people?
link |
It's especially with live video, you get to watch events.
link |
I mean, it's really, it's the way you experience a lot of the world that's out there is better
link |
than TV in many, many ways.
link |
So do you see becoming more than just video?
link |
Do you see creators creating visual experiences and virtual worlds that if I'm, I'm talking
link |
crazy now, but sort of virtual reality and entering that space, or is that at least for
link |
now totally outside what YouTube is thinking about?
link |
I mean, I think Google is thinking about virtual reality.
link |
I don't think about virtual reality too much.
link |
I know that we would want to make sure that YouTube is there when virtual reality becomes
link |
something or if virtual reality becomes something that a lot of people are interested in.
link |
But I haven't seen it really take off yet.
link |
Well, the future is wide open.
link |
Christos, I've been really looking forward to this conversation.
link |
It's been a huge honor.
link |
Thank you for answering some of the more difficult questions I've asked.
link |
I'm really excited about what YouTube has in store for us.
link |
It's one of the greatest products I've ever used and continues.
link |
So thank you so much for talking to me.
link |
Thanks for asking me.
link |
Thanks for listening to this conversation.
link |
And thank you to our presenting sponsor, Cash App.
link |
Use code LexPodcast.
link |
You'll get $10 and $10 will go to FIRST, a STEM education nonprofit that inspires hundreds
link |
of thousands of young minds to become future leaders and innovators.
link |
If you enjoy this podcast, subscribe on YouTube, give it five stars on Apple Podcast, follow
link |
on Spotify, support on Patreon, or simply connect with me on Twitter.
link |
And now, let me leave you with some words of wisdom from Marcel Proust.
link |
The real voyage of discovery consists not in seeking new landscapes, but in having new
link |
Thank you for listening and hope to see you next time.