back to indexPeter Norvig: Artificial Intelligence: A Modern Approach | Lex Fridman Podcast #42
link |
The following is a conversation with Peter Norvig.
link |
He's the director of research at Google and the coauthor
link |
with Stuart Russell of the book, Artificial Intelligence
link |
and Modern Approach, that educated and inspired
link |
a whole generation of researchers, including myself,
link |
to get into the field of artificial intelligence.
link |
This is the Artificial Intelligence Podcast.
link |
If you enjoy it, subscribe on YouTube,
link |
give it five stars on iTunes, support on Patreon,
link |
or simply connect with me on Twitter.
link |
Alex Friedman, spelled F R I D M A N.
link |
And now, here's my conversation with Peter Norvig.
link |
Most researchers in the AI community, including myself,
link |
own all three editions, red, green, and blue,
link |
of the Artificial Intelligence and Modern Approach.
link |
It's a field defining textbook as many people are aware
link |
that you wrote with Stuart Russell.
link |
How has the book changed and how have you changed
link |
in relation to it from the first edition to the second
link |
to the third and now fourth edition as you work on it?
link |
Yeah, so it's been a lot of years, a lot of changes.
link |
One of the things changing from the first to maybe the second
link |
or third was just the rise of computing power, right?
link |
So I think in the first edition, we said,
link |
here's predicate logic, but that only goes so far
link |
because pretty soon you have millions of short little
link |
predicate expressions and they couldn't possibly
link |
fit in memory, so we're gonna use first order logic
link |
that's more concise.
link |
And then we quickly realized, oh, predicate logic
link |
is pretty nice because there are really fast
link |
SAT solvers and other things and look,
link |
there's only millions of expressions
link |
and that fits easily into memory
link |
or maybe even billions fit into memory now.
link |
So that was a change of the type of technology we needed
link |
just because the hardware expanded.
link |
Even to the second edition, the resource constraints
link |
were loosened significantly for the second.
link |
And that was the early 2000s second edition.
link |
Right, so 95 was the first and then 2000, 2001 or so.
link |
And then moving on from there,
link |
I think we're starting to see that again with the GPUs
link |
and then more specific type of machinery like the TPUs
link |
and we're seeing custom ASICs and so on for deep learning.
link |
So we're seeing another advance in terms of the hardware.
link |
Then I think another thing that we especially noticed
link |
this time around is in all three of the first editions,
link |
we kind of said, well, we're gonna find AI
link |
as maximizing expected utility
link |
and you tell me your utility function
link |
and now we've got 27 chapters
link |
with the cool techniques for how to optimize that.
link |
I think in this edition, we're saying more, you know what?
link |
Maybe that optimization part is the easy part
link |
and the hard part is deciding what is my utility function?
link |
And if I'm a collection of agents or a society,
link |
what do we want as a whole?
link |
So you touch that topic in this edition,
link |
you get a little bit more into utility.
link |
That's really interesting.
link |
On a technical level, we're almost pushing the philosophical.
link |
I guess it is philosophical, right?
link |
So we've always had a philosophy chapter,
link |
which I was glad that we were supporting.
link |
And now it's less kind of the Chinese room type argument
link |
and more of these ethical and societal type issues.
link |
So we get into the issues of fairness and bias
link |
and just the issue of aggregating utilities.
link |
So how do you encode human values into a utility function?
link |
Is this something that you can do purely through data
link |
in a learned way or is there some systematic?
link |
Obviously, there's no good answers yet.
link |
There's just beginnings to this,
link |
to even opening the door to these questions.
link |
So there is no one answer.
link |
Yes, there are techniques to try to learn that.
link |
So we talk about inverse reinforcement learning, right?
link |
So reinforcement learning, you take some actions,
link |
you get some rewards,
link |
and you figure out what actions you should take.
link |
And inverse reinforcement learning,
link |
you observe somebody taking actions
link |
and you figure out,
link |
well, this must be what they were trying to do.
link |
If they did this action, it must be because they wanted it.
link |
Of course, there's restrictions to that, right?
link |
So lots of people take actions that are self destructive
link |
or they're suboptimal in a certain way.
link |
So you don't want to learn that.
link |
You want to somehow learn the perfect actions
link |
rather than the ones they actually take.
link |
So that's a challenge for that field.
link |
Then another big part of it is just kind of theoretical
link |
of saying what can we accomplish?
link |
And so you look at like this work on the programs
link |
to predict recidivism and decide who should get parole
link |
or who should get bail or whatever.
link |
And how are you gonna evaluate that?
link |
And one of the big issues is fairness
link |
across protected classes,
link |
protected classes being things like sex and race and so on.
link |
And so two things you want is you want to say,
link |
well, if I get a score of say a six out of 10,
link |
then I want that to mean the same,
link |
whether no matter what race I'm on, right?
link |
So I want to have a 60% chance of reoccurring regardless.
link |
Regardless, and one of the makers of a commercial program
link |
to do that says, that's what we're trying to optimize.
link |
And look, we achieved that.
link |
We've reached that kind of balance.
link |
And then on the other side, you also want to say,
link |
well, if it makes mistakes,
link |
I want that to affect both sides of the protected class
link |
equally and it turns out they don't do that, right?
link |
So they're twice as likely to make a mistake
link |
that would harm a black person over a white person.
link |
So that seems unfair.
link |
So you'd like to say, well,
link |
I want to achieve both those goals.
link |
And then turns out you do the analysis
link |
and it's theoretically impossible to achieve both those goals.
link |
So you have to trade them off one against the other.
link |
So that analysis is really helpful
link |
to know what you can aim for and how much you can get,
link |
that you can't have everything.
link |
But the analysis certainly can't tell you
link |
where should we make that trade off point.
link |
But nevertheless, then we can, as humans, deliberate
link |
where that trade off should be.
link |
Yeah, so at least now we're arguing in an informed way.
link |
We're not asking for something impossible.
link |
We're saying, here's where we are
link |
and here's what we aim for.
link |
And this strategy is better than that strategy.
link |
So that's, I would argue,
link |
is a really powerful and really important first step.
link |
But it's a doable one,
link |
sort of removing undesirable degrees of bias in systems
link |
in terms of protective classes.
link |
And then there's something,
link |
I listened to your commencement speech,
link |
or there's some fuzzier things like,
link |
you mentioned angry birds.
link |
Do you want to create systems that feed the dopamine
link |
enjoyment, that feed, that optimize for you
link |
returning to the system,
link |
enjoying the moment of playing the game,
link |
of getting likes or whatever this kind of thing,
link |
or some kind of long term improvement.
link |
Right. Are you even thinking about that?
link |
That's really going to the philosophical area.
link |
I think that's a really important issue too,
link |
certainly thinking about that.
link |
I don't think about that as an AI issue as much.
link |
But as you say, the point is we've built this society
link |
in this infrastructure where we say
link |
we have a marketplace for attention
link |
and we've decided as a society
link |
that we like things that are free.
link |
And so we want all apps on our phone to be free.
link |
And that means they're all competing for your attention
link |
and then eventually they make some money some way
link |
through ads or in game sales or whatever.
link |
But they can only win by defeating all the other apps
link |
by instilling your attention.
link |
And we build a marketplace where it seems
link |
like they're working against you rather than working with you.
link |
And I'd like to find a way where we can change
link |
the playing field so we feel more like,
link |
well, these things are on my side.
link |
Yes, they're letting me have some fun in the short term,
link |
but they're also helping me in the long term
link |
rather than competing against me.
link |
And those aren't necessarily conflicting objectives.
link |
They're just the incentives, the direct current incentives
link |
as we try to figure out this whole new world
link |
that seemed to be on the easier part of that,
link |
which is feeding the dopamine, the rush.
link |
But let me be taking a quick step back
link |
at the beginning of the artificial intelligence
link |
and modern approach book of writing.
link |
So here you are in the 90s,
link |
when you first sat down with Stuart to write the book
link |
to cover an entire field,
link |
which is one of the only books that's successfully done
link |
at F4AI and actually in a lot of other computer science
link |
fields, it's a huge undertaking.
link |
So it must have been quite daunting.
link |
What was that process like?
link |
Did you envision that you would be trying to cover
link |
Was there a systematic approach to it
link |
that was more step by step?
link |
So I guess it came about,
link |
go to lunch with the other AI faculty at Berkeley
link |
and we'd say the field is changing,
link |
seems like the current books are a little bit behind,
link |
nobody's come out with a new book recently,
link |
we should do that.
link |
And everybody said, yeah, yeah,
link |
that's a great thing to do.
link |
And we never did anything.
link |
And then I ended up heading off to industry.
link |
I went to Sun Labs.
link |
So I thought, well, that's the end
link |
of my possible academic publishing career.
link |
But I met Stuart again at a conference like a year later
link |
and said, you know, that book we were always talking about,
link |
you guys must be half done with it by now, right?
link |
And he said, well, we keep talking, we never do anything.
link |
So I said, well, you know, we should do it.
link |
And I think the reason is that we all felt
link |
it was a time where the field was changing.
link |
And that was in two ways.
link |
So, you know, the good old fashioned AI
link |
was based primarily on Boolean logic.
link |
And you had a few tricks to deal with uncertainty.
link |
And it was based primarily on knowledge engineering,
link |
that the way you got something done is you went out
link |
and you interviewed an expert
link |
and you wrote down by hand everything they knew.
link |
And we saw in 95 that the field was changing in two ways.
link |
One, we were moving more towards probability
link |
rather than Boolean logic.
link |
And we were moving more towards machine learning
link |
rather than knowledge engineering.
link |
And the other books hadn't caught that way
link |
if they were still in the, more in the old school,
link |
although certainly they had part of that on the way.
link |
But we said, if we start now completely taking
link |
that point of view, we can have a different kind of book
link |
and we were able to put that together.
link |
And what was literally the process, if you remember?
link |
Did you start writing a chapter?
link |
Yeah, I guess we did an outline
link |
and then we sort of assigned chapters to each person.
link |
At the time, I had moved to Boston
link |
and Stuart was in Berkeley.
link |
So basically we did it over the internet.
link |
And that wasn't the same as doing it today.
link |
It meant dial up lines and telnetting in.
link |
You telnetted into one shell
link |
and you type cat file name
link |
and you hoped it was captured at the other end.
link |
And certainly you're not sending images
link |
and figures back and forth.
link |
Right, right, that didn't work.
link |
But did you anticipate where the field would go
link |
from that day, from the 90s?
link |
Did you see the growth into learning based methods
link |
and to data driven methods
link |
that followed in the future decades?
link |
We certainly thought that learning was important.
link |
I guess we missed it as being as important as it is today.
link |
We missed this idea of big data.
link |
We missed that the idea of deep learning
link |
hadn't been invented yet.
link |
We could have taken the book
link |
from a complete machine learning point of view
link |
right from the start.
link |
We chose to do it more from a point of view
link |
of we're gonna first develop the different types
link |
of representations and we're gonna talk
link |
about different types of environments.
link |
Is it fully observable or partially observable
link |
and is it deterministic or stochastic and so on?
link |
And we made it more complex along those axes
link |
rather than focusing on the machine learning axis first.
link |
Do you think, there's some sense in which
link |
the deep learning craze is extremely successful
link |
for a particular set of problems?
link |
And eventually it's going to, in the general case,
link |
So in terms of the difference between perception systems
link |
and robots that have to act in the world,
link |
do you think we're gonna return to AI,
link |
modern approach type breadth in addition five and six
link |
in future decades?
link |
Do you think deep learning will take its place
link |
as a chapter in this bigger view of AI?
link |
Yeah, I think we don't know yet
link |
how it's all gonna play out.
link |
So in the new edition, we have a chapter on deep learning.
link |
We got Ian Goodfellow to be the guest author
link |
for that chapter, so he said he could condense
link |
his whole deep learning book into one chapter.
link |
I think he did a great job.
link |
We were also encouraged that we gave him
link |
the old neural net chapter and said,
link |
I had fun with it.
link |
Modernize that, and he said, you know,
link |
half of that was okay.
link |
That certainly there's lots of new things
link |
that have been developed, but some of the core
link |
was still the same.
link |
So I think we'll gain a better understanding
link |
of what you can do there.
link |
I think we'll need to incorporate all the things
link |
we can do with the other technologies, right?
link |
So deep learning started out,
link |
convolutional networks, and very close to perception.
link |
And it's since moved to be able to do more
link |
with actions and some degree of longer term planning.
link |
But we need to do a better job
link |
with representation and reasoning
link |
and one shot learning and so on.
link |
And I think we don't know yet
link |
how that's gonna play out.
link |
So do you think looking at some success,
link |
but certainly eventual demise,
link |
a partial demise of experts
link |
to symbolic systems in the 80s,
link |
do you think there is kernels of wisdom
link |
in the work that was done there
link |
with logic and reasoning and so on
link |
that will rise again in your view?
link |
So certainly I think the idea of representation
link |
and reasoning is crucial,
link |
that sometimes you just don't have enough data
link |
about the world to learn de novo.
link |
So you've got to have some idea of representation,
link |
whether that was programmed in or told or whatever,
link |
and then be able to take steps of reasoning.
link |
I think the problem with the good old fashioned AI
link |
was one, we tried to base everything on these symbols
link |
that were atomic and that's great
link |
if you're like trying to define the properties of a triangle.
link |
Because they have necessary and sufficient conditions.
link |
But things in the real world don't.
link |
The real world is messy and doesn't have sharp edges
link |
and atomic symbols do.
link |
So that was a poor match.
link |
And then the other aspect was that the reasoning
link |
was universal and applied anywhere,
link |
which in some sense is good,
link |
but it also means there's no guidance
link |
as to where to apply.
link |
And so you started getting these paradoxes,
link |
like well, if I have a mountain
link |
and I remove one grain of sand,
link |
then it's still a mountain.
link |
But if I do that repeatedly at some point, it's not.
link |
And with logic, there's nothing to stop you
link |
from applying things repeatedly.
link |
But maybe with something like deep learning,
link |
and I don't really know what the right name for it is,
link |
we could separate out those ideas.
link |
So one, we could say a mountain isn't just an atomic notion.
link |
It's some sort of something like a word embedding
link |
that has a more complex representation.
link |
And secondly, we could somehow learn,
link |
yeah, there's this rule that you can remove one grain of sand.
link |
You can do that a bunch of times,
link |
but you can't do it a near infinite amount of times.
link |
But on the other hand,
link |
when you're doing induction on the integer, sure,
link |
then it's fine to do it an infinite number of times.
link |
And if we could, somehow we have to learn
link |
when these strategies are applicable,
link |
rather than having the strategies be completely neutral
link |
and available everywhere.
link |
Anytime you use neural networks,
link |
anytime you learn from data,
link |
form representation from data in an automated way,
link |
it's not very explainable as to,
link |
or it's not introspective to us humans
link |
in terms of how this neural network sees the world,
link |
where why does it succeed so brilliantly in so many cases
link |
and fail so miserably in surprising ways and small.
link |
So what do you think is the future there?
link |
Can simply more data, better data,
link |
more organized data solve that problem?
link |
Or is there elements of symbolic systems
link |
that need to be brought in,
link |
which are a little bit more explainable?
link |
Yeah, so I prefer to talk about trust
link |
and validation and verification
link |
rather than just about explainability.
link |
And then I think explanations are one tool
link |
that you use towards those goals.
link |
And I think it is important issue
link |
that we don't wanna use these systems
link |
unless we trust them
link |
and we wanna understand where they work
link |
and where they don't work.
link |
And an explanation can be part of that, right?
link |
So I apply for a loan and I get denied,
link |
I want some explanation of why.
link |
And you have in Europe,
link |
we have the GDPR that says you're required
link |
to be able to get that.
link |
But on the other hand,
link |
an explanation alone is not enough, right?
link |
So we are used to dealing with people
link |
and with organizations and corporations and so on
link |
and they can give you an explanation
link |
and you have no guarantee
link |
that that explanation relates to reality, right?
link |
So the bank can tell me,
link |
well, you didn't get the loan
link |
because you didn't have enough collateral
link |
and that may be true or it may be true
link |
that they just didn't like my religion or something else.
link |
I can't tell from the explanation.
link |
And that's true whether the decision was made
link |
by a computer or by a person.
link |
I do wanna have the explanations
link |
and I wanna be able to have a conversation
link |
to go back and forth and said,
link |
well, you gave this explanation, but what about this?
link |
And what would have happened if this had happened?
link |
And what would I need to change that?
link |
So I think a conversation is a better way to think about it
link |
than just an explanation as a single output.
link |
And I think we need testing of various kinds, right?
link |
So in order to know,
link |
was the decision really based on my collateral
link |
or was it based on my religion or skin color or whatever?
link |
I can't tell if I'm only looking at my case,
link |
but if I look across all the cases,
link |
then I can detect a pattern, right?
link |
So you wanna have that kind of capability.
link |
You wanna have these adversarial testing, right?
link |
So we thought we were doing pretty good
link |
at object recognition and images.
link |
We said, look, we're at sort of pretty close
link |
to human level performance on ImageNet and so on.
link |
And then you start seeing these adversarial images
link |
and you say, wait a minute,
link |
that part is nothing like human performance.
link |
Hey, you can mess with it really easily.
link |
You can mess with it really easily, right?
link |
And yeah, you can do that to humans too, right?
link |
In a different way, perhaps.
link |
Right, humans don't know what color the dress was.
link |
And so they're vulnerable to certain attacks
link |
that are different than the attacks on the machines,
link |
but the attacks on the machines are so striking.
link |
They really change the way you think about what we've done.
link |
And the way I think about it is,
link |
I think part of the problem is we're seduced
link |
by our low dimensional metaphors, right?
link |
Yeah, I like that phrase.
link |
You look in a textbook and you say,
link |
okay, now we've mapped out the space
link |
and cat is here and dog is here
link |
and maybe there's a tiny little spot in the middle
link |
where you can't tell the difference,
link |
but mostly we've got it all covered.
link |
And if you believe that metaphor,
link |
then you say, well, we're nearly there.
link |
And there's only gonna be a couple adversarial images,
link |
but I think that's the wrong metaphor
link |
and what you should really say is,
link |
it's not a 2D flat space that we've got mostly covered.
link |
It's a million dimension space
link |
and cat is this string that goes out in this crazy bath
link |
and if you step a little bit off the path in any direction,
link |
you're in nowhere's land
link |
and you don't know what's gonna happen.
link |
And so I think that's where we are
link |
and now we've got to deal with that.
link |
So it wasn't so much an explanation,
link |
but it was an understanding of what the models are
link |
and what they're doing
link |
and now we can start exploring how do you fix that?
link |
Yeah, validating the robustness of the system so on,
link |
but take it back to this word trust.
link |
Do you think we're a little too hard on our robots
link |
in terms of the standards we apply?
link |
So, you know, there's a dance.
link |
There's a dance and nonverbal
link |
and verbal communication between humans.
link |
You know, if we apply the same kind of standard
link |
in terms of humans, you know,
link |
we trust each other pretty quickly.
link |
You and I have met before
link |
and there's some degree of trust, right?
link |
That nothing's gonna go crazy wrong
link |
and yet to AI, when we look at AI systems
link |
where we seem to approach the skepticism always, always
link |
and it's like they have to prove through a lot of hard work
link |
that they're even worthy of even inkling of our trust.
link |
What do you think about that?
link |
How do we break that barrier, close that gap?
link |
I think that's right.
link |
I think that's a big issue.
link |
Just listening, my friend Mark Moffat is a naturalist
link |
and he says, the most amazing thing about humans
link |
is that you can walk into a coffee shop
link |
or a busy street in a city
link |
and there's lots of people around you
link |
that you've never met before
link |
and you don't kill each other.
link |
He says chimpanzees cannot do that.
link |
If a chimpanzee's in a situation where here's some
link |
that aren't from my tribe, bad things happen.
link |
Especially in a coffee shop, there's delicious food around,
link |
Yeah, yeah, but we humans have figured that out, right?
link |
And, you know, for the most part, we still go to war,
link |
we still do terrible things,
link |
but for the most part,
link |
we've learned to trust each other and live together.
link |
So that's gonna be important for our AI systems as well.
link |
And also, I think, you know, a lot of the emphasis is on AI,
link |
but in many cases, AI is part of the technology
link |
but isn't really the main thing.
link |
So a lot of what we've seen is more due
link |
to communications technology than AI technology.
link |
Yeah, you wanna make these good decisions,
link |
but the reason we're able to have any kind of system at all
link |
is we've got the communication
link |
so that we're collecting the data
link |
and so that we can reach lots of people around the world.
link |
I think that's a bigger change that we're dealing with.
link |
Speaking of reaching a lot of people around the world,
link |
on the side of education,
link |
you've, one of the many things in terms of education
link |
you've done, you've taught the Intro to Artificial
link |
Intelligence course that signed up 160,000 students.
link |
It was one of the first successful examples
link |
of a MOOC, massive open online course.
link |
What did you learn from that experience?
link |
What do you think is the future of MOOCs, of education online?
link |
Yeah, it was a great fun doing it,
link |
particularly being right at the start,
link |
just because it was exciting and new,
link |
but it also meant that we had less competition, right?
link |
So one of the things you hear about,
link |
well, the problem with MOOCs is the completion rates
link |
are so low, so there must be a failure.
link |
And I gotta admit, I'm a prime contributor, right?
link |
I've probably started 50 different courses
link |
that I haven't finished,
link |
but I got exactly what I wanted out of them
link |
because I had never intended to finish them.
link |
I just wanted to dabble in a little bit,
link |
either to see the topic matter
link |
or just to see the pedagogy of how are they doing this class.
link |
So I guess the main thing I learned is when I came in,
link |
I thought the challenge was information,
link |
saying, if I'm just take the stuff I want you to know,
link |
and I'm very clear and explain it well,
link |
then my job is done and good things are gonna happen.
link |
And then in doing the course,
link |
I learned, well, yeah, you gotta have the information,
link |
but really the motivation is the most important thing.
link |
But that if students don't stick with it,
link |
then it doesn't matter how good the content is.
link |
And I think being one of the first classes,
link |
we were helped by sort of exterior motivation.
link |
So we tried to do a good job of making it enticing
link |
and setting up ways for the community
link |
to work with each other to make it more motivating.
link |
But really a lot of it was, hey, this is a new thing
link |
and I'm really excited to be part of a new thing.
link |
And so the students brought their own motivation.
link |
And so I think this is great
link |
because there's lots of people around the world
link |
who have never had this before,
link |
would never have the opportunity to go to Stanford
link |
and take a class or go to MIT
link |
or go to one of the other schools.
link |
But now we can bring that to them.
link |
And if they bring their own motivation,
link |
they can be successful in a way they couldn't before.
link |
But that's really just the top tier of people
link |
that are ready to do that.
link |
The rest of the people just don't see
link |
or don't have their motivation
link |
and don't see how if they push through
link |
and we're able to do it,
link |
what advantage that would get them.
link |
So I think we got a long way to go
link |
before we were able to do that.
link |
And I think it'll be,
link |
some of it is based on technology,
link |
but more of it's based on the idea of community.
link |
You gotta actually get people together.
link |
Some of the getting together can be done online.
link |
I think some of it really has to be done in person
link |
in order to build that type of community and trust.
link |
You know, there's an intentional mechanism
link |
that we've developed a short attention span,
link |
especially younger people,
link |
because sort of short on short of videos online,
link |
there's a whatever the way the brain is developing now
link |
and with people that have grown up with the internet,
link |
they have quite a short attention span.
link |
So, and I would say I had the same
link |
when I was growing up too,
link |
probably for different reasons.
link |
So I probably wouldn't have learned as much as I have
link |
if I wasn't forced to sit in a physical classroom,
link |
sort of bored, sometimes falling asleep,
link |
but sort of forcing myself through that process
link |
to sometimes extremely difficult computer science courses.
link |
What's the difference in your view between in person
link |
education experience, which you first of all,
link |
you yourself had and you yourself taught
link |
and online education?
link |
And how do we close that gap if it's even possible?
link |
Yeah, so I think there's two issues.
link |
One is whether it's in person or online.
link |
So it's sort of the physical location.
link |
And then the other is kind of the affiliation, right?
link |
So you stuck with it in part
link |
because you were in the classroom
link |
and you saw everybody else was suffering
link |
the same way you were,
link |
but also because you were enrolled,
link |
you had paid tuition,
link |
sort of everybody was expecting you to stick with it.
link |
Society, parents, peers, yeah.
link |
And so those are two separate things.
link |
I mean, you could certainly imagine,
link |
I pay a huge amount of tuition
link |
and everybody signed up and says, yes, you're doing this,
link |
but then I'm in my room
link |
and my classmates are in different rooms, right?
link |
We could have things set up that way.
link |
So it's not just the online versus offline.
link |
I think what's more important is the commitment
link |
And certainly it is important
link |
to have that kind of informal,
link |
I meet people outside of class,
link |
we talk together because we're all in it together.
link |
I think that's a really important,
link |
both in keeping your motivation
link |
and also that's where some
link |
of the most important learning goes on.
link |
So you wanna have that.
link |
Maybe, especially now,
link |
we start getting into higher bandwidths
link |
and augmented reality and virtual reality.
link |
You might be able to get that
link |
without being in the same physical place.
link |
Do you think it's possible we'll see a course at Stanford?
link |
For example, that for students,
link |
enrolled students is only online in the near future
link |
who are literally, it's part of the curriculum
link |
and there is no...
link |
Yeah, so you're starting to see that.
link |
I know Georgia Tech has a master's that's done that way.
link |
Oftentimes, it's sort of,
link |
they're creeping in in terms of master's program
link |
or sort of further education,
link |
considering the constraints of students and so on.
link |
But I mean, literally, is it possible
link |
that we just, you know, Stanford, MIT, Berkeley,
link |
all these places go online only in the next few decades?
link |
Yeah, probably not,
link |
because they've got a big commitment to a physical campus.
link |
Sure, so there's a momentum
link |
that's both financial and culturally.
link |
Right, and then there are certain things
link |
that's just hard to do virtually, right?
link |
So, you know, we're in a field where
link |
if you have your own computer and your own paper,
link |
and so on, you can do the work anywhere,
link |
but if you're in a biology lab or something,
link |
you know, you don't have all the right stuff at home.
link |
Right, so our field, programming,
link |
you've also done a lot of,
link |
you've done a lot of programming yourself.
link |
In 2001, you wrote a great article about programming
link |
called Teach Yourself Programming in 10 Years,
link |
sort of response to all the books
link |
that say Teach Yourself Programming in 21 Days.
link |
So if you were giving advice to someone
link |
getting into programming today,
link |
this is a few years since you've written that article,
link |
what's the best way to undertake that journey?
link |
I think there's lots of different ways,
link |
and I think programming means more things now.
link |
And I guess, you know, when I wrote that article,
link |
I was thinking more about becoming
link |
a professional software engineer.
link |
And I thought that's a, you know,
link |
sort of a career long field of study.
link |
But I think there's lots of things now
link |
that people can do where programming is a part
link |
of solving what they want to solve
link |
without achieving that professional level status, right?
link |
So I'm not going to be going
link |
and writing a million lines of code,
link |
but, you know, I'm a biologist or a physicist
link |
or something or even a historian,
link |
and I've got some data and I want to ask a question
link |
And I think for that, you don't need 10 years, right?
link |
So there are many shortcuts to being able
link |
to answer those kinds of questions.
link |
And, you know, you see today a lot of emphasis
link |
on learning to code, teaching kids how to code.
link |
I think that's great,
link |
but I wish they would change the message a little bit, right?
link |
So I think code isn't the main thing.
link |
I don't really care if you know the syntax
link |
of JavaScript or if you can connect these blocks together
link |
in this visual language.
link |
But what I do care about is that you can analyze a problem,
link |
you can think of a solution, you can carry out,
link |
you know, make a model, run that model, test the model,
link |
see the results, verify that they're reasonable,
link |
ask questions and answer them, all right?
link |
So it's more modeling and problem solving
link |
and you use coding in order to do that,
link |
but it's not just learning coding for its own sake.
link |
That's really interesting.
link |
So it's actually almost, in many cases,
link |
it's learning to work with data
link |
to extract something useful out of data.
link |
So when you say problem solving,
link |
you really mean taking some kind of,
link |
maybe collecting some kind of data set, cleaning it up
link |
and saying something interesting about it,
link |
which is useful in all kinds of domains.
link |
And you know, and I see myself being stuck sometimes
link |
in kind of the old ways, right?
link |
So, you know, I'll be working on a project,
link |
maybe with a younger employee and we say,
link |
oh, well here's this new package
link |
that could help solve this problem.
link |
And I'll go and I'll start reading the manuals
link |
and you know, I'll be two hours into reading the manuals
link |
and then my colleague comes back and says, I'm done.
link |
You know, I downloaded the package, I installed it,
link |
I tried calling some things, the first one didn't work,
link |
the second one didn't work, now I'm done.
link |
And I say, but I have under questions about
link |
how does this work and how does that work?
link |
And they say, who cares, right?
link |
I don't need to understand the whole thing.
link |
I answered my question, it's a big complicated package.
link |
I don't understand the rest of it,
link |
but I got the right answer.
link |
And I'm just, it's hard for me to get into that mindset.
link |
I want to understand the whole thing.
link |
And you know, if they wrote a manual,
link |
I should probably read it.
link |
And, but that's not necessarily the right way.
link |
And I think I have to get used to dealing with more,
link |
being more comfortable with uncertainty
link |
and not knowing everything.
link |
Yeah, so I struggle with the same instead of the,
link |
the spectrum between Donald and Don Knuth,
link |
looks kind of the very, you know,
link |
before you can say anything about a problem,
link |
he really has to get down to the machine code to assembly.
link |
And versus exactly what you said of several students
link |
in my group that, you know, like 20 years old,
link |
and they can solve almost any problem within a few hours
link |
that would take me probably weeks
link |
because I would try to, as you said, read the manual.
link |
So do you think the nature of mastery,
link |
you're mentioning biology, sort of outside disciplines,
link |
applying programming, but computer scientists.
link |
So over time, there's higher and higher levels
link |
of abstraction available now.
link |
So with this week, there's the TensorFlow Summit, right?
link |
So if you're not particularly into deep learning,
link |
but you're still a computer scientist,
link |
you can accomplish an incredible amount with TensorFlow
link |
without really knowing any fundamental internals
link |
of machine learning.
link |
Do you think the nature of mastery is changing,
link |
even for computer scientists, like what it means
link |
to be an expert programmer?
link |
Yeah, I think that's true.
link |
You know, we never really should have focused
link |
on programmer, right?
link |
Because it's still, it's a skill,
link |
and what we really want to focus on is the result.
link |
So we built this ecosystem where the way you can get stuff done
link |
is by programming it yourself.
link |
At least when I started it with it, you know,
link |
library functions meant you had square root,
link |
and that was about it, right?
link |
Everything else you built from scratch.
link |
And then we built up an ecosystem where a lot of times,
link |
well, you can download a lot of stuff
link |
that does a big part of what you need.
link |
And so now it's more a question of assembly
link |
rather than manufacturing.
link |
And that's a different way of looking at problems.
link |
From another perspective, in terms of mastery
link |
and looking at programmers or people that reason
link |
about problems in a computational way,
link |
so Google, you know, from the hiring perspective,
link |
from the perspective of hiring
link |
or building a team of programmers,
link |
how do you determine if someone's a good programmer?
link |
Or if somebody, again, so I want to deviate from,
link |
I want to move away from the word programmer,
link |
but somebody who can solve problems of large scale data
link |
and so on, what's, how do you build a team like that
link |
through the interviewing process?
link |
Yeah, and I think as a company grows,
link |
you get more expansive
link |
in the types of people you're looking for, right?
link |
So I think, you know, in the early days,
link |
we'd interview people and the question we were trying
link |
to ask is how close are they to Jeff Dean?
link |
And most people were pretty far away,
link |
but we take the ones that were, you know, not that far away.
link |
And so we got kind of a homogeneous group
link |
of people who are really great programmers.
link |
Then as a company grows, you say,
link |
well, we don't want everybody to be the same,
link |
to have the same skill set.
link |
And so now we're hiring biologists in our health areas
link |
and we're hiring physicists
link |
and we're hiring mechanical engineers
link |
and we're hiring, you know, social scientists
link |
and ethnographers and people with different backgrounds
link |
who bring different skills.
link |
So you have mentioned that you still make part
link |
taking code reviews, given that you have a wealth
link |
of experiences, you've also mentioned it.
link |
What errors do you often see and tend to highlight
link |
in the code of junior developers of people coming up now,
link |
given your background from Blisp
link |
to a couple of decades of programming?
link |
Yeah, that's a great question.
link |
You know, sometimes I try to look at flexibility
link |
of the design of, yes, this API solves this problem
link |
but where's it gonna go in the future?
link |
Who else is gonna wanna call this?
link |
And are you making it easier for them to do that?
link |
That's a matter of design, is it documentation,
link |
is it sort of an amorphous thing
link |
you can't really put into words, it's just how it feels.
link |
If you put yourself in the shoes of a developer,
link |
would you use this kind of thing?
link |
I think it is how you feel, right?
link |
And so yeah, documentation is good
link |
but it's more a design question, right?
link |
If you get the design right,
link |
then people will figure it out
link |
whether the documentation is good or not
link |
and if the design's wrong, then it'll be harder to use.
link |
How have you yourself changed as a programmer
link |
over the years in a way you already started to say,
link |
you want to read the manual,
link |
you want to understand the core of the syntax
link |
to how the language is supposed to be used and so on,
link |
but what's the evolution been like
link |
from the 80s, 90s to today?
link |
I guess one thing is you don't have to worry
link |
about the small details of efficiency
link |
as much as you used to, right?
link |
So like I remember I did my list book in the 90s
link |
and one of the things I wanted to do was say,
link |
here's how you do an object system
link |
and basically we're going to make it so each object
link |
is a hash table and you look up the methods
link |
and here's how it works and then I said,
link |
of course, the real common list object system
link |
is much more complicated,
link |
it's got all these efficiency type issues
link |
and this is just a toy and nobody would do this
link |
in real life and it turns out Python pretty much
link |
did exactly what I said and said objects are just dictionaries
link |
and yeah, they have a few little tricks as well,
link |
but mostly the thing that would have been 100 times
link |
too slow in the 80s is now plenty fast for most everything.
link |
So you had to, as a programmer, let go of perhaps
link |
an obsession that I remember coming up with
link |
of trying to write efficient code.
link |
Yeah, to say what really matters is the total time
link |
it takes to get the project done
link |
and most of that's going to be the programmer time,
link |
so if you're a little bit less efficient
link |
but it makes it easier to understand and modify,
link |
then that's the right trade off.
link |
So you've written quite a bit about Lisp,
link |
your book on programing is in Lisp,
link |
you have a lot of code out there that's in Lisp,
link |
so myself and people who don't know what Lisp is
link |
should look it up, it's my favorite language
link |
for many AI researchers,
link |
it is a favorite language,
link |
the favorite language they never use these days,
link |
so what part of the list do you find most beautiful
link |
So I think the beautiful part is the simplicity
link |
that in half a page you can define the whole language
link |
and other languages don't have that,
link |
so you feel like you can hold everything in your head
link |
and then a lot of people say well then that's too simple,
link |
here's all these things I want to do
link |
and my Java or Python or whatever
link |
has 100 or 200 or 300 different syntax rules
link |
and don't I need all those,
link |
and Lisp's answer was no,
link |
we're only going to give you eight or so syntax rules,
link |
but we're going to allow you to define your own
link |
and so that was a very powerful idea
link |
and I think this idea of saying
link |
I can start with my problem and with my data
link |
and then I can build the language I want
link |
for that problem and for that data
link |
and then I can make Lisp define that language,
link |
so you're sort of mixing levels
link |
and saying I'm simultaneously a programmer
link |
in a language and a language designer
link |
and that allows a better match
link |
between your problem and your eventual code
link |
and I think Lisp had done that better than other languages.
link |
Yeah, it's a very elegant implementation
link |
of functional programming,
link |
but why do you think Lisp has not had
link |
the mass adoption and success of languages like Python?
link |
Is it the parentheses?
link |
Is it all the parentheses?
link |
Yeah, so I think a couple of things.
link |
So one was, I think it was designed
link |
for a single programmer or a small team
link |
and a skilled programmer who had the good taste
link |
to say, well, I am doing language design
link |
and I have to make good choices
link |
and if you make good choices, that's great.
link |
If you make bad choices, you can hurt yourself
link |
and it can be hard for other people on the team to understand it.
link |
So I think there was a limit to the scale
link |
of the size of a project in terms of number of people
link |
that Lisp was good for
link |
and as an industry, we kind of grew beyond that.
link |
I think it is in part the parentheses.
link |
One of the jokes is the acronym for Lisp
link |
is lots of irritating silly parentheses.
link |
My acronym was Lisp is syntactically pure
link |
saying all you need is parentheses and atoms.
link |
But I remember as we had the AI textbook
link |
and because we did it in the 90s,
link |
we had pseudocode in the book
link |
but then we said, well, we'll have Lisp online
link |
because that's the language of AI at the time.
link |
And I remember some of the students complaining
link |
because they hadn't had Lisp before
link |
and they didn't quite understand what was going on
link |
and I remember one student complained,
link |
I don't understand how this pseudocode
link |
corresponds to this Lisp
link |
and there was a one to one correspondence
link |
between the symbols in the code and the pseudocode
link |
and the only thing difference was the parentheses.
link |
So I said it must be that for some people
link |
a certain number of left parentheses
link |
shuts off their brain.
link |
Yeah, it's very possible in that sense
link |
and Python just goes the other way.
link |
So that was the point at which I said,
link |
okay, can't have only Lisp as a language
link |
because I don't want to,
link |
you only got 10 or 12 or 15 weeks
link |
to whatever it is to teach AI
link |
and I don't want to waste two weeks of that teaching Lisp.
link |
So I said, I got to have another language.
link |
Java was the most popular language at the time.
link |
I started doing that and then I said,
link |
it's really hard to have a one to one correspondence
link |
between the pseudocode and the Java
link |
because Java is so verbose.
link |
So then I said, I'm going to do a survey
link |
and find the language that's most like my pseudocode
link |
and turned out Python basically was my pseudocode.
link |
Somehow I had channeled Guido
link |
and designed a pseudocode that was the same as Python,
link |
although I hadn't heard of Python at that point.
link |
And from then on, that's what I've been using
link |
because it's been a good match.
link |
So what's the story in Python behind Pytudes?
link |
You're a GitHub repository with puzzles and exercises
link |
and Python is pretty fun.
link |
Yeah, it seems like fun.
link |
You know, I like doing puzzles and I like being an educator.
link |
I did a class with Udacity, Udacity 2.12.
link |
I think it was, it was basically problem solving,
link |
using Python and looking at different problems.
link |
Does Pytudes feed that class in terms of the exercises?
link |
I was wondering what the...
link |
Yeah, so the class came first.
link |
Some of the stuff that's in Pytudes
link |
was write ups of what was in the class
link |
and then some of it was just continuing to work on new problems.
link |
So what's the organizing madness of Pytudes?
link |
Is it just a collection of cool exercises?
link |
Just whatever I thought was fun.
link |
So you were the director of search quality at Google
link |
from 2001 to 2005 in the early days
link |
when there were just a few employees
link |
and when the company was growing like crazy.
link |
So, I mean, Google revolutionized the way we discover,
link |
share and aggregate knowledge.
link |
So this is one of the fundamental aspects of civilization,
link |
is information being shared
link |
and there's different mechanisms throughout history
link |
but Google has just 10x improved that.
link |
And you're a part of that, people discovering that information.
link |
So what were some of the challenges
link |
on the philosophical or the technical level in those early days?
link |
It definitely was an exciting time
link |
and as you say, we were doubling in size every year
link |
and the challenges were we wanted to get the right answers.
link |
And we had to figure out what that meant.
link |
We had to implement that and we had to make it all efficient
link |
and we had to keep on testing
link |
and seeing if we were delivering good answers.
link |
And now when you say good answers,
link |
it means whatever people are typing in in terms of keywords
link |
in terms of that kind of thing,
link |
that the results they get are ordered by the desirability
link |
for them of those results.
link |
The first thing they click on will likely be the thing
link |
that they were actually looking for.
link |
Right, one of the metrics we had was focused on the first thing.
link |
Some of it was focused on the whole page,
link |
some of it was focused on top three or so.
link |
So we looked at a lot of different metrics
link |
for how well we were doing
link |
and we broke it down into subclasses of, you know,
link |
maybe here's a type of query that we're not doing well on
link |
and we try to fix that.
link |
Early on, we started to realize
link |
that we were in an adversarial position.
link |
So we started thinking,
link |
well, we're kind of like the card catalog in the library.
link |
The books are here and we're off to the side
link |
and we're just reflecting what's there.
link |
And then we realized every time we make a change,
link |
the webmasters make a change.
link |
And it's game theoretic.
link |
And so we had to think not only is this the right move
link |
for us to make now,
link |
but also if we make this move,
link |
what's the counter move going to be?
link |
Is that going to get us into a worse place,
link |
in which case we won't make that move,
link |
we'll make a different move.
link |
And did you find, I mean, I assume,
link |
with the popularity and the growth of the internet
link |
that people were creating new content,
link |
so you're almost helping guide the creation of content?
link |
Yeah, so that's certainly true, right?
link |
So we definitely changed the structure of the network, right?
link |
So if you think back, you know, in the very early days,
link |
Larry and Sergey had the PageRank paper
link |
and John Kleinberg had this hubs and authorities model,
link |
which says the web is made out of these hubs,
link |
which will be my page of cool links about dogs or whatever,
link |
and people would just list links.
link |
And then there'd be authorities which were the ones,
link |
the page about dogs that most people linked to.
link |
That doesn't happen anymore.
link |
People don't bother to say my page of cool links
link |
because we took over that function, right?
link |
So we changed the way that worked.
link |
Did you imagine back then that the internet
link |
would be as massively vibrant as it is today?
link |
I mean, it was already growing quickly,
link |
but it's just another...
link |
I don't know if you've ever...
link |
Today, if you sit back and just look at the internet
link |
with wonder, the amount of content
link |
that's just constantly being created,
link |
constantly being shared and employed.
link |
Yeah, it's always been surprising to me.
link |
Because I'm not very good at predicting the future.
link |
And I remember being a graduate student in 1980 or so,
link |
and we had the ARPANET,
link |
and then there was this proposal to commercialize it
link |
and have this internet and this crazy Senator Gore
link |
thought that might be a good idea.
link |
And I remember thinking, oh, come on.
link |
You can't expect a commercial company
link |
to understand this technology.
link |
They'll never be able to do it.
link |
Yeah, okay, we can have this.com domain,
link |
but it won't go anywhere.
link |
So I was wrong, Al Gore was right.
link |
At the same time, the nature of what it means
link |
to be a commercial company has changed, too.
link |
So Google, many ways, at its founding,
link |
is different than what companies were before, I think.
link |
Right, so there's all these business models
link |
that are so different than what was possible back then.
link |
So in terms of predicting the future,
link |
what do you think it takes to build a system
link |
that approaches human level intelligence?
link |
You've talked about, of course,
link |
that we shouldn't be so obsessed
link |
about creating human level intelligence.
link |
We just create systems that are very useful for humans.
link |
But what do you think it takes to approach that level?
link |
Right, so certainly I don't think human level intelligence
link |
is one thing, right?
link |
So I think there's lots of different tasks,
link |
lots of different capabilities.
link |
I also don't think that should be the goal, right?
link |
So I wouldn't want to create a calculator
link |
that could do multiplication at human level.
link |
That would be a step backwards.
link |
And so for many things,
link |
we should be aiming far beyond human level.
link |
maybe human level is a good level to aim at.
link |
And for others, we'd say,
link |
well, let's not bother doing this,
link |
because we already have humans who can take on those tasks.
link |
So as you say, I'd like to focus on what's a useful tool.
link |
And in some cases,
link |
being at human level is an important part
link |
of crossing that threshold to make the tool useful.
link |
So we see in things like these personal assistants now
link |
that you get either on your phone
link |
or on a speaker that sits on the table,
link |
you want to be able to have a conversation with those.
link |
And I think as an industry,
link |
we haven't quite figured out what the right model is
link |
for what these things can do.
link |
And we're aiming towards,
link |
well, you just have a conversation with them
link |
the way you can with a person.
link |
But we haven't delivered on that model yet, right?
link |
So you can ask it, what's the weather?
link |
You can ask it, play some nice songs
link |
and five or six other things,
link |
and then you run out of stuff that it can do.
link |
In terms of a deep, meaningful connection.
link |
So you've mentioned the movie Her
link |
as one of your favorite A.I. movies.
link |
Do you think it's possible for a human being
link |
to fall in love with an A.I. system,
link |
A.I. assistant, as you mentioned,
link |
to have taken this big leap from what's the weather
link |
to having a deep connection?
link |
Yeah, I think as people,
link |
that's what we love to do.
link |
And I was at a showing of Her
link |
where we had a panel discussion
link |
and somebody asked me,
link |
what other movie do you think Her is similar to?
link |
And my answer was Life of Brian,
link |
which is not a science fiction movie,
link |
but both movies are about wanting to believe
link |
in something that's not necessarily real.
link |
Yeah, by the way, for people who don't know,
link |
it's Monty Python.
link |
Yeah, it's brilliantly put.
link |
So I mean, I think that's just the way we are.
link |
We want to trust, we want to believe,
link |
we want to fall in love,
link |
and it doesn't necessarily take that much, right?
link |
So, you know, my kids fell in love with their teddy bear,
link |
and the teddy bear was not very interactive, right?
link |
So that's all us pushing our feelings
link |
onto our devices and our things,
link |
and I think that that's what we like to do,
link |
so we'll continue to do that.
link |
So yeah, as human beings, we long for that connection,
link |
and just A.I. has to do a little bit of work
link |
to catch us on the other end.
link |
Yeah, and certainly, you know,
link |
if you can get to dog level,
link |
a lot of people have invested a lot of love in their pets.
link |
Some people, as I've been told,
link |
in working with autonomous vehicles,
link |
have invested a lot of love into their inanimate cars,
link |
so it really doesn't take much.
link |
So what is a good test to linger on a topic
link |
that may be silly or a little bit philosophical?
link |
What is a good test of intelligence in your view?
link |
Is natural conversation, like in the touring test,
link |
a good test, put another way,
link |
what would impress you if you saw a computer do it these days?
link |
Yeah, I mean, I get impressed all the time.
link |
Right, so, you know, go playing,
link |
Starcraft playing, those are all pretty cool.
link |
You know, and I think, sure,
link |
conversation is important.
link |
I think, you know,
link |
we sometimes have these tests where it's easy to fool the system,
link |
where you can have a chat bot that can have a conversation,
link |
but it never gets into a situation where it has to be deep enough
link |
that it really reveals itself as being intelligent or not.
link |
I think, you know, Turing suggested that,
link |
but I think if he were alive, he'd say,
link |
you know, I didn't really mean that seriously.
link |
And I think, you know, this is just my opinion,
link |
but I think Turing's point was not that this test of conversation
link |
I think his point was having a test is the right thing.
link |
So rather than having the philosopher say,
link |
oh, no, AI is impossible,
link |
you should say, well, we'll just have a test.
link |
And then the result of that will tell us the answer
link |
and doesn't necessarily have to be a conversation test.
link |
And coming up with you,
link |
but a test as a technology evolves is probably the right way.
link |
Do you worry, as a lot of the general public does about,
link |
not a lot, but some vocal part of the general public
link |
about the existential threat of artificial intelligence?
link |
So looking farther into the future, as you said,
link |
most of us are not able to predict much.
link |
So when shrouded in such mystery, there's a concern of,
link |
well, you start to think about worst case.
link |
Is that something that occupies your mind space much?
link |
So I certainly think about threats.
link |
I think about dangers.
link |
And I think any new technology has positives and negatives.
link |
And if it's a powerful technology,
link |
it can be used for bad as well as for good.
link |
So I'm certainly not worried about the robot apocalypse
link |
and the Terminator type scenarios.
link |
I am worried about change in employment.
link |
And are we going to be able to react fast enough to deal with that?
link |
I think we're already seeing it today where a lot of people
link |
are disgruntled about the way income inequality is working.
link |
And automation could help accelerate those kinds of problems.
link |
I see powerful technologies can always be used as weapons,
link |
whether they're robots or drones or whatever.
link |
Some of that we're seeing in due to AI.
link |
A lot of it, you don't need AI.
link |
And I don't know what's a worst threat.
link |
If it's an autonomous drone or it's a CRISPR technology
link |
becoming available or we have lots of threats to face
link |
and some of them involve AI and some of them don't.
link |
So the threats that technology presents
link |
are, for the most part, optimistic about technology
link |
also alleviating those threats or creating new opportunities
link |
or protecting us from the more detrimental effects of these.
link |
Yeah, I don't know.
link |
Again, it's hard to predict the future.
link |
And as a society so far,
link |
we've survived nuclear bombs and other things.
link |
Of course, only societies that have survived
link |
are having this conversation.
link |
Maybe that's survivorship bias there.
link |
What problem stands out to you as exciting, challenging,
link |
impactful to work on in the near future
link |
for yourself, for the community, and broadly?
link |
So we talked about these assistance and conversation.
link |
I think that's a great area.
link |
I think combining common sense reasoning
link |
with the power of data is a great area.
link |
In which application?
link |
In conversational issues, just broadly speaking?
link |
Just in general, yeah.
link |
As a programmer, I'm interested in programming tools,
link |
both in terms of the current systems we have today
link |
with TensorFlow and so on.
link |
Can we make them much easier to use for a broader class of people?
link |
And also, can we apply machine learning to the more traditional
link |
type of programming?
link |
So when you go to Google and you type in a query
link |
and you spell something wrong, it says, did you mean?
link |
And the reason we're able to do that is because lots of other people
link |
made a similar error and then they corrected it.
link |
We should be able to go into our code bases
link |
and our bugfix spaces.
link |
And when I type a line of code, it should be able to say,
link |
did you mean such and such?
link |
If you type this today, you're probably going to type in this bugfix tomorrow.
link |
Yeah, that's a really exciting application of almost an assistant
link |
for the coding program experience at every level.
link |
So I think I could safely speak for the entire AI community.
link |
First of all, thank you for the amazing work you've done.
link |
Certainly for the amazing work you've done with AI, a modern approach book.
link |
I think we're all looking forward very much for the fourth edition
link |
and then the fifth edition and so on.
link |
So Peter, thank you so much for talking today.