back to indexJim Keller: Moore's Law, Microprocessors, and First Principles | Lex Fridman Podcast #70
link |
The following is a conversation with Jim Keller,
link |
legendary microprocessor engineer
link |
who has worked at AMD, Apple, Tesla, and now Intel.
link |
He's known for his work on AMD K7, K8, K12,
link |
and Zen microarchitectures, Apple A4 and A5 processors,
link |
and coauthor of the specification
link |
for the x8664 instruction set
link |
and hypertransport interconnect.
link |
He's a brilliant first principles engineer
link |
and out of the box thinker,
link |
and just an interesting and fun human being to talk to.
link |
This is the Artificial Intelligence Podcast.
link |
If you enjoy it, subscribe on YouTube,
link |
give it five stars on Apple Podcast,
link |
follow on Spotify, support it on Patreon,
link |
or simply connect with me on Twitter,
link |
at Lex Friedman, spelled F R I D M A N.
link |
I recently started doing ads
link |
at the end of the introduction.
link |
I'll do one or two minutes after introducing the episode
link |
and never any ads in the middle
link |
that can break the flow of the conversation.
link |
I hope that works for you
link |
and doesn't hurt the listening experience.
link |
This show is presented by Cash App,
link |
the number one finance app in the App Store.
link |
I personally use Cash App to send money to friends,
link |
but you can also use it to buy, sell,
link |
and deposit Bitcoin in just seconds.
link |
Cash App also has a new investing feature.
link |
You can buy fractions of a stock, say $1 worth,
link |
no matter what the stock price is.
link |
Broker services are provided by Cash App Investing,
link |
a subsidiary of Square and member SIPC.
link |
I'm excited to be working with Cash App
link |
to support one of my favorite organizations called First,
link |
best known for their FIRST Robotics and Lego competitions.
link |
They educate and inspire hundreds of thousands of students
link |
in over 110 countries and have a perfect rating
link |
at Charity Navigator,
link |
which means that donated money
link |
is used to maximum effectiveness.
link |
When you get Cash App from the App Store or Google Play
link |
and use code LEXPODCAST,
link |
you'll get $10 and Cash App will also donate $10 to FIRST,
link |
which again is an organization
link |
that I've personally seen inspire girls and boys
link |
to dream of engineering a better world.
link |
And now here's my conversation with Jim Keller.
link |
What are the differences and similarities
link |
between the human brain and a computer
link |
with the microprocessor at its core?
link |
Let's start with the philosophical question perhaps.
link |
Well, since people don't actually understand
link |
how human brains work, I think that's true.
link |
I think that's true.
link |
So it's hard to compare them.
link |
Computers are, you know, there's really two things.
link |
There's memory and there's computation, right?
link |
And to date, almost all computer architectures
link |
are global memory, which is a thing, right?
link |
And then computation where you pull data
link |
and you do relatively simple operations on it
link |
and write data back.
link |
So it's decoupled in modern computers.
link |
And you think in the human brain,
link |
everything's a mesh, a mess that's combined together?
link |
What people observe is there's, you know,
link |
some number of layers of neurons
link |
which have local and global connections
link |
and information is stored in some distributed fashion
link |
and people build things called neural networks in computers
link |
where the information is distributed
link |
in some kind of fashion.
link |
You know, there's a mathematics behind it.
link |
I don't know that the understanding of that is super deep.
link |
The computations we run on those
link |
are straightforward computations.
link |
I don't believe anybody has said
link |
a neuron does this computation.
link |
So to date, it's hard to compare them, I would say.
link |
So let's get into the basics before we zoom back out.
link |
How do you build a computer from scratch?
link |
What is a microprocessor?
link |
What is a microarchitecture?
link |
What's an instruction set architecture?
link |
Maybe even as far back as what is a transistor?
link |
So the special charm of computer engineering
link |
is there's a relatively good understanding
link |
of abstraction layers.
link |
So down at the bottom, you have atoms
link |
and atoms get put together in materials like silicon
link |
or dope silicon or metal and we build transistors.
link |
On top of that, we build logic gates, right?
link |
And then functional units, like an adder or a subtractor
link |
or an instruction parsing unit.
link |
And then we assemble those into processing elements.
link |
Modern computers are built out of probably 10 to 20
link |
locally organic processing elements
link |
or coherent processing elements.
link |
And then that runs computer programs, right?
link |
So there's abstraction layers and then software,
link |
there's an instruction set you run
link |
and then there's assembly language C, C++, Java, JavaScript.
link |
There's abstraction layers,
link |
essentially from the atom to the data center, right?
link |
So when you build a computer,
link |
first there's a target, like what's it for?
link |
Like how fast does it have to be?
link |
Which today there's a whole bunch of metrics
link |
about what that is.
link |
And then in an organization of 1,000 people
link |
who build a computer, there's lots of different disciplines
link |
that you have to operate on.
link |
Does that make sense?
link |
So there's a bunch of levels of abstraction
link |
in an organization like Intel and in your own vision,
link |
there's a lot of brilliance that comes in
link |
at every one of those layers.
link |
Some of it is science, some of it is engineering,
link |
some of it is art, what's the most,
link |
if you could pick favorites,
link |
what's the most important, your favorite layer
link |
on these layers of abstractions?
link |
Where does the magic enter this hierarchy?
link |
I don't really care.
link |
That's the fun, you know, I'm somewhat agnostic to that.
link |
So I would say for relatively long periods of time,
link |
instruction sets are stable.
link |
So the x86 instruction set, the ARM instruction set.
link |
What's an instruction set?
link |
So it says, how do you encode the basic operations?
link |
Load, store, multiply, add, subtract, conditional, branch.
link |
You know, there aren't that many interesting instructions.
link |
Look, if you look at a program and it runs,
link |
you know, 90% of the execution is on 25 opcodes,
link |
you know, 25 instructions.
link |
And those are stable, right?
link |
What does it mean, stable?
link |
Intel architecture's been around for 25 years.
link |
And that's because the basics, you know,
link |
are defined a long time ago, right?
link |
Now, the way an old computer ran is you fetched
link |
instructions and you executed them in order.
link |
Do the load, do the add, do the compare.
link |
The way a modern computer works is you fetch
link |
large numbers of instructions, say 500.
link |
And then you find the dependency graph
link |
between the instructions.
link |
And then you execute in independent units
link |
those little micrographs.
link |
So a modern computer, like people like to say,
link |
computers should be simple and clean.
link |
But it turns out the market for simple,
link |
clean, slow computers is zero, right?
link |
We don't sell any simple, clean computers.
link |
No, you can, how you build it can be clean,
link |
but the computer people want to buy,
link |
that's, say, in a phone or a data center,
link |
fetches a large number of instructions,
link |
computes the dependency graph,
link |
and then executes it in a way that gets the right answers.
link |
And optimizes that graph somehow.
link |
Yeah, they run deeply out of order.
link |
And then there's semantics around how memory ordering works
link |
and other things work.
link |
So the computer sort of has a bunch of bookkeeping tables
link |
that says what order should these operations finish in
link |
or appear to finish in?
link |
But to go fast, you have to fetch a lot of instructions
link |
and find all the parallelism.
link |
Now, there's a second kind of computer,
link |
which we call GPUs today.
link |
And I call it the difference.
link |
There's found parallelism, like you have a program
link |
with a lot of dependent instructions.
link |
You fetch a bunch and then you go figure out
link |
the dependency graph and you issue instructions out of order.
link |
That's because you have one serial narrative to execute,
link |
which, in fact, can be done out of order.
link |
Did you call it a narrative?
link |
Yeah, so humans think of serial narrative.
link |
So read a book, right?
link |
There's a sentence after sentence after sentence,
link |
and there's paragraphs.
link |
Now, you could diagram that.
link |
Imagine you diagrammed it properly and you said,
link |
which sentences could be read in any order,
link |
any order without changing the meaning, right?
link |
That's a fascinating question to ask of a book, yeah.
link |
Yeah, you could do that, right?
link |
So some paragraphs could be reordered,
link |
some sentences can be reordered.
link |
You could say, he is tall and smart and X, right?
link |
And it doesn't matter the order of tall and smart.
link |
But if you say the tall man is wearing a red shirt,
link |
what colors, you can create dependencies, right?
link |
And so GPUs, on the other hand,
link |
run simple programs on pixels,
link |
but you're given a million of them.
link |
And the first order, the screen you're looking at
link |
doesn't care which order you do it in.
link |
So I call that given parallelism.
link |
Simple narratives around the large numbers of things
link |
where you can just say,
link |
it's parallel because you told me it was.
link |
So found parallelism where the narrative is sequential,
link |
but you discover like little pockets of parallelism versus.
link |
Turns out large pockets of parallelism.
link |
Large, so how hard is it to discover?
link |
Well, how hard is it?
link |
That's just transistor count, right?
link |
So once you crack the problem, you say,
link |
here's how you fetch 10 instructions at a time.
link |
Here's how you calculate the dependencies between them.
link |
Here's how you describe the dependencies.
link |
Here's, you know, these are pieces, right?
link |
So once you describe the dependencies,
link |
then it's just a graph.
link |
Sort of, it's an algorithm that finds,
link |
I'm sure there's a graph theoretical answer here
link |
In general, programs, modern programs
link |
that human beings write,
link |
how much found parallelism is there in them?
link |
What does 10X mean?
link |
So if you execute it in order, you would get
link |
what's called cycles per instruction,
link |
and it would be about, you know,
link |
three instructions, three cycles per instruction
link |
because of the latency of the operations and stuff.
link |
And in a modern computer, excuse it,
link |
but like 0.2, 0.25 cycles per instruction.
link |
So it's about, we today find 10X.
link |
And there's two things.
link |
One is the found parallelism in the narrative, right?
link |
And the other is the predictability of the narrative, right?
link |
So certain operations say, do a bunch of calculations,
link |
and if greater than one, do this, else do that.
link |
That decision is predicted in modern computers
link |
to high 90% accuracy.
link |
So branches happen a lot.
link |
So imagine you have a decision
link |
to make every six instructions,
link |
which is about the average, right?
link |
But you want to fetch 500 instructions,
link |
figure out the graph, and execute them all in parallel.
link |
That means you have, let's say,
link |
if you fetch 600 instructions and it's every six,
link |
you have to fetch, you have to predict
link |
99 out of 100 branches correctly
link |
for that window to be effective.
link |
Okay, so parallelism, you can't parallelize branches.
link |
No, you can predict.
link |
What does predicted branch mean?
link |
What does predicted branch mean?
link |
So imagine you do a computation over and over.
link |
So while n is greater than one, do.
link |
And you go through that loop a million times.
link |
So every time you look at the branch,
link |
you say, it's probably still greater than one.
link |
And you're saying you could do that accurately.
link |
How the heck do you do that?
link |
Well, you want to know?
link |
This is really sad.
link |
20 years ago, you simply recorded
link |
which way the branch went last time
link |
and predicted the same thing.
link |
What's the accuracy of that?
link |
So then somebody said, hey, let's keep a couple of bits
link |
and have a little counter so when it predicts one way,
link |
we count up and then pins.
link |
So say you have a three bit counter.
link |
So you count up and then you count down.
link |
And you can use the top bit as the signed bit
link |
so you have a signed two bit number.
link |
So if it's greater than one, you predict taken.
link |
And less than one, you predict not taken, right?
link |
Or less than zero, whatever the thing is.
link |
And that got us to 92%.
link |
Okay, no, it gets better.
link |
This branch depends on how you got there.
link |
So if you came down the code one way,
link |
you're talking about Bob and Jane, right?
link |
And then said, does Bob like Jane?
link |
But if you're talking about Bob and Jill,
link |
does Bob like Jane?
link |
You go a different way.
link |
Right, so that's called history.
link |
So you take the history and a counter.
link |
That's cool, but that's not how anything works today.
link |
They use something that looks a little like a neural network.
link |
So modern, you take all the execution flows.
link |
And then you do basically deep pattern recognition
link |
of how the program is executing.
link |
And you do that multiple different ways.
link |
And you have something that chooses what the best result is.
link |
There's a little supercomputer inside the computer.
link |
That's trying to predict branching.
link |
That calculates which way branches go.
link |
So the effective window that it's worth finding grass
link |
Why was that gonna make me sad?
link |
Because that's amazing.
link |
It's amazingly complicated.
link |
Well, here's the funny thing.
link |
So to get to 85% took 1,000 bits.
link |
To get to 99% takes tens of megabits.
link |
So this is one of those, to get the result,
link |
to get from a window of say 50 instructions to 500,
link |
it took three orders of magnitude
link |
or four orders of magnitude more bits.
link |
Now if you get the prediction of a branch wrong,
link |
what happens then?
link |
You flush the pipe.
link |
You flush the pipe, so it's just the performance cost.
link |
But it gets even better.
link |
So we're starting to look at stuff that says,
link |
so they executed down this path,
link |
and then you had two ways to go.
link |
But far away, there's something that doesn't matter
link |
which path you went.
link |
So you took the wrong path.
link |
You executed a bunch of stuff.
link |
Then you had the mispredicting.
link |
You remembered all the results you already calculated.
link |
Some of those are just fine.
link |
Like if you read a book and you misunderstand a paragraph,
link |
your understanding of the next paragraph
link |
sometimes is invariant to that understanding.
link |
Sometimes it depends on it.
link |
And you can kind of anticipate that invariance.
link |
Yeah, well, you can keep track of whether the data changed.
link |
And so when you come back through a piece of code,
link |
should you calculate it again or do the same thing?
link |
Okay, how much of this is art and how much of it is science?
link |
Because it sounds pretty complicated.
link |
Well, how do you describe a situation?
link |
So imagine you come to a point in the road
link |
where you have to make a decision, right?
link |
And you have a bunch of knowledge about which way to go.
link |
Maybe you have a map.
link |
So you wanna go the shortest way,
link |
or do you wanna go the fastest way,
link |
or do you wanna take the nicest road?
link |
So there's some set of data.
link |
So imagine you're doing something complicated
link |
like building a computer.
link |
And there's hundreds of decision points,
link |
all with hundreds of possible ways to go.
link |
And the ways you pick interact in a complicated way.
link |
And then you have to pick the right spot.
link |
So that's art or science, I don't know.
link |
You avoided the question.
link |
You just described the Robert Frost problem
link |
of road less taken.
link |
I described the Robert Frost problem?
link |
That's what we do as computer designers.
link |
Yeah, I don't know how to describe that
link |
because some people are very good
link |
at making those intuitive leaps.
link |
It seems like just combinations of things.
link |
Some people are less good at it,
link |
but they're really good at evaluating the alternatives.
link |
Right, and everybody has a different way to do it.
link |
And some people can't make those leaps,
link |
but they're really good at analyzing it.
link |
So when you see computers are designed
link |
by teams of people who have very different skill sets.
link |
And a good team has lots of different kinds of people.
link |
I suspect you would describe some of them
link |
as artistic, but not very many.
link |
Unfortunately, or fortunately.
link |
Well, you know, computer design's hard.
link |
It's 99% perspiration.
link |
And the 1% inspiration is really important.
link |
But you still need the 99.
link |
Yeah, you gotta do a lot of work.
link |
And then there are interesting things to do
link |
at every level of that stack.
link |
So at the end of the day,
link |
if you run the same program multiple times,
link |
does it always produce the same result?
link |
Is there some room for fuzziness there?
link |
That's a math problem.
link |
So if you run a correct C program,
link |
the definition is every time you run it,
link |
you get the same answer.
link |
Yeah, well that's a math statement.
link |
But that's a language definitional statement.
link |
So for years when people did,
link |
when we first did 3D acceleration of graphics,
link |
you could run the same scene multiple times
link |
and get different answers.
link |
Right, and then some people thought that was okay
link |
and some people thought it was a bad idea.
link |
And then when the HPC world used GPUs for calculations,
link |
they thought it was a really bad idea.
link |
Okay, now in modern AI stuff,
link |
people are looking at networks
link |
where the precision of the data is low enough
link |
that the data is somewhat noisy.
link |
And the observation is the input data is unbelievably noisy.
link |
So why should the calculation be not noisy?
link |
And people have experimented with algorithms
link |
that say can get faster answers by being noisy.
link |
Like as a network starts to converge,
link |
if you look at the computation graph,
link |
it starts out really wide and then it gets narrower.
link |
And you can say is that last little bit that important
link |
or should I start the graph on the next rev
link |
before we whittle it all the way down to the answer, right?
link |
So you can create algorithms that are noisy.
link |
Now if you're developing something
link |
and every time you run it, you get a different answer,
link |
it's really annoying.
link |
And so most people think even today,
link |
every time you run the program, you get the same answer.
link |
No, I know, but the question is
link |
that's the formal definition of a programming language.
link |
There is a definition of languages
link |
that don't get the same answer,
link |
but people who use those, you always want something
link |
because you get a bad answer and then you're wondering
link |
is it because of something in the algorithm
link |
or because of this?
link |
And so everybody wants a little switch that says
link |
no matter what, do it deterministically.
link |
And it's really weird because almost everything
link |
going into modern calculations is noisy.
link |
So why do the answers have to be so clear?
link |
Right, so where do you stand?
link |
I design computers for people who run programs.
link |
So if somebody says I want a deterministic answer,
link |
like most people want that.
link |
Can you deliver a deterministic answer,
link |
I guess is the question.
link |
Yeah, hopefully, sure.
link |
What people don't realize is you get a deterministic answer
link |
even though the execution flow is very undeterministic.
link |
So you run this program 100 times,
link |
it never runs the same way twice, ever.
link |
And the answer, it arrives at the same answer.
link |
But it gets the same answer every time.
link |
It's just amazing.
link |
Okay, you've achieved, in the eyes of many people,
link |
legend status as a chip art architect.
link |
What design creation are you most proud of?
link |
Perhaps because it was challenging,
link |
because of its impact, or because of the set
link |
of brilliant ideas that were involved in bringing it to life?
link |
I find that description odd.
link |
And I have two small children, and I promise you,
link |
they think it's hilarious.
link |
So I'm really interested in building computers.
link |
And I've worked with really, really smart people.
link |
I'm not unbelievably smart.
link |
I'm fascinated by how they go together,
link |
both as a thing to do and as an endeavor that people do.
link |
How people and computers go together?
link |
Like how people think and build a computer.
link |
And I find sometimes that the best computer architects
link |
aren't that interested in people,
link |
or the best people managers aren't that good
link |
at designing computers.
link |
So the whole stack of human beings is fascinating.
link |
So the managers, the individual engineers.
link |
Yeah, I said I realized after a lot of years
link |
of building computers, where you sort of build them
link |
out of transistors, logic gates, functional units,
link |
computational elements, that you could think of people
link |
the same way, so people are functional units.
link |
And then you could think of organizational design
link |
as a computer architecture problem.
link |
And then it was like, oh, that's super cool,
link |
because the people are all different,
link |
just like the computational elements are all different.
link |
And they like to do different things.
link |
And so I had a lot of fun reframing
link |
how I think about organizations.
link |
Just like with computers, we were saying execution paths,
link |
you can have a lot of different paths that end up
link |
at the same good destination.
link |
So what have you learned about the human abstractions
link |
from individual functional human units
link |
to the broader organization?
link |
What does it take to create something special?
link |
Well, most people don't think simple enough.
link |
All right, so the difference between a recipe
link |
and the understanding.
link |
There's probably a philosophical description of this.
link |
So imagine you're gonna make a loaf of bread.
link |
The recipe says get some flour, add some water,
link |
add some yeast, mix it up, let it rise,
link |
put it in a pan, put it in the oven.
link |
Understanding bread, you can understand biology,
link |
supply chains, grain grinders, yeast, physics,
link |
thermodynamics, there's so many levels of understanding.
link |
And then when people build and design things,
link |
they frequently are executing some stack of recipes.
link |
And the problem with that is the recipes
link |
all have limited scope.
link |
Like if you have a really good recipe book
link |
for making bread, it won't tell you anything
link |
about how to make an omelet.
link |
But if you have a deep understanding of cooking,
link |
right, than bread, omelets, you know, sandwich,
link |
you know, there's a different way of viewing everything.
link |
And most people, when you get to be an expert at something,
link |
you know, you're hoping to achieve deeper understanding,
link |
not just a large set of recipes to go execute.
link |
And it's interesting to walk groups of people
link |
because executing recipes is unbelievably efficient
link |
if it's what you want to do.
link |
If it's not what you want to do, you're really stuck.
link |
And that difference is crucial.
link |
And everybody has a balance of, let's say,
link |
deeper understanding of recipes.
link |
And some people are really good at recognizing
link |
when the problem is to understand something deeply.
link |
Does that make sense?
link |
It totally makes sense, does every stage of development,
link |
deep understanding on the team needed?
link |
Oh, this goes back to the art versus science question.
link |
If you constantly unpack everything
link |
for deeper understanding, you never get anything done.
link |
And if you don't unpack understanding when you need to,
link |
you'll do the wrong thing.
link |
And then at every juncture, like human beings
link |
are these really weird things because everything you tell them
link |
has a million possible outputs, right?
link |
And then they all interact in a hilarious way.
link |
Yeah, it's very interesting.
link |
And then having some intuition about what you tell them,
link |
what you do, when do you intervene, when do you not,
link |
It's essentially computationally unsolvable.
link |
Yeah, it's an intractable problem, sure.
link |
Humans are a mess.
link |
But with deep understanding,
link |
do you mean also sort of fundamental questions
link |
of things like what is a computer?
link |
Or why, like the why questions,
link |
why are we even building this, like of purpose?
link |
Or do you mean more like going towards
link |
the fundamental limits of physics,
link |
sort of really getting into the core of the science?
link |
In terms of building a computer, think a little simpler.
link |
So common practice is you build a computer,
link |
and then when somebody says, I wanna make it 10% faster,
link |
you'll go in and say, all right,
link |
I need to make this buffer bigger,
link |
and maybe I'll add an add unit.
link |
Or I have this thing that's three instructions wide,
link |
I'm gonna make it four instructions wide.
link |
And what you see is each piece
link |
gets incrementally more complicated, right?
link |
And then at some point you hit this limit,
link |
like adding another feature or buffer
link |
doesn't seem to make it any faster.
link |
And then people will say,
link |
well, that's because it's a fundamental limit.
link |
And then somebody else will look at it and say,
link |
well, actually the way you divided the problem up
link |
and the way the different features are interacting
link |
is limiting you, and it has to be rethought, rewritten.
link |
So then you refactor it and rewrite it,
link |
and what people commonly find is the rewrite
link |
is not only faster, but half as complicated.
link |
From scratch? Yes.
link |
So how often in your career, but just have you seen
link |
is needed, maybe more generally,
link |
to just throw the whole thing out and start over?
link |
This is where I'm on one end of it,
link |
every three to five years.
link |
Which end are you on?
link |
Rewrite more often.
link |
Rewrite, and three to five years is?
link |
If you wanna really make a lot of progress
link |
on computer architecture, every five years
link |
you should do one from scratch.
link |
So where does the x86.64 standard come in?
link |
I was the coauthor of that spec in 98.
link |
That's 20 years ago.
link |
Yeah, so that's still around.
link |
The instruction set itself has been extended
link |
quite a few times.
link |
And instruction sets are less interesting
link |
than the implementation underneath.
link |
There's been, on x86 architecture, Intel's designed a few,
link |
AIM designed a few very different architectures.
link |
And I don't wanna go into too much of the detail
link |
about how often, but there's a tendency
link |
to rewrite it every 10 years,
link |
and it really should be every five.
link |
So you're saying you're an outlier in that sense.
link |
Rewrite more often.
link |
Rewrite more often.
link |
Well, and here's the problem.
link |
Well, scary to who?
link |
To everybody involved, because like you said,
link |
repeating the recipe is efficient.
link |
Companies wanna make money.
link |
No, individual engineers wanna succeed,
link |
so you wanna incrementally improve,
link |
increase the buffer from three to four.
link |
Well, this is where you get
link |
into the diminishing return curves.
link |
I think Steve Jobs said this, right?
link |
So every, you have a project, and you start here,
link |
and it goes up, and you have diminishing return.
link |
And to get to the next level, you have to do a new one,
link |
and the initial starting point will be lower
link |
than the old optimization point, but it'll get higher.
link |
So now you have two kinds of fear,
link |
short term disaster and long term disaster.
link |
And you're, you're haunted.
link |
So grown ups, right, like, you know,
link |
people with a quarter by quarter business objective
link |
are terrified about changing everything.
link |
And people who are trying to run a business
link |
or build a computer for a long term objective
link |
know that the short term limitations block them
link |
from the long term success.
link |
So if you look at leaders of companies
link |
that had really good long term success,
link |
every time they saw that they had to redo something, they did.
link |
And so somebody has to speak up.
link |
Or you do multiple projects in parallel,
link |
like you optimize the old one while you build a new one.
link |
But the marketing guys are always like,
link |
make promise me that the new computer
link |
is faster on every single thing.
link |
And the computer architect says,
link |
well, the new computer will be faster on the average,
link |
but there's a distribution of results and performance,
link |
and you'll have some outliers that are slower.
link |
And that's very hard,
link |
because they have one customer who cares about that one.
link |
So speaking of the long term, for over 50 years now,
link |
Moore's Law has served, for me and millions of others,
link |
as an inspiring beacon of what kind of amazing future
link |
brilliant engineers can build.
link |
I'm just making your kids laugh all of today.
link |
So first, in your eyes, what is Moore's Law,
link |
if you could define for people who don't know?
link |
Well, the simple statement was, from Gordon Moore,
link |
was double the number of transistors every two years.
link |
Something like that.
link |
And then my operational model is,
link |
we increase the performance of computers
link |
by two X every two or three years.
link |
And it's wiggled around substantially over time.
link |
And also, in how we deliver, performance has changed.
link |
But the foundational idea was
link |
two X to transistors every two years.
link |
The current cadence is something like,
link |
they call it a shrink factor, like 0.6 every two years,
link |
But that's referring strictly, again,
link |
to the original definition of just.
link |
A transistor count.
link |
A shrink factor's just getting them
link |
smaller and smaller and smaller.
link |
Well, it's for a constant chip area.
link |
If you make the transistors smaller by 0.6,
link |
then you get one over 0.6 more transistors.
link |
So can you linger on it a little longer?
link |
What's a broader, what do you think should be
link |
the broader definition of Moore's Law?
link |
When you mentioned how you think of performance,
link |
just broadly, what's a good way to think about Moore's Law?
link |
Well, first of all, I've been aware
link |
of Moore's Law for 30 years.
link |
Well, I've been designing computers for 40.
link |
You're just watching it before your eyes kind of thing.
link |
And somewhere where I became aware of it,
link |
I was also informed that Moore's Law
link |
was gonna die in 10 to 15 years.
link |
And then I thought that was true at first.
link |
But then after 10 years, it was gonna die in 10 to 15 years.
link |
And then at one point, it was gonna die in five years.
link |
And then it went back up to 10 years.
link |
And at some point, I decided not to worry
link |
about that particular prognostication
link |
for the rest of my life, which is fun.
link |
And then I joined Intel and everybody said
link |
Moore's Law is dead.
link |
And I thought that's sad,
link |
because it's the Moore's Law company.
link |
And it's not dead.
link |
And it's always been gonna die.
link |
And humans like these apocryphal kind of statements,
link |
like we'll run out of food, or we'll run out of air,
link |
or we'll run out of room, or we'll run out of something.
link |
Right, but it's still incredible
link |
that it's lived for as long as it has.
link |
And yes, there's many people who believe now
link |
that Moore's Law is dead.
link |
You know, they can join the last 50 years
link |
of people who had the same idea.
link |
Yeah, there's a long tradition.
link |
But why do you think, if you can try to understand it,
link |
why do you think it's not dead?
link |
Well, let's just think, people think Moore's Law
link |
is one thing, transistors get smaller.
link |
But actually, under the sheet,
link |
there's literally thousands of innovations.
link |
And almost all those innovations
link |
have their own diminishing return curves.
link |
So if you graph it, it looks like a cascade
link |
of diminishing return curves.
link |
I don't know what to call that.
link |
But the result is an exponential curve.
link |
Well, at least it has been.
link |
So, and we keep inventing new things.
link |
So if you're an expert in one of the things
link |
on a diminishing return curve, right,
link |
and you can see it's plateau,
link |
you will probably tell people, well, this is done.
link |
Meanwhile, some other pile of people
link |
are doing something different.
link |
So that's just normal.
link |
So then there's the observation of
link |
how small could a switching device be?
link |
So a modern transistor is something like
link |
a thousand by a thousand by a thousand atoms, right?
link |
And you get quantum effects down around two to 10 atoms.
link |
So you can imagine the transistor
link |
as small as 10 by 10 by 10.
link |
So that's a million times smaller.
link |
And then the quantum computational people
link |
are working away at how to use quantum effects.
link |
A thousand by a thousand by a thousand.
link |
That's a really clean way of putting it.
link |
Well, a fan, like a modern transistor,
link |
if you look at the fan, it's like 120 atoms wide,
link |
but we can make that thinner.
link |
And then there's a gate wrapped around it,
link |
and then there's spacing.
link |
There's a whole bunch of geometry.
link |
And a competent transistor designer
link |
could count both atoms in every single direction.
link |
Like there's techniques now to already put down atoms
link |
in a single atomic layer.
link |
And you can place atoms if you want to.
link |
It's just from a manufacturing process,
link |
if placing an atom takes 10 minutes
link |
and you need to put 10 to the 23rd atoms together
link |
to make a computer, it would take a long time.
link |
So the methods are both shrinking things
link |
and then coming up with effective ways
link |
to control what's happening.
link |
Manufacture stably and cheaply.
link |
So the innovation stock's pretty broad.
link |
There's equipment, there's optics, there's chemistry,
link |
there's physics, there's material science,
link |
there's metallurgy, there's lots of ideas
link |
about when you put different materials together,
link |
how do they interact, are they stable,
link |
is it stable over temperature, like are they repeatable?
link |
There's like literally thousands of technologies involved.
link |
But just for the shrinking, you don't think
link |
we're quite yet close to the fundamental limits of physics?
link |
I did a talk on Moore's Law and I asked for a roadmap
link |
to a path of 100 and after two weeks,
link |
they said we only got to 50.
link |
We only got to 50.
link |
And I said, why don't you give it another two weeks?
link |
Well, here's the thing about Moore's Law, right?
link |
So I believe that the next 10 or 20 years
link |
of shrinking is gonna happen, right?
link |
Now, as a computer designer, you have two stances.
link |
You think it's going to shrink, in which case
link |
you're designing and thinking about architecture
link |
in a way that you'll use more transistors.
link |
Or conversely, not be swamped by the complexity
link |
of all the transistors you get, right?
link |
You have to have a strategy, you know?
link |
So you're open to the possibility and waiting
link |
for the possibility of a whole new army
link |
of transistors ready to work.
link |
I'm expecting more transistors every two or three years
link |
by a number large enough that how you think about design,
link |
how you think about architecture has to change.
link |
Like, imagine you build buildings out of bricks,
link |
and every year the bricks are half the size,
link |
or every two years.
link |
Well, if you kept building bricks the same way,
link |
so many bricks per person per day,
link |
the amount of time to build a building
link |
would go up exponentially, right?
link |
But if you said, I know that's coming,
link |
so now I'm gonna design equipment that moves bricks faster,
link |
uses them better, because maybe you're getting something
link |
out of the smaller bricks, more strength, thinner walls,
link |
you know, less material, efficiency out of that.
link |
So once you have a roadmap with what's gonna happen,
link |
transistors, we're gonna get more of them,
link |
then you design all this collateral around it
link |
to take advantage of it, and also to cope with it.
link |
Like, that's the thing people don't understand.
link |
It's like, if I didn't believe in Moore's Law,
link |
and then Moore's Law transistors showed up,
link |
my design teams would all drown.
link |
So what's the hardest part of this inflow
link |
of new transistors?
link |
I mean, even if you just look historically,
link |
throughout your career, what's the thing,
link |
what fundamentally changes when you add more transistors
link |
in the task of designing an architecture?
link |
Well, there's two constants, right?
link |
One is people don't get smarter.
link |
By the way, there's some science showing
link |
that we do get smarter because of nutrition or whatever.
link |
Sorry to bring that up.
link |
Yeah, I'm familiar with it.
link |
Nobody understands it, nobody knows if it's still going on.
link |
Or whether it's real or not.
link |
But yeah, it's a...
link |
Anyway, but not exponentially.
link |
I would believe for the most part,
link |
people aren't getting much smarter.
link |
The evidence doesn't support it, that's right.
link |
And then teams can't grow that much.
link |
Right, so human beings, you know,
link |
we're really good in teams of 10,
link |
you know, up to teams of 100, they can know each other.
link |
Beyond that, you have to have organizational boundaries.
link |
So you're kind of, you have,
link |
those are pretty hard constraints, right?
link |
So then you have to divide and conquer,
link |
like as the designs get bigger,
link |
you have to divide it into pieces.
link |
You know, the power of abstraction layers is really high.
link |
We used to build computers out of transistors.
link |
Now we have a team that turns transistors into logic cells
link |
and another team that turns them into functional units,
link |
another one that turns them into computers, right?
link |
So we have abstraction layers in there
link |
and you have to think about when do you shift gears on that.
link |
We also use faster computers to build faster computers.
link |
So some algorithms run twice as fast on new computers,
link |
but a lot of algorithms are N squared.
link |
So, you know, a computer with twice as many transistors
link |
and it might take four times as long to run.
link |
So you have to refactor the software.
link |
Like simply using faster computers
link |
to build bigger computers doesn't work.
link |
So you have to think about all these things.
link |
So in terms of computing performance
link |
and the exciting possibility
link |
that more powerful computers bring,
link |
is shrinking the thing which you've been talking about,
link |
for you, one of the biggest exciting possibilities
link |
of advancement in performance?
link |
Or is there other directions that you're interested in,
link |
like in the direction of sort of enforcing given parallelism
link |
or like doing massive parallelism
link |
in terms of many, many CPUs,
link |
you know, stacking CPUs on top of each other,
link |
that kind of parallelism or any kind of parallelism?
link |
Well, think about it a different way.
link |
So old computers, you know, slow computers,
link |
you said A equal B plus C times D, pretty simple, right?
link |
And then we made faster computers with vector units
link |
and you can do proper equations and matrices, right?
link |
And then modern like AI computations
link |
or like convolutional neural networks,
link |
where you convolve one large data set against another.
link |
And so there's sort of this hierarchy of mathematics,
link |
you know, from simple equation to linear equations,
link |
to matrix equations, to deeper kind of computation.
link |
And the data sets are getting so big
link |
that people are thinking of data as a topology problem.
link |
You know, data is organized in some immense shape.
link |
And then the computation, which sort of wants to be,
link |
get data from immense shape and do some computation on it.
link |
So what computers have allowed people to do
link |
is have algorithms go much, much further.
link |
So that paper you reference, the Sutton paper,
link |
they talked about, you know, like when AI started,
link |
it was apply rule sets to something.
link |
That's a very simple computational situation.
link |
And then when they did first chess thing,
link |
they solved deep searches.
link |
So have a huge database of moves and results, deep search,
link |
but it's still just a search, right?
link |
Now we take large numbers of images
link |
and we use it to train these weight sets
link |
that we convolve across.
link |
It's a completely different kind of phenomena.
link |
Now they're doing the next generation.
link |
And if you look at it,
link |
they're going up this mathematical graph, right?
link |
And then computations, both computation and data sets
link |
support going up that graph.
link |
Yeah, the kind of computation that might,
link |
I mean, I would argue that all of it is still a search,
link |
Just like you said, a topology problem with data sets,
link |
you're searching the data sets for valuable data
link |
and also the actual optimization of neural networks
link |
is a kind of search for the...
link |
I don't know, if you had looked at the interlayers
link |
of finding a cat, it's not a search.
link |
It's a set of endless projections.
link |
So, you know, a projection,
link |
here's a shadow of this phone, right?
link |
And then you can have a shadow of that on the something
link |
and a shadow on that of something.
link |
And if you look in the layers, you'll see
link |
this layer actually describes pointy ears
link |
and round eyeness and fuzziness.
link |
But the computation to tease out the attributes
link |
Like the inference part might be search,
link |
but the training's not search.
link |
And then in deep networks, they look at layers
link |
and they don't even know it's represented.
link |
And yet, if you take the layers out, it doesn't work.
link |
So I don't think it's search.
link |
But you'd have to talk to a mathematician
link |
about what that actually is.
link |
Well, we could disagree, but it's just semantics,
link |
I think, it's not, but it's certainly not...
link |
I would say it's absolutely not semantics, but...
link |
Okay, all right, well, if you want to go there.
link |
So optimization to me is search,
link |
and we're trying to optimize the ability
link |
of a neural network to detect cat ears.
link |
And the difference between chess and the space,
link |
the incredibly multidimensional,
link |
100,000 dimensional space that neural networks
link |
are trying to optimize over is nothing like
link |
the chessboard database.
link |
So it's a totally different kind of thing.
link |
And okay, in that sense, you can say it loses the meaning.
link |
I can see how you might say, if you...
link |
The funny thing is, it's the difference
link |
between given search space and found search space.
link |
Yeah, maybe that's a different way to describe it.
link |
That's a beautiful way to put it, okay.
link |
But you're saying, what's your sense
link |
in terms of the basic mathematical operations
link |
and the architectures, computer hardware
link |
that enables those operations?
link |
Do you see the CPUs of today still being
link |
a really core part of executing
link |
those mathematical operations?
link |
Well, the operations continue to be add, subtract,
link |
load, store, compare, and branch.
link |
So it's interesting, the building blocks
link |
of computers or transistors under that atoms.
link |
So you got atoms, transistors, logic gates, computers,
link |
functional units of computers.
link |
The building blocks of mathematics at some level
link |
are things like adds and subtracts and multiplies,
link |
but the space mathematics can describe
link |
is, I think, essentially infinite.
link |
But the computers that run the algorithms
link |
are still doing the same things.
link |
Now, a given algorithm might say, I need sparse data,
link |
or I need 32 bit data, or I need, you know,
link |
like a convolution operation that naturally takes
link |
eight bit data, multiplies it, and sums it up a certain way.
link |
So like the data types in TensorFlow
link |
imply an optimization set.
link |
But when you go right down and look at the computers,
link |
it's and and or gates doing adds and multiplies.
link |
Like that hasn't changed much.
link |
Now, the quantum researchers think
link |
they're going to change that radically,
link |
and then there's people who think about analog computing
link |
because you look in the brain, and it
link |
seems to be more analogish.
link |
You know, that maybe there's a way to do that more
link |
But we have a million X on computation,
link |
and I don't know the relationship
link |
between computational, let's say,
link |
intensity and ability to hit mathematical abstractions.
link |
I don't know any way to describe that, but just like you saw
link |
in AI, you went from rule sets to simple search
link |
to complex search to, say, found search.
link |
Like those are orders of magnitude more computation
link |
And as we get the next two orders of magnitude,
link |
like a friend, Roger Gaduri, said,
link |
like every order of magnitude changes the computation.
link |
Fundamentally changes what the computation is doing.
link |
Oh, you know the expression the difference in quantity
link |
is the difference in kind.
link |
You know, the difference between ant and anthill, right?
link |
Or neuron and brain.
link |
You know, there's this indefinable place
link |
where the quantity changed the quality, right?
link |
And we've seen that happen in mathematics multiple times,
link |
and you know, my guess is it's going to keep happening.
link |
So your sense is, yeah, if you focus head down
link |
and shrinking the transistor.
link |
Well, it's not just head down, we're aware of the software
link |
stacks that are running in the computational loads,
link |
and we're kind of pondering what do you
link |
do with a petabyte of memory that wants
link |
to be accessed in a sparse way and have, you know,
link |
the kind of calculations AI programmers want.
link |
So there's a dialogue interaction,
link |
but when you go in the computer chip,
link |
you know, you find adders and subtractors and multipliers.
link |
So if you zoom out then with, as you mentioned very sudden,
link |
the idea that most of the development in the last many
link |
decades in AI research came from just leveraging computation
link |
and just simple algorithms waiting for the computation
link |
Well, software guys have a thing that they call it
link |
the problem of early optimization.
link |
So you write a big software stack,
link |
and if you start optimizing like the first thing you write,
link |
the odds of that being the performance limiter is low.
link |
But when you get the whole thing working,
link |
can you make it 2x faster by optimizing the right things?
link |
While you're optimizing that, could you
link |
have written a new software stack, which
link |
would have been a better choice?
link |
Now you have creative tension.
link |
But the whole time as you're doing the writing,
link |
that's the software we're talking about.
link |
The hardware underneath gets faster and faster.
link |
Well, this goes back to the Moore's law.
link |
If Moore's law is going to continue, then your AI research
link |
should expect that to show up, and then you
link |
make a slightly different set of choices then.
link |
We've hit the wall.
link |
Nothing's going to happen.
link |
And from here, it's just us rewriting algorithms.
link |
That seems like a failed strategy for the last 30
link |
years of Moore's law's death.
link |
So can you just linger on it?
link |
I think you've answered it, but I'll just
link |
ask the same dumb question over and over.
link |
So why do you think Moore's law is not going to die?
link |
Which is the most promising, exciting possibility
link |
of why it won't die in the next 5, 10 years?
link |
So is it the continued shrinking of the transistor,
link |
or is it another S curve that steps in and it totally sort
link |
Shrinking the transistor is literally
link |
thousands of innovations.
link |
Right, so there's stacks of S curves in there.
link |
There's a whole bunch of S curves just kind
link |
of running their course and being reinvented
link |
The semiconductor fabricators and technologists have all
link |
announced what's called nanowires.
link |
So they took a fan, which had a gate around it,
link |
and turned that into little wires
link |
so you have better control of that, and they're smaller.
link |
And then from there, there are some obvious steps
link |
about how to shrink that.
link |
The metallurgy around wire stacks and stuff
link |
has very obvious abilities to shrink.
link |
And there's a whole combination of things there to do.
link |
Your sense is that we're going to get a lot
link |
if this innovation performed just that, shrinking.
link |
Yeah, like a factor of 100 is a lot.
link |
Yeah, I would say that's incredible.
link |
And it's totally unknown.
link |
It's only 10 or 15 years.
link |
Now, you're smarter, you might know,
link |
but to me it's totally unpredictable
link |
of what that 100x would bring in terms
link |
of the nature of the computation that people would be.
link |
Yeah, are you familiar with Bell's law?
link |
So for a long time, it was mainframes, minis, workstation,
link |
Moore's law drove faster, smaller computers.
link |
And then when we were thinking about Moore's law,
link |
Rajagaduri said, every 10x generates a new computation.
link |
So scalar, vector, matrix, topological computation.
link |
And if you go look at the industry trends,
link |
there was mainframes, and then minicomputers, and then PCs,
link |
and then the internet took off.
link |
And then we got mobile devices.
link |
And now we're building 5G wireless
link |
with one millisecond latency.
link |
And people are starting to think about the smart world
link |
where everything knows you, recognizes you.
link |
The transformations are going to be unpredictable.
link |
How does it make you feel that you're
link |
one of the key architects of this kind of future?
link |
So we're not talking about the architects
link |
of the high level people who build the Angry Bird apps,
link |
Maybe that's the whole point of the universe.
link |
I'm going to take a stand at that,
link |
and the attention distracting nature of mobile phones.
link |
I'll take a stand.
link |
But anyway, in terms of the side effects of smartphones,
link |
or the attention distraction, which part?
link |
Well, who knows where this is all leading?
link |
It's changing so fast.
link |
My parents used to yell at my sisters
link |
for hiding in the closet with a wired phone with a dial on it.
link |
Stop talking to your friends all day.
link |
Now my wife yells at my kids for talking to their friends
link |
It looks the same to me.
link |
It's always echoes of the same thing.
link |
But you are one of the key people
link |
architecting the hardware of this future.
link |
How does that make you feel?
link |
Do you feel responsible?
link |
Do you feel excited?
link |
So we're in a social context.
link |
So there's billions of people on this planet.
link |
There are literally millions of people working on technology.
link |
I feel lucky to be doing what I do and getting paid for it,
link |
and there's an interest in it.
link |
But there's so many things going on in parallel.
link |
The actions are so unpredictable.
link |
If I wasn't here, somebody else would do it.
link |
The vectors of all these different things
link |
are happening all the time.
link |
You know, there's a, I'm sure, some philosopher
link |
or metaphilosopher is wondering about how
link |
we transform our world.
link |
So you can't deny the fact that these tools are
link |
changing our world.
link |
Do you think it's changing for the better?
link |
I read this thing recently.
link |
It said the two disciplines with the highest GRE scores in college
link |
are physics and philosophy.
link |
And they're both sort of trying to answer the question,
link |
why is there anything?
link |
And the philosophers are on the kind of theological side,
link |
and the physicists are obviously on the material side.
link |
And there's 100 billion galaxies with 100 billion stars.
link |
It seems, well, repetitive at best.
link |
So you know, there's on our way to 10 billion people.
link |
I mean, it's hard to say what it's all for,
link |
if that's what you're asking.
link |
Yeah, I guess I am.
link |
Things do tend to significantly increase in complexity.
link |
And I'm curious about how computation,
link |
like our physical world inherently
link |
generates mathematics.
link |
It's kind of obvious, right?
link |
So we have x, y, z coordinates.
link |
You take a sphere, you make it bigger.
link |
You get a surface that grows by r squared.
link |
Like, it generally generates mathematics.
link |
And the mathematicians and the physicists
link |
have been having a lot of fun talking to each other for years.
link |
And computation has been, let's say, relatively pedestrian.
link |
Like, computation in terms of mathematics
link |
has been doing binary algebra, while those guys have
link |
been gallivanting through the other realms of possibility.
link |
Now recently, the computation lets
link |
you do mathematical computations that
link |
are sophisticated enough that nobody understands
link |
how the answers came out.
link |
It used to be you get data set, you guess at a function.
link |
The function is considered physics
link |
if it's predictive of new functions, new data sets.
link |
Modern, you can take a large data set
link |
with no intuition about what it is
link |
and use machine learning to find a pattern that
link |
has no function, right?
link |
And it can arrive at results that I
link |
don't know if they're completely mathematically describable.
link |
So computation has kind of done something interesting compared
link |
to a equal b plus c.
link |
There's something reminiscent of that step
link |
from the basic operations of addition
link |
to taking a step towards neural networks that's
link |
reminiscent of what life on Earth at its origins was doing.
link |
Do you think we're creating sort of the next step
link |
in our evolution in creating artificial intelligence
link |
systems that will?
link |
I mean, there's so much in the universe already,
link |
Where we stand in this whole thing.
link |
Are human beings working on additional abstraction
link |
layers and possibilities?
link |
Yeah, it appears so.
link |
Does that mean that human beings don't need dogs?
link |
Like, there's so many things that
link |
are all simultaneously interesting and useful.
link |
Well, you've seen, throughout your career,
link |
you've seen greater and greater level abstractions built
link |
in artificial machines, right?
link |
Do you think, when you look at humans,
link |
do you think that the look of all life on Earth
link |
is a single organism building this thing,
link |
this machine with greater and greater levels of abstraction?
link |
Do you think humans are the peak,
link |
the top of the food chain in this long arc of history
link |
Or do you think we're just somewhere in the middle?
link |
Are we the basic functional operations of a CPU?
link |
Are we the C++ program, the Python program,
link |
or the neural network?
link |
Like, somebody's, you know, people
link |
have calculated, like, how many operations does the brain do?
link |
Something, you know, I've seen the number 10 to the 18th
link |
a bunch of times, arrive different ways.
link |
So could you make a computer that
link |
did 10 to the 20th operations?
link |
We're going to do that.
link |
Now, is there something magical about how brains compute things?
link |
You know, my personal experience is interesting,
link |
because, you know, you think you know how you think,
link |
and then you have all these ideas,
link |
and you can't figure out how they happened.
link |
And if you meditate, you know, what you can be aware of
link |
So I don't know if brains are magical or not.
link |
You know, the physical evidence says no.
link |
Lots of people's personal experience says yes.
link |
So what would be funny is if brains are magical,
link |
and yet we can make brains with more computation.
link |
You know, I don't know what to say about that.
link |
But do you think magic is an emergent phenomena?
link |
I have no explanation for it.
link |
Let me ask Jim Keller of what in your view is consciousness?
link |
With consciousness?
link |
Yeah, like what, you know, consciousness, love,
link |
things that are these deeply human things that
link |
seems to emerge from our brain, is that something
link |
that we'll be able to make encode in chips that get
link |
faster and faster and faster and faster?
link |
That's like a 10 hour conversation.
link |
Nobody really knows.
link |
Can you summarize it in a couple of sentences?
link |
Many people have observed that organisms run
link |
at lots of different levels, right?
link |
If you had two neurons, somebody said
link |
you'd have one sensory neuron and one motor neuron, right?
link |
So we move towards things and away from things.
link |
And we have physical integrity and safety or not, right?
link |
And then if you look at the animal kingdom,
link |
you can see brains that are a little more complicated.
link |
And at some point, there's a planning system.
link |
And then there's an emotional system
link |
that's happy about being safe or unhappy about being threatened.
link |
And then our brains have massive numbers of structures,
link |
like planning and movement and thinking and feeling
link |
and drives and emotions.
link |
And we seem to have multiple layers of thinking systems.
link |
And we have a dream system that nobody understands whatsoever,
link |
which I find completely hilarious.
link |
And you can think in a way that those systems are
link |
And you can observe the different parts of yourself
link |
I don't know which one's magical.
link |
I don't know which one's not computational.
link |
Is it possible that it's all computation?
link |
Is there a limit to computation?
link |
Do you think the universe is a computer?
link |
It's a weird kind of computer.
link |
Because if it was a computer, like when
link |
they do calculations on how much calculation
link |
it takes to describe quantum effects, it's unbelievably high.
link |
So if it was a computer, wouldn't you
link |
have built it out of something that was easier to compute?
link |
That's a funny system.
link |
But then the simulation guys pointed out
link |
that the rules are kind of interesting.
link |
When you look really close, it's uncertain.
link |
And the speed of light says you can only look so far.
link |
And things can't be simultaneous,
link |
except for the odd entanglement problem where they seem to be.
link |
The rules are all kind of weird.
link |
And somebody said physics is like having
link |
50 equations with 50 variables to define 50 variables.
link |
Physics itself has been a shit show for thousands of years.
link |
It seems odd when you get to the corners of everything.
link |
It's either uncomputable or undefinable or uncertain.
link |
It's almost like the designers of the simulation
link |
are trying to prevent us from understanding it perfectly.
link |
But also, the things that require calculations
link |
require so much calculation that our idea
link |
of the universe of a computer is absurd,
link |
because every single little bit of it
link |
takes all the computation in the universe to figure out.
link |
So that's a weird kind of computer.
link |
You say the simulation is running
link |
in a computer, which has, by definition, infinite computation.
link |
Oh, you mean if the universe is infinite?
link |
Well, every little piece of our universe
link |
seems to take infinite computation to figure out.
link |
Not infinite, just a lot.
link |
Some pretty big number.
link |
Compute this little teeny spot takes all the mass
link |
in the local one light year by one light year space.
link |
It's close enough to infinite.
link |
Well, it's a heck of a computer if it is one.
link |
It's a weird description, because the simulation
link |
description seems to break when you look closely at it.
link |
But the rules of the universe seem to imply something's up.
link |
That seems a little arbitrary.
link |
The universe, the whole thing, the laws of physics,
link |
it just seems like, how did it come out to be the way it is?
link |
Well, lots of people talk about that.
link |
Like I said, the two smartest groups of humans
link |
are working on the same problem.
link |
From different aspects.
link |
And they're both complete failures.
link |
So that's kind of cool.
link |
They might succeed eventually.
link |
Well, after 2,000 years, the trend isn't good.
link |
Oh, 2,000 years is nothing in the span
link |
of the history of the universe.
link |
We have some time.
link |
But the next 1,000 years doesn't look good either.
link |
That's what everybody says at every stage.
link |
But with Moore's law, as you've just described,
link |
not being dead, the exponential growth of technology,
link |
the future seems pretty incredible.
link |
Well, it'll be interesting, that's for sure.
link |
So what are your thoughts on Ray Kurzweil's sense
link |
that exponential improvement in technology
link |
will continue indefinitely?
link |
Is that how you see Moore's law?
link |
Do you see Moore's law more broadly,
link |
in the sense that technology of all kinds
link |
has a way of stacking S curves on top of each other,
link |
where it'll be exponential, and then we'll see all kinds of...
link |
What does an exponential of a million mean?
link |
That's a pretty amazing number.
link |
And that's just for a local little piece of silicon.
link |
Now let's imagine you, say, decided
link |
to get 1,000 tons of silicon to collaborate in one computer
link |
at a million times the density.
link |
Now you're talking, I don't know, 10 to the 20th more
link |
computation power than our current, already unbelievably
link |
Nobody knows what that's going to mean.
link |
The sci fi guys call it computronium,
link |
like when a local civilization turns the nearby star
link |
I don't know if that's true, but...
link |
So just even when you shrink a transistor, the...
link |
That's only one dimension.
link |
The ripple effects of that.
link |
People tend to think about computers as a cost problem.
link |
So computers are made out of silicon and minor amounts
link |
of metals and this and that.
link |
None of those things cost any money.
link |
There's plenty of sand.
link |
You could just turn the beach and a little bit of ocean water
link |
So all the cost is in the equipment to do it.
link |
And the trend on equipment is once you
link |
figure out how to build the equipment,
link |
the trend of cost is zero.
link |
Elon said, first you figure out what
link |
configuration you want the atoms in,
link |
and then how to put them there.
link |
His great insight is people are how constrained.
link |
I have this thing, I know how it works,
link |
and then little tweaks to that will generate something,
link |
as opposed to what do I actually want,
link |
and then figure out how to build it.
link |
It's a very different mindset.
link |
And almost nobody has it, obviously.
link |
Well, let me ask on that topic,
link |
you were one of the key early people
link |
in the development of autopilot, at least in the hardware
link |
side, Elon Musk believes that autopilot
link |
and vehicle autonomy, if you just look at that problem,
link |
can follow this kind of exponential improvement.
link |
In terms of the how question that we're talking about,
link |
there's no reason why you can't.
link |
What are your thoughts on this particular space
link |
of vehicle autonomy, and your part of it
link |
and Elon Musk's and Tesla's vision for vehicle autonomy?
link |
Well, the computer you need to build is straightforward.
link |
And you could argue, well, does it need to be
link |
two times faster or five times or 10 times?
link |
But that's just a matter of time or price in the short run.
link |
So that's not a big deal.
link |
You don't have to be especially smart to drive a car.
link |
So it's not like a super hard problem.
link |
I mean, the big problem with safety is attention,
link |
which computers are really good at, not skills.
link |
Well, let me push back on one.
link |
You see, everything you said is correct,
link |
but we as humans tend to take for granted
link |
how incredible our vision system is.
link |
So you can drive a car with 20, 50 vision,
link |
and you can train a neural network to extract
link |
the distance of any object in the shape of any surface
link |
from a video and data.
link |
Yeah, but that's really simple.
link |
No, it's not simple.
link |
That's a simple data problem.
link |
It's not, it's not simple.
link |
It's because it's not just detecting objects,
link |
it's understanding the scene,
link |
and it's being able to do it in a way
link |
that doesn't make errors.
link |
So the beautiful thing about the human vision system
link |
and our entire brain around the whole thing
link |
is we're able to fill in the gaps.
link |
It's not just about perfectly detecting cars.
link |
It's inferring the occluded cars.
link |
It's trying to, it's understanding the physics.
link |
I think that's mostly a data problem.
link |
So you think what data would compute
link |
with improvement of computation
link |
with improvement in collection of data?
link |
Well, there is a, you know, when you're driving a car
link |
and somebody cuts you off, your brain has theories
link |
about why they did it.
link |
You know, they're a bad person, they're distracted,
link |
they're dumb, you know, you can listen to yourself, right?
link |
So, you know, if you think that narrative is important
link |
to be able to successfully drive a car,
link |
then current autopilot systems can't do it.
link |
But if cars are ballistic things with tracks
link |
and probabilistic changes of speed and direction,
link |
and roads are fixed and given, by the way,
link |
they don't change dynamically, right?
link |
You can map the world really thoroughly.
link |
You can place every object really thoroughly.
link |
Right, you can calculate trajectories
link |
of things really thoroughly, right?
link |
But everything you said about really thoroughly
link |
has a different degree of difficulty, so.
link |
And you could say at some point,
link |
computer autonomous systems will be way better
link |
at things that humans are lousy at.
link |
Like, they'll be better at attention,
link |
they'll always remember there was a pothole in the road
link |
that humans keep forgetting about,
link |
they'll remember that this set of roads
link |
has these weirdo lines on it
link |
that the computers figured out once,
link |
and especially if they get updates,
link |
so if somebody changes a given,
link |
like, the key to robots and stuff somebody said
link |
is to maximize the givens, right?
link |
So having a robot pick up this bottle cap
link |
is way easier if you put a red dot on the top,
link |
because then you'll have to figure out,
link |
and if you wanna do a certain thing with it,
link |
maximize the givens is the thing.
link |
And autonomous systems are happily maximizing the givens.
link |
Like, humans, when you drive someplace new,
link |
you remember it, because you're processing it
link |
the whole time, and after the 50th time you drove to work,
link |
you get to work, you don't know how you got there, right?
link |
You're on autopilot, right?
link |
Autonomous cars are always on autopilot.
link |
But the cars have no theories about why they got cut off,
link |
or why they're in traffic.
link |
So they also never stop paying attention.
link |
Right, so I tend to believe you do have to have theories,
link |
meta models of other people,
link |
especially with pedestrian cyclists,
link |
but also with other cars.
link |
So everything you said is actually essential to driving.
link |
Driving is a lot more complicated than people realize,
link |
I think, so to push back slightly, but to...
link |
So to cut into traffic, right?
link |
You can't just wait for a gap,
link |
you have to be somewhat aggressive.
link |
You'll be surprised how simple a calculation for that is.
link |
I may be on that particular point,
link |
but there's, maybe I actually have to push back.
link |
I would be surprised.
link |
You know what, yeah, I'll just say where I stand.
link |
I would be very surprised,
link |
but I think you might be surprised how complicated it is.
link |
I tell people, progress disappoints in the short run,
link |
and surprises in the long run.
link |
It's very possible, yeah.
link |
I suspect in 10 years it'll be just taken for granted.
link |
But you're probably right, not look like...
link |
It's gonna be a $50 solution that nobody cares about.
link |
It's like GPSes, like, wow, GPSes.
link |
We have satellites in space
link |
that tell you where your location is.
link |
It was a really big deal, now everything has a GPS in it.
link |
Yeah, that's true, but I do think that systems
link |
that involve human behavior are more complicated
link |
than we give them credit for.
link |
So we can do incredible things with technology
link |
that don't involve humans, but when you...
link |
I think humans are less complicated than people.
link |
You know, frequently ascribed.
link |
We tend to operate out of large numbers of patterns
link |
and just keep doing it over and over.
link |
But I can't trust you because you're a human.
link |
That's something a human would say.
link |
But my hope is on the point you've made is,
link |
even if, no matter who's right,
link |
I'm hoping that there's a lot of things
link |
that humans aren't good at
link |
that machines are definitely good at,
link |
like you said, attention and things like that.
link |
Well, they'll be so much better
link |
that the overall picture of safety and autonomy
link |
will be, obviously cars will be safer,
link |
even if they're not as good at understanding.
link |
I'm a big believer in safety.
link |
I mean, there are already the current safety systems,
link |
like cruise control that doesn't let you run into people
link |
There are so many features
link |
that you just look at the parade of accidents
link |
and knocking off like 80% of them is super doable.
link |
Just to linger on the autopilot team
link |
and the efforts there,
link |
it seems to be that there's a very intense scrutiny
link |
by the media and the public in terms of safety,
link |
the pressure, the bar put before autonomous vehicles.
link |
What are your, sort of as a person there
link |
working on the hardware and trying to build a system
link |
that builds a safe vehicle and so on,
link |
what was your sense about that pressure?
link |
Is it expected of new technology?
link |
Yeah, it seems reasonable.
link |
I was interested, I talked to both American
link |
and European regulators,
link |
and I was worried that the regulations
link |
would write into the rules technology solutions,
link |
like modern brake systems imply hydraulic brakes.
link |
So if you read the regulations,
link |
to meet the letter of the law for brakes,
link |
it sort of has to be hydraulic, right?
link |
And the regulator said they're interested in the use cases,
link |
like a head on crash, an offset crash,
link |
don't hit pedestrians, don't run into people,
link |
don't leave the road, don't run a red light or a stoplight.
link |
They were very much into the scenarios.
link |
And they had all the data about which scenarios
link |
injured or killed the most people.
link |
And for the most part, those conversations were like,
link |
what's the right thing to do to take the next step?
link |
Now, Elon's very interested also in the benefits
link |
of autonomous driving or freeing people's time
link |
and attention, as well as safety.
link |
And I think that's also an interesting thing,
link |
but building autonomous systems so they're safe
link |
and safer than people seemed,
link |
since the goal is to be 10X safer than people,
link |
having the bar to be safer than people
link |
and scrutinizing accidents seems philosophically correct.
link |
So I think that's a good thing.
link |
What are, is different than the things you worked at,
link |
Intel, AMD, Apple, with autopilot chip design
link |
and hardware design, what are interesting
link |
or challenging aspects of building this specialized
link |
kind of computing system in the automotive space?
link |
I mean, there's two tricks to building
link |
like an automotive computer.
link |
One is the software team, the machine learning team
link |
is developing algorithms that are changing fast.
link |
So as you're building the accelerator,
link |
you have this, you know, worry or intuition
link |
that the algorithms will change enough
link |
that the accelerator will be the wrong one, right?
link |
And there's the generic thing, which is,
link |
if you build a really good general purpose computer,
link |
say its performance is one, and then GPU guys
link |
will deliver about 5X to performance
link |
for the same amount of silicon,
link |
because instead of discovering parallelism,
link |
you're given parallelism.
link |
And then special accelerators get another two to 5X
link |
on top of a GPU, because you say,
link |
I know the math is always eight bit integers
link |
into 32 bit accumulators, and the operations
link |
are the subset of mathematical possibilities.
link |
So AI accelerators have a claimed performance benefit
link |
over GPUs because in the narrow math space,
link |
you're nailing the algorithm.
link |
Now, you still try to make it programmable,
link |
but the AI field is changing really fast.
link |
So there's a, you know, there's a little
link |
creative tension there of, I want the acceleration
link |
afforded by specialization without being over specialized
link |
so that the new algorithm is so much more effective
link |
that you'd have been better off on a GPU.
link |
So there's a tension there.
link |
To build a good computer for an application
link |
like automotive, there's all kinds of sensor inputs
link |
and safety processors and a bunch of stuff.
link |
So one of Elon's goals is to make it super affordable.
link |
So every car gets an autopilot computer.
link |
So some of the recent startups you look at,
link |
and they have a server in the trunk,
link |
because they're saying, I'm gonna build
link |
this autopilot computer, replaces the driver.
link |
So their cost budget's 10 or $20,000.
link |
And Elon's constraint was, I'm gonna put one in every car,
link |
whether people buy autonomous driving or not.
link |
So the cost constraint he had in mind was great, right?
link |
And to hit that, you had to think about the system design.
link |
That's complicated, and it's fun.
link |
You know, it's like, it's like, it's craftsman's work.
link |
Like, you know, a violin maker, right?
link |
You can say, Stradivarius is this incredible thing,
link |
the musicians are incredible.
link |
But the guy making the violin, you know,
link |
picked wood and sanded it, and then he cut it,
link |
you know, and he glued it, you know,
link |
and he waited for the right day
link |
so that when he put the finish on it,
link |
it didn't, you know, do something dumb.
link |
That's craftsman's work, right?
link |
You may be a genius craftsman
link |
because you have the best techniques
link |
and you discover a new one,
link |
but most engineers, craftsman's work.
link |
And humans really like to do that.
link |
You know the expression?
link |
I used to, I dug ditches when I was in college.
link |
I got really good at it.
link |
Digging ditches is also craftsman's work.
link |
So there's an expression called complex mastery behavior.
link |
So when you're learning something,
link |
that's fine, because you're learning something.
link |
When you do something, it's relatively simple.
link |
It's not that satisfying.
link |
But if the steps that you have to do are complicated
link |
and you're good at them, it's satisfying to do them.
link |
And then if you're intrigued by it all,
link |
as you're doing them, you sometimes learn new things
link |
that you can raise your game.
link |
But craftsman's work is good.
link |
And engineers, like engineering is complicated enough
link |
that you have to learn a lot of skills.
link |
And then a lot of what you do is then craftsman's work,
link |
Autonomous driving, building a very resource
link |
constrained computer.
link |
So a computer has to be cheap enough
link |
to put in every single car.
link |
That essentially boils down to craftsman's work.
link |
It's engineering, it's innovation.
link |
Yeah, you know, there's thoughtful decisions
link |
and problems to solve and trade offs to make.
link |
Do you need 10 camera and ports or eight?
link |
You know, you're building for the current car
link |
You know, how do you do the safety stuff?
link |
You know, there's a whole bunch of details.
link |
It's not like I'm building a new type of neural network,
link |
which has a new mathematics and a new computer to work.
link |
You know, that's like, there's more invention than that.
link |
But the rejection to practice,
link |
once you pick the architecture, you look inside
link |
and what do you see?
link |
Adders and multipliers and memories and, you know,
link |
So computers is always this weird set of abstraction layers
link |
of ideas and thinking that reduction to practice
link |
is transistors and wires and, you know, pretty basic stuff.
link |
And that's an interesting phenomenon.
link |
By the way, like factory work,
link |
like lots of people think factory work
link |
is road assembly stuff.
link |
I've been on the assembly line.
link |
Like the people who work there really like it.
link |
It's a really great job.
link |
It's really complicated.
link |
Putting cars together is hard, right?
link |
And the car is moving and the parts are moving
link |
and sometimes the parts are damaged
link |
and you have to coordinate putting all the stuff together
link |
and people are good at it.
link |
They're good at it.
link |
And I remember one day I went to work
link |
and the line was shut down for some reason
link |
and some of the guys sitting around were really bummed
link |
because they had reorganized a bunch of stuff
link |
and they were gonna hit a new record
link |
for the number of cars built that day.
link |
And they were all gung ho to do it.
link |
And these were big, tough buggers.
link |
And, you know, but what they did was complicated
link |
and you couldn't do it.
link |
Well, after a while you could,
link |
but you'd have to work your way up
link |
because, you know, like putting the bright,
link |
what's called the brights, the trim on a car
link |
on a moving assembly line
link |
where it has to be attached 25 places
link |
in a minute and a half is unbelievably complicated.
link |
And human beings can do it, it's really good.
link |
I think that's harder than driving a car, by the way.
link |
Putting together, working at a.
link |
Working on a factory.
link |
Two smart people can disagree.
link |
I think driving a car.
link |
We'll get you in the factory someday
link |
and then we'll see how you do.
link |
No, not for us humans driving a car is easy.
link |
I'm saying building a machine that drives a car
link |
Driving a car is easy for humans
link |
because we've been evolving for billions of years.
link |
Yeah, I noticed that.
link |
The pale of the cars are super cool.
link |
No, now you join the rest of the internet
link |
I wasn't mocking, I was just.
link |
Intrigued by your anthropology.
link |
I'll have to go dig into that.
link |
There's some inaccuracies there, yes.
link |
Okay, but in general,
link |
what have you learned in terms of
link |
thinking about passion, craftsmanship,
link |
The whole mess of it.
link |
What have you learned, have taken away from your time
link |
working with Elon Musk, working at Tesla,
link |
which is known to be a place of chaos innovation,
link |
craftsmanship, and all of those things.
link |
I really like the way you thought.
link |
You think you have an understanding
link |
about what first principles of something is,
link |
and then you talk to Elon about it,
link |
and you didn't scratch the surface.
link |
He has a deep belief that no matter what you do,
link |
it's a local maximum, right?
link |
And I had a friend, he invented a better electric motor,
link |
and it was a lot better than what we were using.
link |
And one day he came by, he said,
link |
I'm a little disappointed, because this is really great,
link |
and you didn't seem that impressed.
link |
And I said, when the super intelligent aliens come,
link |
are they going to be looking for you?
link |
Like, where is he?
link |
The guy who built the motor.
link |
You know, like, but doing interesting work
link |
that's both innovative and, let's say,
link |
craftsman's work on the current thing
link |
is really satisfying, and it's good.
link |
And then Elon was good at taking everything apart,
link |
and like, what's the deep first principle?
link |
Oh, no, what's really, no, what's really?
link |
You know, that ability to look at it without assumptions
link |
and how constraints is super wild.
link |
You know, he built a rocket ship, and an electric car,
link |
and you know, everything.
link |
And that's super fun, and he's into it, too.
link |
Like, when they first landed two SpaceX rockets at Tesla,
link |
we had a video projector in the big room,
link |
and like, 500 people came down,
link |
and when they landed, everybody cheered,
link |
and some people cried.
link |
All right, but how did you do that?
link |
Well, it was super hard, and then people say,
link |
well, it's chaotic, really?
link |
To get out of all your assumptions,
link |
you think that's not gonna be unbelievably painful?
link |
And is Elon tough?
link |
Do people look back on it and say,
link |
boy, I'm really happy I had that experience
link |
to go take apart that many layers of assumptions?
link |
Sometimes super fun, sometimes painful.
link |
So it could be emotionally and intellectually painful,
link |
that whole process of just stripping away assumptions.
link |
Yeah, imagine 99% of your thought process
link |
is protecting your self conception,
link |
and 98% of that's wrong.
link |
Now you got the math right.
link |
How do you think you're feeling
link |
when you get back into that one bit that's useful,
link |
and now you're open,
link |
and you have the ability to do something different?
link |
I don't know if I got the math right.
link |
It might be 99.9, but it ain't 50.
link |
Imagining it, the 50% is hard enough.
link |
Now, for a long time, I've suspected you could get better.
link |
Like you can think better, you can think more clearly,
link |
you can take things apart.
link |
And there's lots of examples of that, people who do that.
link |
And Nilan is an example of that, you are an example.
link |
I don't know if I am, I'm fun to talk to.
link |
I've learned a lot of stuff.
link |
Well, here's the other thing, I joke, like I read books,
link |
and people think, oh, you read books.
link |
Well, no, I've read a couple of books a week for 55 years.
link |
because I didn't learn to read until I was age or something.
link |
And it turns out when people write books,
link |
they often take 20 years of their life
link |
where they passionately did something,
link |
reduce it to 200 pages.
link |
That's kind of fun.
link |
And then you go online,
link |
and you can find out who wrote the best books
link |
and who liked, you know, that's kind of wild.
link |
So there's this wild selection process,
link |
and then you can read it,
link |
and for the most part, understand it.
link |
And then you can go apply it.
link |
Like I went to one company,
link |
I thought, I haven't managed much before.
link |
So I read 20 management books,
link |
and I started talking to them,
link |
and basically compared to all the VPs running around,
link |
I'd read 19 more management books than anybody else.
link |
It wasn't even that hard.
link |
And half the stuff worked, like first time.
link |
It wasn't even rocket science.
link |
But at the core of that is questioning the assumptions,
link |
or sort of entering the thinking,
link |
first principles thinking,
link |
sort of looking at the reality of the situation,
link |
and using that knowledge, applying that knowledge.
link |
So I would say my brain has this idea
link |
that you can question first assumptions.
link |
But I can go days at a time and forget that,
link |
and you have to kind of like circle back that observation.
link |
Because it is emotionally challenging.
link |
Well, it's hard to just keep it front and center,
link |
because you operate on so many levels all the time,
link |
and getting this done takes priority,
link |
or being happy takes priority,
link |
or screwing around takes priority.
link |
Like how you go through life is complicated.
link |
And then you remember, oh yeah,
link |
I could really think first principles.
link |
Oh shit, that's tiring.
link |
But you do for a while, and that's kind of cool.
link |
So just as a last question in your sense,
link |
from the big picture, from the first principles,
link |
do you think, you kind of answered it already,
link |
but do you think autonomous driving is something
link |
we can solve on a timeline of years?
link |
So one, two, three, five, 10 years,
link |
as opposed to a century?
link |
Just to linger on it a little longer,
link |
where's the confidence coming from?
link |
Is it the fundamentals of the problem,
link |
the fundamentals of building the hardware and the software?
link |
As a computational problem, understanding ballistics,
link |
roles, topography, it seems pretty solvable.
link |
And you can see this, like speech recognition,
link |
for a long time people are doing frequency
link |
and domain analysis, and all kinds of stuff,
link |
and that didn't work at all, right?
link |
And then they did deep learning about it,
link |
and it worked great.
link |
And it took multiple iterations.
link |
And autonomous driving is way past
link |
the frequency analysis point.
link |
Use radar, don't run into things.
link |
And the data gathering's going up,
link |
and the computation's going up,
link |
and the algorithm understanding's going up,
link |
and there's a whole bunch of problems
link |
getting solved like that.
link |
The data side is really powerful,
link |
but I disagree with both you and Elon.
link |
I'll tell Elon once again, as I did before,
link |
that when you add human beings into the picture,
link |
it's no longer a ballistics problem.
link |
It's something more complicated,
link |
but I could be very well proven wrong.
link |
Cars are highly damped in terms of rate of change.
link |
Like the steering system's really slow
link |
compared to a computer.
link |
The acceleration of the acceleration's really slow.
link |
Yeah, on a certain timescale, on a ballistics timescale,
link |
but human behavior, I don't know.
link |
Human beings are really slow too.
link |
Weirdly, we operate half a second behind reality.
link |
Nobody really understands that one either.
link |
It's pretty funny.
link |
We very well could be surprised,
link |
and I think with the rate of improvement
link |
in all aspects on both the compute
link |
and the software and the hardware,
link |
there's gonna be pleasant surprises all over the place.
link |
Speaking of unpleasant surprises,
link |
many people have worries about a singularity
link |
in the development of AI.
link |
Forgive me for such questions.
link |
When AI improves the exponential
link |
and reaches a point of superhuman level
link |
general intelligence, beyond the point,
link |
there's no looking back.
link |
Do you share this worry of existential threats
link |
from artificial intelligence,
link |
from computers becoming superhuman level intelligent?
link |
We already have a very stratified society,
link |
and then if you look at the whole animal kingdom
link |
of capabilities and abilities and interests,
link |
and smart people have their niche,
link |
and normal people have their niche,
link |
and craftsmen have their niche,
link |
and animals have their niche.
link |
I suspect that the domains of interest
link |
for things that are astronomically different,
link |
like the whole something got 10 times smarter than us
link |
and wanted to track us all down because what?
link |
We like to have coffee at Starbucks?
link |
Like, it doesn't seem plausible.
link |
No, is there an existential problem
link |
that how do you live in a world
link |
where there's something way smarter than you,
link |
and you based your kind of self esteem
link |
on being the smartest local person?
link |
Well, there's what, 0.1% of the population who thinks that?
link |
Because the rest of the population's been dealing with it
link |
since they were born.
link |
So the breadth of possible experience
link |
that can be interesting is really big.
link |
And, you know, superintelligence seems likely,
link |
although we still don't know if we're magical,
link |
but I suspect we're not.
link |
And it seems likely that it'll create possibilities
link |
that are interesting for us,
link |
and its interests will be interesting for that,
link |
for whatever it is.
link |
It's not obvious why its interests would somehow
link |
want to fight over some square foot of dirt,
link |
or, you know, whatever the usual fears are about.
link |
So you don't think it'll inherit
link |
some of the darker aspects of human nature?
link |
Depends on how you think reality's constructed.
link |
So for whatever reason,
link |
human beings are in, let's say,
link |
creative tension and opposition
link |
with both our good and bad forces.
link |
Like, there's lots of philosophical understanding of that.
link |
I don't know why that would be different.
link |
So you think the evil is necessary for the good?
link |
I mean, the tension.
link |
I don't know about evil,
link |
but like we live in a competitive world
link |
where your good is somebody else's evil.
link |
You know, there's the malignant part of it,
link |
but that seems to be self limiting,
link |
although occasionally it's super horrible.
link |
But yes, there's a debate over ideas,
link |
and some people have different beliefs,
link |
and that debate itself is a process.
link |
So the arriving at something.
link |
Yeah, and why wouldn't that continue?
link |
But you don't think that whole process
link |
will leave humans behind in a way that's painful?
link |
Emotionally painful, yes.
link |
For the 0.1%, they'll be.
link |
Why isn't it already painful
link |
for a large percentage of the population?
link |
I mean, society does have a lot of stress in it,
link |
about the 1%, and about the this, and about the that,
link |
but you know, everybody has a lot of stress in their life
link |
about what they find satisfying,
link |
and you know, know yourself seems to be the proper dictum,
link |
and pursue something that makes your life meaningful
link |
seems proper, and there's so many avenues on that.
link |
Like, there's so much unexplored space
link |
at every single level, you know.
link |
I'm somewhat of, my nephew called me a jaded optimist.
link |
And you know, so it's.
link |
There's a beautiful tension in that label,
link |
but if you were to look back at your life,
link |
and could relive a moment, a set of moments,
link |
because there were the happiest times of your life,
link |
outside of family, what would that be?
link |
I don't want to relive any moments.
link |
I like that situation where you have some amount of optimism
link |
and then the anxiety of the unknown.
link |
So you love the unknown, the mystery of it.
link |
I don't know about the mystery.
link |
It sure gets your blood pumping.
link |
What do you think is the meaning of this whole thing?
link |
Of life, on this pale blue dot?
link |
It seems to be what it does.
link |
Like, the universe, for whatever reason,
link |
makes atoms, which makes us, which we do stuff.
link |
And we figure out things, and we explore things, and.
link |
That's just what it is.
link |
Jim, I don't think there's a better place to end it
link |
is a huge honor, and.
link |
Well, that was super fun.
link |
Thank you so much for talking today.
link |
Thanks for listening to this conversation,
link |
and thank you to our presenting sponsor, Cash App.
link |
Download it, use code LexPodcast.
link |
You'll get $10, and $10 will go to FIRST,
link |
a STEM education nonprofit that inspires hundreds
link |
of thousands of young minds to become future leaders
link |
If you enjoy this podcast, subscribe on YouTube.
link |
Give it five stars on Apple Podcast.
link |
Follow on Spotify, support it on Patreon,
link |
or simply connect with me on Twitter.
link |
And now, let me leave you with some words of wisdom
link |
from Gordon Moore.
link |
If everything you try works,
link |
you aren't trying hard enough.
link |
Thank you for listening, and hope to see you next time.