back to index

Jim Keller: Moore's Law, Microprocessors, and First Principles | Lex Fridman Podcast #70


small model | large model

link |
00:00:00.000
The following is a conversation with Jim Keller, legendary microprocessor engineer who has worked at AMD, Apple, Tesla, and now Intel.
link |
00:00:10.000
He's known for his work on AMD K7, K8, K12, and Zen microarchitectures, Apple A4 and A5 processors,
link |
00:00:18.000
and coauthor of the specification for the X8664 instruction set and hypertransport interconnect.
link |
00:00:26.000
He's a brilliant first principles engineer and out of the box thinker and just an interesting and fun human being to talk to.
link |
00:00:33.000
This is the Artificial Intelligence Podcast.
link |
00:00:36.000
If you enjoy it, subscribe on YouTube, give it 5 stars on Apple Podcast, follow on Spotify, support it on Patreon, or simply connect with me on Twitter.
link |
00:00:45.000
Alex Friedman, spelled F R I D M A N.
link |
00:00:49.000
I recently started doing ads at the end of the introduction. I'll do one or two minutes after introducing the episode and never any ads in the middle that can break the flow of the conversation.
link |
00:00:59.000
I hope that works for you. It doesn't hurt the listening experience.
link |
00:01:03.000
This show is presented by Cash App, the number one finance app in the App Store.
link |
00:01:08.000
I personally use Cash App to send money to friends, but you can also use it to buy, sell, and deposit Bitcoin in just seconds.
link |
00:01:15.000
Cash App also has a new investing feature. You can buy fractions of a stock, say $1 worth, no matter what the stock price is.
link |
00:01:23.000
Roker's services are provided by Cash App Investing, a subsidiary of Square and member SIPC.
link |
00:01:29.000
I'm excited to be working with Cash App to support one of my favorite organizations called FIRST, best known for their first robotics and legal competitions.
link |
00:01:38.000
They educate and inspire hundreds of thousands of students in over 110 countries and have a perfect rating on charity navigator, which means that donated money is used to maximum effectiveness.
link |
00:01:50.000
When you get Cash App from the App Store, Google Play, and use code LEX
link |
00:01:38.000
You'll get $10 and Cash App will also donate $10 to FIRST, which again is an organization that I've personally seen inspire girls and boys to dream of engineering a better world.
link |
00:02:08.000
And now, here's my conversation with Jim Keller.
link |
00:02:12.000
What are the differences and similarities between the human brain and a computer with the microprocessor at its core? Let's start with the philosophical question, perhaps.
link |
00:02:22.000
Well, since people don't actually understand how human brains work, I think that's true.
link |
00:02:29.000
I think that's true.
link |
00:02:30.000
So it's hard to compare them.
link |
00:02:54.000
Computers are, you know, there's really two things.
link |
00:02:58.000
You think in the human brain, everything's a mesh, a mess that's combined together.
link |
00:03:24.000
I don't know that the understanding of that is super deep.
link |
00:03:50.000
What is a microprocessor, what is a microarchitecture? What's an instruction set architecture?
link |
00:04:18.000
transistors. On top of that, we build logic gates, right, and
link |
00:04:23.840
then functional units, like an adder, a subtractor, an
link |
00:04:27.480
instruction parsing unit, and then we assemble those into,
link |
00:04:31.000
you know, processing elements, modern computers are built out
link |
00:04:34.040
of, you know, probably 10 to 20 locally, you know, organic
link |
00:04:39.840
processing elements or coherent processing elements, and then
link |
00:04:42.880
that runs computer programs. Right. So there's abstraction
link |
00:04:47.560
layers, and then software, you know, there's an instruction set
link |
00:04:50.840
you run. And then there's assembly language C, C plus
link |
00:04:54.360
plus Java JavaScript, you know, there's abstraction layers,
link |
00:04:58.200
you know, essentially from the atom to the data center. Right.
link |
00:05:02.440
So when you when you build a computer, you know, first, there's
link |
00:05:07.160
a target like what's it for, like how fast does it have to be,
link |
00:05:09.920
which, you know, today, there's a whole bunch of metrics about
link |
00:05:12.480
what that is. And then in an organization of, you know, 1000
link |
00:05:16.720
people who build a computer, there's lots of different
link |
00:05:20.600
disciplines that you have to operate on. Does that make sense?
link |
00:05:25.480
And so
link |
00:05:27.160
there's a bunch of levels of abstraction of in an organization
link |
00:05:32.400
I can tell, and in your own vision, there's a lot of
link |
00:05:36.320
brilliance that comes in at every one of those layers. Some of
link |
00:05:40.080
it is science, some of it is engineering, some of it is art.
link |
00:05:42.760
What's the most, if you could pick favorites, what's the most
link |
00:05:46.360
important, your favorite layer on these layers of abstractions?
link |
00:05:50.480
Where does the magic enter this hierarchy?
link |
00:05:54.760
I don't really care. That's the fun, you know, I'm somewhat
link |
00:05:58.920
agnostic to that. So I would say, for relatively long periods
link |
00:06:04.480
of time, instruction sets are stable. So the x86 instruction
link |
00:06:09.160
set, the arm instruction set.
link |
00:06:10.720
What's an instruction set? So it says, how do you encode the
link |
00:06:14.320
basic operations, load, store, multiply, add, subtract,
link |
00:06:17.440
conditional branch, you know, there aren't that many
link |
00:06:21.440
interesting instructions. Look, if you look at a program and it
link |
00:06:24.240
runs, you know, 90% of the execution is on 25 op codes,
link |
00:06:28.960
you know, 25 instructions on those are stable. Right?
link |
00:06:32.960
What does it mean stable?
link |
00:06:34.320
Intel architecture has been around for 25 years.
link |
00:06:36.920
It works. It works. And that's because the
link |
00:06:40.000
basics, you know, are defined a long time ago. Right? Now, the
link |
00:06:45.520
way an old computer ran is you fetched instructions and you
link |
00:06:49.480
executed them in order. Do the load, do the add, do the
link |
00:06:54.720
compare. The way a modern computer works is you fetch large
link |
00:06:59.440
numbers of instructions, say 500. And then you find the
link |
00:07:04.480
dependency graph between the instructions. And then you execute
link |
00:07:09.520
in independent units, those little micrographs. So a modern
link |
00:07:15.440
computer, like people like to say computers should be simple
link |
00:07:18.400
and clean. But it turns out the market for a simple complete
link |
00:07:22.320
clean slow computers is zero. Right? We don't sell any simple
link |
00:07:27.040
clean computers. Now you can, there's how you build it can
link |
00:07:31.840
be clean, but the computer people want to buy, let's say in a
link |
00:07:36.960
phone or data center, fetches a large number of instructions,
link |
00:07:42.320
computes the dependency graph, and then executes it in a way
link |
00:07:47.040
that gets the right answers.
link |
00:07:48.720
And optimize that graph somehow.
link |
00:07:50.400
Yeah, they run deeply out of order. And then there's
link |
00:07:54.480
semantics around how memory ordering works and other
link |
00:07:57.440
things work. So the computer sort of has a bunch of
link |
00:08:00.400
bookkeeping tables that says what order cities operations
link |
00:08:04.160
should finish in or appear to finish in. But to go fast, you
link |
00:08:09.200
have to fetch a lot of instructions and find all the
link |
00:08:11.840
parallelism. Now there's a second kind of computer, which we
link |
00:08:16.160
call GPUs today. And I call it the difference. There's found
link |
00:08:20.640
parallelism, like you have a program with a lot of
link |
00:08:22.800
dependent instructions, you fetch a bunch and then you go
link |
00:08:26.000
figure out the dependency graph and you issues instructions
link |
00:08:28.640
at order. That's because you have one serial narrative to
link |
00:08:32.240
use, which in fact is in can be done out of order.
link |
00:08:36.000
Did you call it a narrative? Yeah.
link |
00:08:37.840
Wow. So yeah, so humans think of serial narrative. So read a
link |
00:08:41.440
book, right? There's a you know, there's a sentence after
link |
00:08:44.480
sentence after sentence, and there's paragraphs. Now you
link |
00:08:47.280
could diagram that. Imagine you diagrammed it properly and
link |
00:08:51.200
you said, which sentences could be read in anti order, any
link |
00:08:56.080
order without changing the meaning, right?
link |
00:09:00.080
It's a fascinating question to ask of a book. Yeah. Yeah, you
link |
00:09:02.800
could do that. Right? So some paragraphs could be
link |
00:09:05.600
reordered, some sentences can be reordered. You could say he
link |
00:09:10.400
is tall and smart and X, right? And it doesn't matter the order
link |
00:09:17.200
of tall and smart. But if you say the tall man is wearing a
link |
00:09:22.160
red shirt, what colors, you know, like you can create
link |
00:09:26.240
dependencies, right? Right. And so GPUs, on the other hand,
link |
00:09:31.920
run simple programs on pixels, but you're given a million of
link |
00:09:36.480
them. And the first order, the screen you're looking at
link |
00:09:40.080
doesn't care which order you do it in. So I call that given
link |
00:09:43.280
parallelism, simple narratives around the large numbers of
link |
00:09:46.880
things where you can just say it's parallel because you told
link |
00:09:51.360
me it was. So found parallelism where the narrative
link |
00:09:55.920
is sequential, but you discover like little pockets
link |
00:10:00.160
of parallelism versus. Turns out large pockets of
link |
00:10:02.880
parallelism. Large. So how hard is it to discover?
link |
00:10:05.840
Well, how hard is it? That's just transistor count, right?
link |
00:10:08.720
So once you crack the problem, you say here's how you fetch
link |
00:10:12.000
10 instructions at a time, here's how you calculate the
link |
00:10:14.720
dependencies between them, here's how you describe the
link |
00:10:17.360
dependencies, here's, you know, these are pieces, right?
link |
00:10:21.680
So, Anna, once you describe the dependencies, then it's just
link |
00:10:26.080
a graph, sort of, it's an algorithm that finds, what is
link |
00:10:31.520
that? I'm sure there's a graph theory, a theoretical answer
link |
00:10:34.320
here that's solvable. In general, programs, modern programs
link |
00:10:40.560
that human beings write, how much found parallelism is
link |
00:10:44.320
there in them? About 10x. What does 10x mean?
link |
00:10:47.120
Well, you execute it in order. Versus, yeah. You would get
link |
00:10:52.000
what's called cycles per instruction and it would be
link |
00:10:54.480
about, you know, three instructions, three cycles
link |
00:10:58.640
per instruction because of the latency of the operations
link |
00:11:01.520
and stuff. And in a modern computer, execute it like
link |
00:11:05.440
.2,.25 cycles per instruction. So it's about,
link |
00:11:09.600
we today find 10x. And there's two things. One is
link |
00:11:14.000
the found parallelism in the narrative, right? And the other
link |
00:11:17.840
is the predictability of the narrative, right? So certain
link |
00:11:23.120
operations, they do a bunch of calculations and if greater
link |
00:11:26.320
than one, do this, else do that. That, that decision is
link |
00:11:31.360
predicted in modern computers to high 90% accuracy. So
link |
00:11:37.120
branches happen a lot. So imagine you have, you have a
link |
00:11:39.920
decision to make every six instructions, which is about
link |
00:11:42.160
the average, right? But you want to fetch 500
link |
00:11:44.800
instructions, figure out the graph and execute them all
link |
00:11:46.880
in parallel. That means you have, let's say, if you
link |
00:11:51.680
affect 600 instructions and it's every six, you have to
link |
00:11:55.440
fetch, you have to predict 99 out of 100 branches
link |
00:11:58.640
correctly for that window to be effective.
link |
00:12:02.240
Okay. So parallelism, you can't parallelize branches
link |
00:12:06.640
or you can. You can predict. What does predict a branch
link |
00:12:10.080
mean? What's predicted? So imagine you do a computation
link |
00:12:12.880
over and over. You're in a loop. Yep. So while n is greater
link |
00:12:16.800
than one, do. And you go through that loop a million
link |
00:12:20.640
times. So every time you look at the branch, you say, it's
link |
00:12:23.440
probably still greater than one. And you're saying you could
link |
00:12:26.320
do that accurately. Very accurately. Modern computer. My
link |
00:12:29.280
mind is blown. How the heck do you do that? Wait a minute.
link |
00:12:32.480
Well, you want to know? This is really sad. 20 years ago.
link |
00:12:36.720
Yes. You simply recorded which way the branch went last time
link |
00:12:40.560
and predicted the same thing. Right. Okay. What's the accuracy
link |
00:12:45.040
of that? 85%. So then somebody said, hey, let's keep a
link |
00:12:50.160
couple of bits and have a little counter. So when it
link |
00:12:53.440
predicts one way, we count up and then pins. So say you have
link |
00:12:57.280
a three bit counter. So you count up and then you count
link |
00:13:00.320
down. And if it's, you know, you can use the top bit as a
link |
00:13:02.720
signed bit. So you have a signed two bit number. So if it's
link |
00:13:05.360
greater than one, you predict taken and less than one, you
link |
00:13:09.200
predict not taken, right? Or less than zero, whatever the
link |
00:13:12.640
thing is. And that got us to 92%. Oh. Okay, you know, it's
link |
00:13:18.160
better. This branch depends on how you got there. So if you
link |
00:13:23.280
came down the code one way, you're talking about Bob and
link |
00:13:26.720
Jane, right? And then said is just Bob like Jane, it went
link |
00:13:30.720
one way. But if you're talking about Bob and Jill, just Bob
link |
00:13:33.280
like Jane, you go a different way, right? So that's called
link |
00:13:36.240
history. So you take the history and a counter. That's
link |
00:13:40.080
cool. But that's not how anything works today. They
link |
00:13:43.440
use something that looks a little like a neural network. So
link |
00:13:48.160
modern, you take all the execution flows. And then you
link |
00:13:53.200
do basically deep pattern recognition of how the program
link |
00:13:57.520
is executing. And you do that multiple different ways. And
link |
00:14:03.840
you have something that chooses what the best result is.
link |
00:14:07.440
There's a little supercomputer inside the computer. That's
link |
00:14:10.640
trying to predict that calculates which way branches go. So
link |
00:14:14.400
the effective window that it's worth finding grass and gets
link |
00:14:17.600
bigger. Why was that gonna make me sad? Because that's
link |
00:14:22.240
amazing. It's amazingly complicated. Oh, well, here's
link |
00:14:25.680
the here's the funny thing. So to get to 85% took a
link |
00:14:30.160
thousand bits. To get to 99% takes tens of megabits. So
link |
00:14:38.960
this is one of those to get the result, you know, to get
link |
00:14:42.960
from a window of say, 50 instructions to 500. It took
link |
00:14:48.080
three orders of magnitude or four orders of magnitude
link |
00:14:50.560
toward bits. Now, if you get the prediction of a branch
link |
00:14:54.320
wrong, what happens then flush the pipe flush the pipe. So
link |
00:14:58.080
it's just a performance cost. But it gets even better. Yeah.
link |
00:15:01.280
So we're starting to look at stuff that says so executed
link |
00:15:04.640
down this path. And then you had two ways to go. But far, far
link |
00:15:10.720
away, there's something that doesn't matter which path
link |
00:15:12.960
you went. So you missed you took the wrong path, you
link |
00:15:17.680
executed a bunch of stuff. Then you had the
link |
00:15:21.040
mispredicting to backed it up. But you remembered all the
link |
00:15:23.360
results you already calculated. Some of those are just
link |
00:15:26.480
fine. Like if you read a book and you misunderstand a
link |
00:15:29.600
paragraph, your understanding of the next paragraph,
link |
00:15:32.480
sometimes is invariant to the understanding. Sometimes it
link |
00:15:36.640
depends on it. And you can kind of anticipate that
link |
00:15:41.760
invariance. Yeah, well, you can keep track of whether the
link |
00:15:46.080
data changed. And so when you come back to me a piece of
link |
00:15:48.880
code, should you calculate it again or do the same thing?
link |
00:15:51.760
Okay, how much of this is art and how much of it is science?
link |
00:15:55.440
Because it sounds pretty complicated. Well, how do you
link |
00:15:59.440
describe a situation? So imagine you come to a point in the
link |
00:16:02.000
road where you have to make a decision. And you have a
link |
00:16:05.360
bunch of knowledge about which way to go. Maybe you have a
link |
00:16:07.520
map. So you want to go the shortest way. Or do you want to
link |
00:16:11.840
go the fastest way? Or do you want to take the nicest
link |
00:16:14.240
road? So there's some set of data. So imagine you're
link |
00:16:18.400
doing something complicated like building a computer. And
link |
00:16:21.920
there's hundreds of decision points, all with hundreds of
link |
00:16:25.600
possible ways to go. And the ways you pick interact in a
link |
00:16:29.680
complicated way. Right. And then you have to pick the right
link |
00:16:34.400
spot. Right. So that's an order of science. I don't know.
link |
00:16:37.360
You avoided the question. You just described the Robert Frost
link |
00:16:40.880
poem of road less taken. I described the Robin Frost
link |
00:16:45.040
problem. That's what we do as computer designers. It's all
link |
00:16:49.600
poetry. Okay. Great. Yeah, I don't know how to describe that
link |
00:16:54.080
because some people are very good at making those
link |
00:16:57.040
intuitive leaps. It seems like the this combinations of
link |
00:16:59.760
things. Some people are less good at it, but they're really
link |
00:17:02.800
good at evaluating the alternatives. Right. And
link |
00:17:06.080
everybody has a different way to do it. And some people can't
link |
00:17:10.800
make those leaps, but they're really good at analyzing it.
link |
00:17:14.160
So when you see computers are designed by teams of people
link |
00:17:16.720
who have very different skill sets and a good team has lots
link |
00:17:22.320
of different kinds of people. I suspect you would describe
link |
00:17:25.600
some of them as artistic. Right. But not very many.
link |
00:17:30.320
Unfortunately. Or fortunately. Well, you know, computer
link |
00:17:35.760
design is hard. It's 99% perspiration. And the 1%
link |
00:17:41.440
perspiration is really important. But you still need the 99.
link |
00:17:45.680
Yeah, you gotta do a lot of work. And then there's there are
link |
00:17:49.520
interesting things to do at every level that stack. So at the
link |
00:17:54.320
end of the day, if you run the same program multiple times,
link |
00:17:58.720
does it always produce the same result? Is is is there some
link |
00:18:02.800
room for fuzziness there? That's a math problem. So if you
link |
00:18:07.120
run a correct C program, the definition is every time you
link |
00:18:11.120
run it, you get the same answer. Yeah, well, that's a math
link |
00:18:13.680
statement. But that's a that's a language
link |
00:18:15.840
definitional statement. So for years, when people did when we
link |
00:18:20.080
first did 3D acceleration of graphics, you could run the
link |
00:18:25.440
same scene multiple times and get different answers. Right.
link |
00:18:29.600
Right. And then some people thought that was okay. And
link |
00:18:32.400
some people thought it was a bad idea. And then when the
link |
00:18:35.760
HPC world used GPUs for calculations, they thought it
link |
00:18:39.520
was a really bad idea. Okay. Now, in modern AI stuff,
link |
00:18:44.320
people are looking at networks where the precision of the
link |
00:18:49.200
data is low enough that the data is somewhat noisy. And the
link |
00:18:53.920
observation is the input data is unbelievably noisy. So why
link |
00:18:57.600
should the calculation be not noisy? And people have
link |
00:19:00.640
experimented with algorithms that say can get faster answers
link |
00:19:04.320
by being noisy. Like as the network starts to converge, if
link |
00:19:08.160
you look at the computation graph, it starts out really
link |
00:19:10.240
wide and it gets narrower. And you can say is that last
link |
00:19:13.200
little bit that important? Or should I start the graph on
link |
00:19:15.680
the next rev before we whittle it all the way down to the
link |
00:19:19.680
answer? Right. So you can create algorithms that are
link |
00:19:22.640
noisy. Now, if you're developing something and every
link |
00:19:25.760
time you run it, you get a different answer. It's really
link |
00:19:27.920
annoying. And so most people think even today, every time
link |
00:19:34.240
you run the program, you get the same answer. No, I know.
link |
00:19:37.040
But the question is, that's the formal definition of a
link |
00:19:40.960
programming language. There is a definition of languages that
link |
00:19:44.640
don't get the same answer. But people who use those, you
link |
00:19:48.640
always want something because you get a bad answer. And then
link |
00:19:50.960
you're wondering, is it because of something in the algorithm
link |
00:19:54.320
or because of this? And so everybody wants a little switch
link |
00:19:56.720
that says no matter what, do it deterministically. And it's
link |
00:20:00.560
really weird because almost everything going into modern
link |
00:20:03.200
calculations is noisy. So the answers have to be so clear.
link |
00:20:07.440
Right. So where do you stand? I design computers for people who
link |
00:20:10.960
run programs. So somebody says, I want a deterministic
link |
00:20:15.440
answer. Like most people want that. Can you deliver a
link |
00:20:19.200
deterministic answer? I guess is the question. Like when you
link |
00:20:22.160
Yeah, hopefully, sure. What people don't realize is you get
link |
00:20:25.840
a deterministic answer even though the execution flow is
link |
00:20:28.880
very undeterministic. So you run this program 100 times.
link |
00:20:33.040
It never runs the same way twice, ever. And the answer, it
link |
00:20:37.040
arrives at the same answer. But it gets the same answer
link |
00:20:38.800
every time. It's just, it's just amazing. Okay, you've
link |
00:20:43.680
achieved in the eyes of many people, legend status as a
link |
00:20:50.800
chip art architect, what design creation are you most proud
link |
00:20:55.360
of? Perhaps because it was challenging, because of its
link |
00:21:00.000
impact or because of the set of brilliant ideas that that
link |
00:21:04.960
were involved in. Well, I find that description odd and I
link |
00:21:10.160
have two small children and I promise you they think it's
link |
00:21:14.720
hilarious. This question. Yeah. So I do it for them. So I am
link |
00:21:20.560
I'm really interested in building computers and I've worked
link |
00:21:24.160
with really, really smart people. I'm not unbelievable
link |
00:21:28.400
to be smart. I'm fascinated by how they go together both as a
link |
00:21:33.360
as a thing to do and as endeavor that people do. How
link |
00:21:38.400
people and computers go together. Yeah, like how people
link |
00:21:41.360
think and build a computer. And I find sometimes that the
link |
00:21:45.600
best computer architects aren't that interested in people or
link |
00:21:49.360
the best people managers aren't that good at designing
link |
00:21:52.320
computers. So the whole stack of human beings is
link |
00:21:56.400
fascinating. So the managers, the individual engineers. So
link |
00:21:59.760
yeah, I said I realized after a lot of years of building
link |
00:22:02.640
computers, where you sort of build them out of
link |
00:22:04.560
transistors, logic gates, functional units, computational
link |
00:22:07.520
elements, that you could think of people the same way. So
link |
00:22:10.880
people are functional units. Yes. And then you can think of
link |
00:22:13.440
organizational design as a computer architectural problem.
link |
00:22:16.800
And then it's like, oh, that's super cool because the people
link |
00:22:20.000
are all different just like the computational elements are
link |
00:22:22.400
all different. And they like to do different things and
link |
00:22:26.320
so I had a lot of fun like reframing how I think about
link |
00:22:29.760
organizations. Just like with with computers, we were
link |
00:22:34.320
saying execution paths, you can have a lot of different
link |
00:22:36.880
paths that end up at a at at the same good destination. So
link |
00:22:41.600
what have you learned about the human abstractions from
link |
00:22:45.840
individual functional human units to the the broader
link |
00:22:50.240
organization? What what does it take to create something
link |
00:22:53.440
special? Well, most people don't think simple enough.
link |
00:23:00.080
All right. So do you know the difference between a recipe
link |
00:23:02.640
and the understanding? There's probably a philosophical
link |
00:23:07.760
description of this. So imagine you're gonna make a loaf
link |
00:23:10.000
for bread. Yep. The recipe says get some flour, add some
link |
00:23:13.360
water, add some yeast, mix it up, let it rise, put it in a
link |
00:23:17.120
pan, put it in the oven. It's a recipe. Right.
link |
00:23:21.200
Understanding bread. You can understand biology, supply
link |
00:23:25.040
chains, you know, rain grinders, yeast, physics, you
link |
00:23:32.000
know, thermodynamics, like there's so many levels of
link |
00:23:35.520
understanding there. And then when people build and design
link |
00:23:39.200
things, they frequently are executing some stack of
link |
00:23:42.640
recipes. Right. And the problem with that is the recipes
link |
00:23:46.800
all have limited scope. Look, if you have a really good
link |
00:23:49.840
recipe book for making bread, it won't tell you anything
link |
00:23:52.160
about how to make an omelet. Right. Right. But if you
link |
00:23:55.200
have a deep understanding of cooking, right, then bread,
link |
00:24:00.720
omelets, you know, sandwich, you know, there's there's a
link |
00:24:04.400
different, you know, way of viewing everything. And most
link |
00:24:08.800
people, when you get to be an expert at something, you
link |
00:24:13.040
know, you're you're hoping to achieve deeper understanding
link |
00:24:16.320
not just a large set of recipes to go execute. And it's
link |
00:24:21.200
interesting to walk groups of people because executing
link |
00:24:24.080
recipes is unbelievably efficient. If it's what you
link |
00:24:28.240
want to do. If it's not what you want to do, you're
link |
00:24:32.320
really stuck. And that difference is crucial. And
link |
00:24:37.120
everybody has a balance of, let's say, deeper
link |
00:24:39.760
understanding recipes. And some people are really good
link |
00:24:41.920
at recognizing when the problem is to understand
link |
00:24:45.360
something deeply, deeply. Does that make sense? It
link |
00:24:49.120
totally makes sense. Does it every stage of development
link |
00:24:52.720
deep understanding on the team needed? Oh, this goes
link |
00:24:55.920
back to the art versus science question. Sure. If you
link |
00:24:59.520
constantly unpacked everything for deeper understanding,
link |
00:25:02.080
you never get anything done. Right. And if you don't
link |
00:25:04.800
unpack understanding when you need to, you'll do the
link |
00:25:07.520
wrong thing. And then at every juncture, like human
link |
00:25:11.600
beings are these really weird things because
link |
00:25:14.160
everything you tell them has a million possible
link |
00:25:16.320
outputs, right? And then they all interact in a
link |
00:25:19.360
hilarious way. And then having some intuition about
link |
00:25:23.360
what do you tell them, what do you do, when do you
link |
00:25:25.360
intervene, when do you not, it's, it's complicated.
link |
00:25:28.720
Right. So it's, it's, it's, you know, essentially
link |
00:25:31.680
computationally unsolvable. Yeah, it's an intractable
link |
00:25:34.480
problem. Sure. Humans are a mess. But with deep
link |
00:25:40.480
understanding, do you mean also sort of fundamental
link |
00:25:44.000
questions of things like what is a computer? Like, or
link |
00:25:52.240
why? Like, the why questions, why are we even
link |
00:25:55.760
building this? Like, of purpose? Or do you mean
link |
00:26:00.080
more like going towards the fundamental limits of
link |
00:26:03.760
physics sort of really getting into the core of
link |
00:26:06.800
the science? Well, in terms of building a computer,
link |
00:26:09.520
think simple, think a little simpler. So common
link |
00:26:12.880
practices, you build a computer. And then when
link |
00:26:15.120
somebody says, I want to make it 10% faster, you'll
link |
00:26:17.920
go in and say, all right, I need to make this
link |
00:26:19.920
buffer bigger. And maybe I'll add an ad unit. Or, you
link |
00:26:23.520
know, I have this thing that's three instructions
link |
00:26:25.040
wide, I'm going to make it four instructions wide.
link |
00:26:27.680
And what you see is each piece gets incrementally
link |
00:26:31.440
more complicated. Right. And then at some point, you
link |
00:26:35.280
hit this limit, like adding another feature or
link |
00:26:38.640
buffer doesn't seem to make it any faster. And
link |
00:26:41.280
then people will say, well, that's because it's a
link |
00:26:43.440
fundamental limit. And then somebody else will look
link |
00:26:46.320
at it and say, well, actually the way you divided
link |
00:26:48.320
the problem up and the way that different features
link |
00:26:50.720
are interacting is limiting you and it has to be
link |
00:26:53.520
rethought, rewritten. Right. So then you refactor
link |
00:26:57.360
and rewrite it and what people commonly find is
link |
00:27:00.160
the rewrite is not only faster, but half is
link |
00:27:02.480
complicated. From scratch? Yes. So how often in
link |
00:27:06.640
your career, but just have you seen as needed,
link |
00:27:09.600
maybe more generally, to just throw the whole
link |
00:27:12.640
out, the whole thing out? This is where I'm on one
link |
00:27:15.760
end of it, every three to five years. Which end
link |
00:27:19.520
are you on? Wait. Rewrite more often. Rewrite. And
link |
00:27:23.280
three to five years is... If you want to really
link |
00:27:25.840
make a lot of progress in computer architecture,
link |
00:27:28.000
every five years you should do one from scratch.
link |
00:27:31.920
So where does the x86, x64 standard come in?
link |
00:27:36.800
How often do you... I was the coauthor of that spec
link |
00:27:40.960
in 98. That's 20 years ago. Yeah. So that's still
link |
00:27:44.800
around. The instruction set itself has been
link |
00:27:47.600
extended quite a few times. Yes. And instruction sets
link |
00:27:51.440
are less interesting than the implementation
link |
00:27:53.360
underneath. There's been... Interesting. On x86
link |
00:27:56.320
architecture, Intel's designed a few, AIM just
link |
00:27:59.040
designed a few very different architectures.
link |
00:28:02.400
And I don't want to go into too much of the detail
link |
00:28:06.400
about how often, but there's a tendency to
link |
00:28:10.640
rewrite it every 10 years and it really
link |
00:28:12.960
should be every five. So you're saying you're an
link |
00:28:16.240
outlier in that sense in the... Rewrite more often.
link |
00:28:18.800
Rewrite more often. Well, and here's... Isn't that scary?
link |
00:28:22.000
Yeah, of course. Well, scary to who? To everybody
link |
00:28:26.160
involved, because like you said, repeating the
link |
00:28:28.640
recipe is efficient. Companies want to make money...
link |
00:28:34.080
No. Individual engineers want to succeed. So you want to
link |
00:28:37.600
incrementally improve, increase the buffer from 3 to 4.
link |
00:28:41.120
Well, this is where you get into diminishing return curves.
link |
00:28:45.280
I think Steve Jobs said this, right? So every... You have a project
link |
00:28:48.880
and you start here and it goes up and they have
link |
00:28:50.800
diminishing return. And to get to the next level, you have to
link |
00:28:53.840
do a new one and the initial starting point
link |
00:28:56.640
will be lower than the old optimization point.
link |
00:29:00.000
But it'll get higher. So now you have two kinds of fear.
link |
00:29:03.440
Short term disaster and long term disaster.
link |
00:29:07.440
And you're... So grown ups, right? Like, you know, people with a
link |
00:29:12.720
quarter by quarter business objective are terrified about
link |
00:29:16.400
changing everything. And people who are trying to run
link |
00:29:19.920
a business or build a computer for a long term objective
link |
00:29:23.920
know that the short term limitations block them from the long term success.
link |
00:29:29.200
So if you look at leaders of companies that had really good
link |
00:29:33.520
long term success, every time they saw that they had to redo
link |
00:29:36.960
something, they did. And so somebody has to speak up?
link |
00:29:40.880
Or you do multiple projects in parallel. Like, you optimize the old one while you
link |
00:29:44.720
build a new one and... But the marketing guys are always like,
link |
00:29:48.160
make promise me that the new computer is faster on every single thing.
link |
00:29:52.560
And the computer architect says, well, the new computer will be faster on the
link |
00:29:55.440
average. But there's a distribution of results
link |
00:29:58.560
and performance. And you'll have some outliers that are slower.
link |
00:30:01.760
And that's very hard because they have one customer cares about that one.
link |
00:30:05.200
So speaking of the long term, for over 50 years now, Moore's Law has served a...
link |
00:30:11.280
for me and millions of others as an inspiring beacon
link |
00:30:14.960
of what kind of amazing future brilliant engineers can build.
link |
00:30:19.280
I'm just making your kids laugh all of today. That's great.
link |
00:30:23.520
So first, in your eyes, what is Moore's Law if you could define for people who
link |
00:30:28.800
don't know? Well, the simple statement was,
link |
00:30:33.360
from Gordon Moore, was double the number of transistors every two years.
link |
00:30:37.760
Something like that. And then my operational model is we increase the
link |
00:30:44.240
performance of computers by 2x every two or three years.
link |
00:30:48.400
And it's wiggled around substantially over time.
link |
00:30:51.360
And also, in how we deliver performance has changed.
link |
00:30:56.960
But the foundational idea was 2x the transistors every two years.
link |
00:31:02.800
The current cadence is something like, they call it a shrink factor,
link |
00:31:07.840
like 0.6 every two years, which is not 0.5.
link |
00:31:11.760
But that's referring strictly again to the original definition.
link |
00:31:14.720
Yeah, a transistor count.
link |
00:31:16.480
A shrink factor, just getting them small and small and small.
link |
00:31:18.880
Well, as you use for a constant chip area, if you make the transistors smaller by 0.6,
link |
00:31:24.080
then you get one over 0.6 more transistors.
link |
00:31:27.040
So can you linger a little longer? What's the broader,
link |
00:31:30.560
what do you think should be the broader definition of Moore's Law
link |
00:31:33.760
when you mentioned how you think of performance?
link |
00:31:37.760
Just broadly, what's a good way to think about Moore's Law?
link |
00:31:42.240
Well, first of all, I've been aware of Moore's Law for 30 years.
link |
00:31:46.880
In which sense?
link |
00:31:48.880
Well, I've been designing computers for 40.
link |
00:31:52.160
You're just watching it before your eyes kind of thing.
link |
00:31:55.280
Well, and somewhere where I became aware of it,
link |
00:31:58.000
I was also informed that Moore's Law was going to die in 10 to 15 years.
link |
00:32:02.000
And then I thought that was true at first.
link |
00:32:03.760
But then after 10 years, it was going to die in 10 to 15 years.
link |
00:32:07.360
And then at one point, it was going to die in five years.
link |
00:32:09.600
And then it went back up to 10 years.
link |
00:32:11.120
And at some point, I decided not to worry about that particular product
link |
00:32:15.280
agnostication for the rest of my life, which is fun.
link |
00:32:19.520
And then I joined Intel and everybody said Moore's Law is dead.
link |
00:32:22.640
And I thought that's sad because it's the Moore's Law company.
link |
00:32:25.520
And it's not dead.
link |
00:32:26.800
And it's always been going to die.
link |
00:32:29.120
And humans like these apocryphal kind of statements like,
link |
00:32:34.080
we'll run out of food or we'll run out of air or run out of room
link |
00:32:37.040
or run out of something.
link |
00:32:39.840
Right. But it's still incredible that it's lived for as long as it has.
link |
00:32:43.920
And yes, there's many people who believe now that Moore's Law is dead.
link |
00:32:49.200
You know, they can join the last 50 years of people who had the same idea.
link |
00:32:52.960
Yeah, there's a long tradition.
link |
00:32:54.240
But why do you think, if you can try to understand it, why do you think it's not dead?
link |
00:33:01.760
Well, first, let's just think, people think Moore's Law is one thing.
link |
00:33:06.240
Transistors get smaller.
link |
00:33:08.240
But actually under the sheets, there's literally thousands of innovations.
link |
00:33:11.440
And almost all those innovations have their own diminishing return curves.
link |
00:33:17.200
So if you graph it, it looks like a cascade of diminishing return curves.
link |
00:33:21.280
I don't know what to call that.
link |
00:33:22.560
But the result is an exponential curve.
link |
00:33:26.320
But at least it has been.
link |
00:33:29.040
And we keep inventing new things.
link |
00:33:30.720
So if you're an expert in one of the things on a diminishing return curve,
link |
00:33:35.440
right, and you can see it's plateau, you will probably tell people,
link |
00:33:39.440
well, this is done.
link |
00:33:42.000
Meanwhile, some other pilot people are doing something different.
link |
00:33:46.240
So that's just normal.
link |
00:33:48.160
So then there's the observation of how small could a switching device be?
link |
00:33:53.920
So a modern transistor is something like 1,000 by 1,000 by 1,000 atoms, right?
link |
00:33:59.760
And you get quantum effects down around 2 to 10 atoms.
link |
00:34:04.480
So you can imagine a transistor as small as 10 by 10 by 10.
link |
00:34:08.080
So that's a million times smaller.
link |
00:34:11.920
And then the quantum computational people are working away at how to use quantum effects.
link |
00:34:17.360
So 1,000 by 1,000 by 1,000.
link |
00:34:21.840
Atoms.
link |
00:34:23.520
That's a really clean way of putting it.
link |
00:34:26.480
Well, a fan, like a modern transistor, if you look at the fan, it's like 120 atoms wide.
link |
00:34:31.840
But we can make that thinner.
link |
00:34:33.200
And then there's a gate wrapped around it and then there's spacing.
link |
00:34:36.400
There's a whole bunch of geometry.
link |
00:34:38.640
And a competent transistor designer could count both atoms in every single direction.
link |
00:34:47.760
Like there's techniques now to already put down atoms in a single atomic layer.
link |
00:34:52.880
And you can place atoms if you want to.
link |
00:34:55.680
It's just, from a manufacturing process, if placing an atom takes 10 minutes,
link |
00:35:01.120
and you need to put 10 to the 23rd atoms together to make a computer, it would take a long time.
link |
00:35:08.640
So the methods are both shrinking things.
link |
00:35:13.120
And then coming up with effective ways to control what's happening.
link |
00:35:17.760
Manufacture stably and cheaply.
link |
00:35:19.920
Yeah.
link |
00:35:21.200
So the innovation stock's pretty broad.
link |
00:35:23.680
There's equipment.
link |
00:35:25.120
There's optics.
link |
00:35:25.840
There's chemistry.
link |
00:35:26.720
There's physics.
link |
00:35:27.440
There's material science.
link |
00:35:29.040
There's metallurgy. There's lots of ideas about when you put different materials together,
link |
00:35:33.520
how do they interact?
link |
00:35:34.320
Are they stable?
link |
00:35:35.280
Is it stable over temperature?
link |
00:35:38.720
Like are they repeatable?
link |
00:35:40.800
There's literally thousands of technologies involved.
link |
00:35:44.800
But just for the shrinking, you don't think we're quite yet close to the fundamental limits of physics.
link |
00:35:50.720
I did a talk on Mars Law and I asked for a roadmap to a path of about 100.
link |
00:35:54.800
And after two weeks, they said, we only got to 50.
link |
00:35:57.840
100, what say?
link |
00:35:59.520
100 X shrink.
link |
00:36:00.480
100 X shrink.
link |
00:36:01.680
We only got to 50.
link |
00:36:02.960
And I said, why don't you give us another two weeks?
link |
00:36:07.520
Well, here's the thing about Mars Law.
link |
00:36:09.520
So I believe that the next 10 or 20 years of shrinking is going to happen.
link |
00:36:16.240
Now, as a computer designer, you have two stances.
link |
00:36:20.800
You think it's going to shrink, in which case you're designing and thinking about architecture
link |
00:36:26.000
in a way that you'll use more transistors, or conversely, not be swamped by the complexity
link |
00:36:32.720
of all the transistors you get.
link |
00:36:36.000
You have to have a strategy.
link |
00:36:39.200
So you're open to the possibility and waiting for the possibility of a whole new army of
link |
00:36:44.160
transistors ready to work.
link |
00:36:45.840
I'm expecting more transistors every two or three years by a number large enough
link |
00:36:51.920
that how you think about design, how you think about architecture has to change.
link |
00:36:56.960
Like, imagine you build buildings out of bricks, and every year the bricks are half the size,
link |
00:37:04.240
or every two years.
link |
00:37:05.680
Well, if you kept building bricks the same way, so many bricks per person per day,
link |
00:37:11.120
the amount of time to build a building would go up exponentially.
link |
00:37:15.600
Right.
link |
00:37:16.000
Right.
link |
00:37:16.720
But if you said, I know that's coming, so now I'm going to design equipment.
link |
00:37:20.960
I move bricks faster, use them better, because maybe you're getting something out of the smaller
link |
00:37:24.960
bricks, more strengths, thinner walls, you know, less material efficiency out of that.
link |
00:37:30.160
So once you have a roadmap with what's going to happen, transistors, they're going to get,
link |
00:37:34.720
we're going to get more of them, then you design all this collateral around it to take advantage
link |
00:37:39.440
of it, and also to cope with it.
link |
00:37:42.240
Like, that's the thing people don't understand.
link |
00:37:43.600
It's like, if I didn't believe in Moore's Law, and then Moore's Law Transistors showed up,
link |
00:37:48.080
my design teams were all drowned.
link |
00:37:51.440
So what's the hardest part of this influx of new transistors?
link |
00:37:57.200
I mean, even if you just look historically throughout your career,
link |
00:38:02.080
what's the thing, what fundamentally changes when you add more transistors
link |
00:38:06.880
in the task of designing an architecture?
link |
00:38:10.560
Well, there's two constants, right?
link |
00:38:12.320
One is people don't get smarter.
link |
00:38:13.840
By the way, there's some science shown that we do get smarter because of nutrition, whatever.
link |
00:38:21.120
Sorry to bring that up.
link |
00:38:22.000
In effect.
link |
00:38:22.480
Yes.
link |
00:38:22.880
Yeah, familiar with it.
link |
00:38:23.760
Nobody understands it.
link |
00:38:24.640
Nobody knows if it's still going on.
link |
00:38:26.160
So that's a...
link |
00:38:27.040
Or whether it's real or not.
link |
00:38:28.400
But yeah, I sort of...
link |
00:38:31.200
Anyway, but not exponentially.
link |
00:38:32.000
I would believe for the most part, people aren't getting much smarter.
link |
00:38:35.360
The evidence doesn't support it.
link |
00:38:36.720
That's right.
link |
00:38:37.440
And then teams can't grow that much.
link |
00:38:39.920
Right.
link |
00:38:40.560
So human beings, we're really good in teams of 10, up to teams of 100.
link |
00:38:47.120
They can know each other.
link |
00:38:48.000
Beyond that, you have to have organizational boundaries.
link |
00:38:51.920
Those are pretty hard constraints.
link |
00:38:54.480
So then you have to divide and conquer.
link |
00:38:56.320
Like as the designs get bigger, you have to divide it into pieces.
link |
00:39:00.480
The power of abstraction layers is really high.
link |
00:39:03.040
We used to build computers out of transistors.
link |
00:39:06.000
Now we have a team that turns transistors into logic cells,
link |
00:39:08.720
and our team that turns them into functional units.
link |
00:39:10.480
Another one that turns them into computers.
link |
00:39:12.960
So we have abstraction layers in there.
link |
00:39:15.920
And you have to think about when do you shift gears on that.
link |
00:39:21.120
We also use faster computers to build faster computers.
link |
00:39:24.160
So some algorithms run twice as fast on new computers,
link |
00:39:27.600
but a lot of algorithms are n squared.
link |
00:39:30.320
So a computer with twice as many transistors,
link |
00:39:33.360
and it might take four times as long to run.
link |
00:39:36.320
So you have to refactor the software.
link |
00:39:38.320
Like simply using faster computers
link |
00:39:40.880
to build bigger computers doesn't work.
link |
00:39:44.080
So you have to think about all these things.
link |
00:39:46.160
So in terms of computing performance
link |
00:39:47.760
and the exciting possibility that more powerful computers bring
link |
00:39:51.520
is shrinking the thing we've just been talking about.
link |
00:39:56.400
One of the, for you, one of the biggest exciting possibilities
link |
00:39:59.680
of advancement in performance.
link |
00:40:01.280
Or is there other directions that you're interested in?
link |
00:40:03.840
Like in the direction of sort of enforcing given parallelism,
link |
00:40:09.280
or doing massive parallelism in terms of many, many CPUs,
link |
00:40:15.120
stacking CPUs on top of each other, that kind of parallelism,
link |
00:40:19.600
or any kind of parallelism?
link |
00:40:20.640
Well, think about it in a different way.
link |
00:40:22.160
So all of computers, slow computers,
link |
00:40:25.120
you said a equal b plus c times d.
link |
00:40:28.400
Pretty simple, right?
link |
00:40:30.480
And then we made faster computers with vector units,
link |
00:40:33.360
and you can do proper equations and matrices, right?
link |
00:40:38.400
And then modern like AI computations,
link |
00:40:40.960
or like convolutional neural networks,
link |
00:40:43.280
where you convolve one large data set against another.
link |
00:40:46.960
And so there's sort of this hierarchy of mathematics,
link |
00:40:50.960
you know, from simple equation to linear equations
link |
00:40:53.920
to matrix equations to deeper kind of computation.
link |
00:40:58.640
And the data sets are getting so big
link |
00:41:00.480
that people are thinking of data as a topology problem.
link |
00:41:04.160
You know, data is organized in some immense shape.
link |
00:41:07.840
And then the computation, which sort of wants to be
link |
00:41:11.040
get data from immense shape and do some computation on it.
link |
00:41:15.200
So what computers have a lot of people to do
link |
00:41:18.000
is have algorithms go much, much further.
link |
00:41:22.320
So that paper you referenced, the Sutton paper,
link |
00:41:26.480
they talked about, you know, like when AI started,
link |
00:41:28.960
it was apply rule sets to something.
link |
00:41:31.680
That's a very simple computational situation.
link |
00:41:35.680
And then when they did first chess thing,
link |
00:41:37.680
they solved deep searches.
link |
00:41:39.840
So have a huge database of moves and results, deep search,
link |
00:41:44.560
but it's still just a search, right?
link |
00:41:48.000
Now we take large numbers of images,
link |
00:41:51.040
and we use it to train these weight sets
link |
00:41:54.240
that we convolve across.
link |
00:41:56.080
It's a completely different kind of phenomena.
link |
00:41:58.160
We call that AI.
link |
00:41:59.360
And now they're doing the next generation.
link |
00:42:01.760
And if you look at it,
link |
00:42:03.040
they're going up this mathematical graph, right?
link |
00:42:06.720
And then computations, both computation and data sets,
link |
00:42:10.480
support going up that graph.
link |
00:42:13.040
Yeah, the kind of computation though might,
link |
00:42:14.880
I mean, I would argue that all of it is still a search, right?
link |
00:42:19.200
Just like you said, a topology problem of data sets,
link |
00:42:22.080
you're searching the data sets for valuable data
link |
00:42:26.320
and also the actual optimization of neural networks
link |
00:42:29.360
is a kind of search for the...
link |
00:42:31.920
I don't know.
link |
00:42:32.640
If you had looked at the inner layers of finding a cat,
link |
00:42:36.320
it's not a search.
link |
00:42:38.240
It's a set of endless projections.
link |
00:42:40.400
So a projection, here's a shadow of this phone, right?
link |
00:42:44.480
Yeah.
link |
00:42:44.800
And then you can have a shadow of that on the something,
link |
00:42:46.960
a shadow on that or something.
link |
00:42:48.320
And if you look in the layers,
link |
00:42:49.680
you'll see this layer actually describes pointy ears
link |
00:42:52.800
and round eyedness and fuzziness and...
link |
00:42:55.120
But the computation to tease out the attributes is not search.
link |
00:43:02.880
Right.
link |
00:43:03.600
I mean, well...
link |
00:43:04.240
Like the inference part might be search,
link |
00:43:05.760
but the training's not search.
link |
00:43:07.200
Okay, well, technically...
link |
00:43:07.840
And then in deep networks, they look at layers
link |
00:43:10.640
and they don't even know it's represented.
link |
00:43:14.160
And yet, if you take the layers out, it doesn't work.
link |
00:43:16.320
Okay, so refer...
link |
00:43:17.120
So I don't think it's search.
link |
00:43:18.800
All right, well...
link |
00:43:19.360
But you'll have to talk to my mathematician
link |
00:43:20.880
about what that actually is.
link |
00:43:21.920
Well, we could disagree, but it's just semantics, I think.
link |
00:43:27.200
It's not...
link |
00:43:28.000
But it's certainly not...
link |
00:43:28.960
I would say it's absolutely not semantics, but...
link |
00:43:31.760
Okay.
link |
00:43:33.440
All right, well, if you want to go there.
link |
00:43:36.880
So optimization, to me, is search.
link |
00:43:38.880
And we're trying to optimize the ability
link |
00:43:42.720
of in your own network to detect cat ears.
link |
00:43:45.760
And the difference between chess and the space,
link |
00:43:50.160
the incredibly multidimensional,
link |
00:43:53.280
100,000 dimensional space that, you know,
link |
00:43:56.080
networks are trying to optimize over,
link |
00:43:57.680
is nothing like the chess board database.
link |
00:44:01.040
So it's a totally different kind of thing.
link |
00:44:03.440
And, okay, in that sense, you can say...
link |
00:44:05.440
Yeah, yeah.
link |
00:44:06.000
It loses the meaning.
link |
00:44:06.880
I can see how you might say.
link |
00:44:08.560
If you...
link |
00:44:10.400
The funny thing is, it's the difference between
link |
00:44:13.200
given search space and found search space.
link |
00:44:15.760
Right, exactly.
link |
00:44:16.400
Yeah, maybe that's the different way to describe it.
link |
00:44:17.760
That's a beautiful way to put it, okay.
link |
00:44:19.040
But you're saying, what's your sense in terms of the basic
link |
00:44:22.480
mathematical operations and the architectures
link |
00:44:26.800
computer hardware that enables those operations?
link |
00:44:29.760
Do you see the CPUs of today still being a really core part
link |
00:44:34.960
of executing those mathematical operations?
link |
00:44:37.520
Yes.
link |
00:44:38.320
Well, the operations, you know,
link |
00:44:40.080
continue to be at subtract, load, store,
link |
00:44:42.960
compare, and branch.
link |
00:44:43.840
It's remarkable.
link |
00:44:46.000
So it's interesting that the building blocks
link |
00:44:48.640
of, you know, computers or transistors,
link |
00:44:50.880
and, you know, under that atoms.
link |
00:44:52.480
So you've got atoms, transistors, logic gates, computers,
link |
00:44:55.760
right, you know, functional units and computers.
link |
00:44:58.160
The building blocks of mathematics at some level
link |
00:45:00.880
are things like ads and subtracts and multiplies.
link |
00:45:04.240
But the space mathematics can describe as,
link |
00:45:08.800
I think, essentially infinite.
link |
00:45:11.120
But the computers that run the algorithms
link |
00:45:13.920
are still doing the same things.
link |
00:45:15.840
Now, a given algorithm may say, I need sparse data,
link |
00:45:19.520
or I need 32 bit data, or I need, you know,
link |
00:45:24.000
like a convolution operation that naturally takes
link |
00:45:26.960
8 bit data, multiplies it and sums it up a certain way.
link |
00:45:30.800
So the, like the data types in TensorFlow imply
link |
00:45:34.880
an optimization set.
link |
00:45:37.360
But when you go right down and look at the computers,
link |
00:45:39.680
it's Ann and Orgade still and add some multiplies.
link |
00:45:41.680
Like, that hasn't changed much.
link |
00:45:44.800
Now, the quantum researchers think they're going
link |
00:45:47.520
to change that radically.
link |
00:45:48.640
And then there's people who think about analog computing,
link |
00:45:50.960
because you look in the brain, and it seems to be more analogish.
link |
00:45:54.240
You know, that may be just a way to do that more efficiently.
link |
00:45:57.760
But we have a million X on computation.
link |
00:46:02.080
And I don't know the repris...
link |
00:46:05.440
The relationship between computational, let's say,
link |
00:46:08.400
intensity and ability to hit mathematical abstractions.
link |
00:46:12.960
I don't know anybody's described that, but just like you saw in AI,
link |
00:46:17.680
you went from rule sets, the simple search, the complex search,
link |
00:46:21.680
to, say, found search.
link |
00:46:23.680
Like, those are, you know, orders of magnitude,
link |
00:46:26.720
more computation to do.
link |
00:46:28.720
And as we get to the next two orders of magnitude,
link |
00:46:32.400
like a friend, Roger Gadori, said,
link |
00:46:34.000
like every order of magnitude changes to computation.
link |
00:46:37.680
Fundamentally changes what the computation is doing.
link |
00:46:40.480
Fundamentally changes what the computation is doing.
link |
00:46:43.040
Oh, you know, the expression of the difference in quantity
link |
00:46:46.720
is a difference in kind.
link |
00:46:49.280
You know, the difference between ant and anthill, right?
link |
00:46:52.880
Or neuron and brain.
link |
00:46:55.760
You know, there's indefinable place where the quantity changed,
link |
00:47:00.560
the quality, right?
link |
00:47:02.320
And we've seen that happen in mathematics multiple times.
link |
00:47:04.880
And, you know, my guess is it's going to keep happening.
link |
00:47:08.480
So your sense is, yeah, if you focus head down
link |
00:47:12.080
and shrinking the transistor.
link |
00:47:14.640
Well, it's not just head down.
link |
00:47:15.600
We're aware of the software stacks
link |
00:47:18.240
that are running in the computational loads.
link |
00:47:20.240
And we're kind of pondering, what do you do
link |
00:47:22.560
with a petabyte of memory that wants to be accessed
link |
00:47:25.440
in a sparse way and have, you know,
link |
00:47:28.080
the kind of calculations AI programmers want?
link |
00:47:32.560
So there's a dialogue and interaction.
link |
00:47:34.640
But when you go in the computer chip,
link |
00:47:36.720
you know, you find addersons, subtractors, and multipliers.
link |
00:47:41.280
And so if you zoom out, then with, as you mentioned,
link |
00:47:45.440
the idea that most of the development
link |
00:47:49.120
in the last many decades in AI research
link |
00:47:51.440
came from just leveraging computation
link |
00:47:54.240
and just simple algorithms waiting
link |
00:47:58.160
for the computation to improve.
link |
00:47:59.840
Well, software guys have a thing that they call it
link |
00:48:03.600
the problem of early optimization.
link |
00:48:06.080
Right.
link |
00:48:06.960
So you write a big software stack
link |
00:48:09.040
and if you start optimizing like the first thing you write,
link |
00:48:12.240
the odds of that being the performance limit is low.
link |
00:48:15.200
But when you get the whole thing working,
link |
00:48:16.800
can you make it 2x faster by optimizing the right things?
link |
00:48:19.760
Sure. While you're optimizing that,
link |
00:48:22.400
could you've written a new software stack,
link |
00:48:24.160
which would have been a better choice?
link |
00:48:26.000
Maybe. Now you have creative tension.
link |
00:48:29.440
So.
link |
00:48:30.080
But the whole time as you're doing the writing,
link |
00:48:32.560
the, that's the software we're talking about,
link |
00:48:34.800
the hardware underneath gets faster and faster.
link |
00:48:36.640
Well, it goes back to the Moore's Law.
link |
00:48:38.000
If Moore's Law is going to continue,
link |
00:48:39.840
then your AI research should expect that to show up.
link |
00:48:45.680
And then you make a slightly different set of choices than
link |
00:48:48.560
we've hit the wall, nothing's going to happen.
link |
00:48:50.640
Yeah.
link |
00:48:51.200
And from here, it's just us rewriting algorithms.
link |
00:48:54.880
Like that seems like a failed strategy
link |
00:48:56.400
for the last 30 years of Moore's Law's death.
link |
00:48:59.600
So.
link |
00:49:00.000
So can you just linger on it?
link |
00:49:01.680
I think you've answered it,
link |
00:49:04.320
but I'll just ask the same dumb question over and over.
link |
00:49:06.880
So why do you think Moore's Law is not going to die?
link |
00:49:12.560
Which is the most promising, exciting possibility
link |
00:49:15.600
of why it won't die in the next 5, 10 years?
link |
00:49:17.920
So is it the continued shrinking of the transistor,
link |
00:49:20.560
or is it another S curve that steps in,
link |
00:49:23.840
and a totally sort of.
link |
00:49:25.360
Well, shrinking the transistor is literally thousands of innovations.
link |
00:49:30.080
Right.
link |
00:49:30.320
So there's a whole bunch of S curves just kind of running
link |
00:49:35.760
their course and being reinvented and new things.
link |
00:49:41.600
The semiconductor fabricators and technologists
link |
00:49:45.440
have all announced what's called nanowires.
link |
00:49:47.280
So they took a fin, which had a gate around it
link |
00:49:51.040
and turned that into a little wire.
link |
00:49:52.480
So you have better control that and they're smaller.
link |
00:49:55.200
And then from there, there are some obvious steps
link |
00:49:57.120
about how to shrink that.
link |
00:49:58.480
So the metallurgy around wire stacks and stuff
link |
00:50:03.600
has very obvious abilities to shrink.
link |
00:50:07.120
And there's a whole combination of things there to do.
link |
00:50:10.880
Your sense is that we're going to get a lot
link |
00:50:13.360
if this innovation from just that shrinking.
link |
00:50:16.480
Yeah, like a factor of a hundred's a lot.
link |
00:50:19.360
Yeah, I would say that's incredible.
link |
00:50:22.080
And it's totally unknown.
link |
00:50:23.600
It's only 10 or 15 years.
link |
00:50:25.040
Now, you're smarter, you might know,
link |
00:50:26.320
but to me, it's totally unpredictable of what that 100x
link |
00:50:29.200
would bring in terms of the nature of the computation
link |
00:50:33.200
that people would be familiar with Bell's law.
link |
00:50:37.120
So for a long time, it was mainframes,
link |
00:50:39.280
many's workstation, PC, mobile.
link |
00:50:42.400
Moore's law drove faster, smaller computers.
link |
00:50:46.160
And then when we were thinking about Moore's law,
link |
00:50:49.360
Roger Gadori said every 10x generates a new computation.
link |
00:50:53.200
So scalar, vector, matrix, topological computation.
link |
00:51:00.960
And if you go look at the industry trends,
link |
00:51:03.680
there was mainframes and many computers and PCs.
link |
00:51:07.280
And then the internet took off and then we got mobile devices.
link |
00:51:10.640
And now we're building 5G wireless with one millisecond latency.
link |
00:51:14.800
And people are starting to think about the smart world
link |
00:51:17.040
where everything knows you, recognizes you.
link |
00:51:20.720
Like the transformations are gonna be unpredictable.
link |
00:51:27.360
How does it make you feel that you're one of the key architects
link |
00:51:33.440
of this kind of future?
link |
00:51:35.040
So you're not, we're not talking about the architects
link |
00:51:37.040
of the high level people who build the Angry Bird apps and Snapchat.
link |
00:51:43.760
Angry Bird apps.
link |
00:51:44.560
Who knows, maybe that's the whole point of the universe.
link |
00:51:46.960
I'm gonna take a stand at that.
link |
00:51:48.160
And the attention distracting nature of mobile phones.
link |
00:51:52.160
I'll take a stand.
link |
00:51:53.040
But anyway, in terms of the side effects of smartphones
link |
00:52:00.400
or the attention distraction, which part?
link |
00:52:03.040
Well, who knows, you know, where this is all leading.
link |
00:52:05.440
It's changing so fast.
link |
00:52:06.640
Wait, so back to the...
link |
00:52:07.600
My parents used to yell at my sisters for hiding in the closet
link |
00:52:09.920
with a wired phone with a dial on it.
link |
00:52:12.400
Stop talking to your friends all day.
link |
00:52:13.840
Right.
link |
00:52:13.840
Now, my wife yells at my kids for talking to her friends
link |
00:52:17.040
all day on text.
link |
00:52:18.160
It looks the same to me.
link |
00:52:19.840
It's always, it echoes at the same time.
link |
00:52:21.840
Okay, but you are the one of the key people architecting
link |
00:52:25.840
the hardware of this future.
link |
00:52:27.440
How does that make you feel?
link |
00:52:28.560
Do you feel responsible?
link |
00:52:31.840
Do you feel excited?
link |
00:52:34.240
So we're in a social context.
link |
00:52:36.240
So there's billions of people on this planet.
link |
00:52:39.040
There are literally millions of people working on technology.
link |
00:52:42.640
I feel lucky to be, you know, doing what I do
link |
00:52:47.440
and getting paid for it.
link |
00:52:48.640
And there's an interest in it.
link |
00:52:50.640
But there's so many things going on in parallel.
link |
00:52:53.040
It's like the actions are so unpredictable.
link |
00:52:56.240
If I wasn't here, somebody else would do it.
link |
00:52:58.640
The vectors of all these different things
link |
00:53:01.040
are happening all the time.
link |
00:53:03.840
You know, there's a, I'm sure some philosopher,
link |
00:53:08.240
a metaphilosopher is, you know, wondering about how we transform our world.
link |
00:53:16.240
So you can't deny the fact that these tools,
link |
00:53:19.040
whether these tools are changing our world.
link |
00:53:24.240
That's right.
link |
00:53:25.040
So do you think it's changing for the better?
link |
00:53:28.640
Somebody, I read this thing recently.
link |
00:53:31.040
It said the two disciplines with the highest GRE scores in college
link |
00:53:36.240
are physics and philosophy, right?
link |
00:53:39.600
And they're both sort of trying to answer the question,
link |
00:53:41.680
why is there anything, right?
link |
00:53:43.920
And the philosophers, you know, are on the kind of theological side
link |
00:53:47.600
and the physicists are obviously on the, you know, the material side.
link |
00:53:52.480
And there's a hundred billion galaxies with a hundred billion stars.
link |
00:53:56.800
It seems, well, repetitive at best.
link |
00:53:59.680
So, you know, there's on our way to 10 billion people.
link |
00:54:05.440
I mean, it's hard to say what it's all for, if that's what you're asking.
link |
00:54:08.960
Yeah, I guess, I guess I am.
link |
00:54:10.560
I mean, things do tend to significantly increase this in complexity.
link |
00:54:16.160
And I'm curious about how computation, like our world, our physical world,
link |
00:54:23.840
inherently generates mathematics.
link |
00:54:25.680
It's kind of obvious, right?
link |
00:54:26.720
So we have XYZ coordinates, you take a sphere, you make it bigger,
link |
00:54:30.000
you get a surface that falls, you know, grows by R squared.
link |
00:54:33.920
Like, it generally generates mathematics and the mathematicians
link |
00:54:37.360
and the physicists have been having a lot of fun talking to each other for years.
link |
00:54:41.120
And computation has been, let's say, relatively pedestrian.
link |
00:54:46.000
Like, computation in terms of mathematics has been doing binary algebra,
link |
00:54:51.840
while those guys have been galavanting through the other realms of possibility, right?
link |
00:54:58.000
Now, recently, the computation lets you do mathematical computations that are sophisticated
link |
00:55:05.680
enough that nobody understands how the answers came out, right?
link |
00:55:09.920
Machine learning.
link |
00:55:10.640
Machine learning.
link |
00:55:11.440
Yeah, yeah.
link |
00:55:11.760
Right, it used to be, you get data set, you guess at a function.
link |
00:55:16.640
The function is considered physics if it's predictive of new functions,
link |
00:55:20.960
new data sets.
link |
00:55:22.960
Modern, you can take a large data set with no intuition about what it is
link |
00:55:29.840
and use machine learning to find a pattern that has no function, right?
link |
00:55:34.160
And it can arrive at results that I don't know if they're completely
link |
00:55:38.320
mathematically describable.
link |
00:55:39.840
So computation has kind of done something interesting compared to A equal B plus C.
link |
00:55:46.000
There's something reminiscent of that step from the basic operations of addition
link |
00:55:54.640
to taking a step towards neural networks that's reminiscent of what life on earth
link |
00:55:58.880
at its origins was doing.
link |
00:56:00.880
Do you think we're creating sort of the next step in our evolution
link |
00:56:04.720
in creating artificial intelligence systems that will?
link |
00:56:07.920
I don't know.
link |
00:56:08.480
I mean, there's so much in the universe already, it's hard to say.
link |
00:56:12.560
What we stand in this whole thing.
link |
00:56:13.920
Are human beings working on additional abstraction layers and possibilities?
link |
00:56:18.320
Yeah, it appears so.
link |
00:56:20.160
Does that mean that human beings don't need dogs?
link |
00:56:22.880
You know, no.
link |
00:56:24.640
Like there's so many things that are all simultaneously interesting and useful.
link |
00:56:30.240
What you've seen throughout your career, you've seen great and greater level
link |
00:56:33.760
abstractions built in artificial machines, right?
link |
00:56:38.800
When you look at humans, you think of all life on earth as a single organism building
link |
00:56:45.360
this thing, this machine with greater and greater levels of abstraction.
link |
00:56:49.680
Do you think humans are the peak, the top of the food chain in this long arc of history
link |
00:56:57.200
on earth?
link |
00:56:58.160
Or do you think we're just somewhere in the middle?
link |
00:57:00.240
Are we the basic functional operations of a CPU?
link |
00:57:05.120
Are we the C++ program, the Python program, or with the neural network?
link |
00:57:11.760
People have calculated like how many operations does the brain do.
link |
00:57:15.920
I've seen the number 10 to the 18th about a bunch of times arrive different ways.
link |
00:57:20.480
So could you make a computer that did 10 to the 20th operations?
link |
00:57:23.680
Yes.
link |
00:57:24.240
Sure.
link |
00:57:25.040
So you think?
link |
00:57:25.600
We're going to do that.
link |
00:57:26.960
Now, is there something magical about how brains compute things?
link |
00:57:31.440
I don't know.
link |
00:57:31.920
My personal experience is interesting because you think you know how you think and then you
link |
00:57:37.920
have all these ideas and you can't figure out how they happened.
link |
00:57:41.280
And if you meditate, what you can be aware of is interesting.
link |
00:57:48.400
So I don't know if brains are magical or not.
link |
00:57:52.080
The physical evidence says no.
link |
00:57:54.560
Lots of people's personal experience says yes.
link |
00:57:56.720
So what would be funny is if brains are magical and yet we can make brains with more computation.
link |
00:58:04.480
You know, I don't know what to say about that, but.
link |
00:58:06.880
What do you think magic is an emergent phenomena?
link |
00:58:10.720
It could be.
link |
00:58:11.280
I have no explanation for it.
link |
00:58:13.200
Let me ask Jim Keller of what in your view is consciousness?
link |
00:58:19.120
With consciousness?
link |
00:58:20.400
Yeah, like what consciousness, love, things that are these deeply human things that seems
link |
00:58:28.160
to emerge from our brain?
link |
00:58:29.440
Is that something that we'll be able to make in code in chips that get faster and faster
link |
00:58:37.040
and faster and faster?
link |
00:58:38.000
That's like a 10 hour conversation.
link |
00:58:40.000
Nobody really knows.
link |
00:58:40.880
Can you summarize it in a couple of sentences?
link |
00:58:43.600
Many people have observed that organisms run at lots of different levels, right?
link |
00:58:51.280
If you had two neurons, somebody said you'd have one sensory neuron and one motor neuron,
link |
00:58:56.000
right?
link |
00:58:56.640
So we move towards things and away from things and we have physical integrity and safety or not,
link |
00:59:02.240
right?
link |
00:59:03.040
And then if you look at the animal kingdom, you can see brains that are a little more complicated.
link |
00:59:08.160
And at some point there's a planning system and then there's an emotional system that's,
link |
00:59:12.560
you know, happy about being safe or unhappy about being threatened, right?
link |
00:59:17.120
And then our brains have massive numbers of structures, you know, like planning and movement
link |
00:59:24.080
and thinking and feeling and drives and emotions.
link |
00:59:27.760
And we seem to have multiple layers of thinking systems.
link |
00:59:30.960
And we have a brain, a dream system that nobody understands whatsoever, which I find completely
link |
00:59:36.080
hilarious and you can think in a way that those systems are more independent and you can observe,
link |
00:59:46.400
you know, the different parts of yourself can observe them.
link |
00:59:49.360
I don't know which one's magical.
link |
00:59:51.200
I don't know which one's not computational.
link |
00:59:55.280
So is it possible that it's all computation?
link |
00:59:58.800
Probably.
link |
00:59:59.920
Is there a limit to computation?
link |
01:00:01.440
I don't think so.
link |
01:00:02.080
Do you think the universe is a computer?
link |
01:00:07.120
It's a weird kind of computer because if it was a computer, right, like when they do calculations
link |
01:00:14.320
on what it, how much calculation it takes to describe quantum effects is unbelievably high.
link |
01:00:20.800
So if it was a computer, when you built it out of something that was easier to compute,
link |
01:00:25.840
right, that's, that's a funny, it's a funny system.
link |
01:00:28.800
But then the simulation guys have pointed out that the rules are kind of interesting.
link |
01:00:32.000
Like when you look really close, it's uncertain and the speed of light says you can only look
link |
01:00:36.000
so far and things can't be simultaneous except for the odd entanglement problem where they seem
link |
01:00:40.960
to be like the rules are all kind of weird.
link |
01:00:44.320
And somebody said physics is like having 50 equations with 50 variables to define 50 variables.
link |
01:00:50.560
Like, you know, it's, you know, like physics itself has been a shit show for thousands of years.
link |
01:00:58.880
It seems odd when you get to the corners of everything, you know, it's either
link |
01:01:02.480
uncomputable or undefinable or uncertain.
link |
01:01:07.040
It's almost like the designers of the simulation are trying to prevent us from understanding it
link |
01:01:11.680
perfectly.
link |
01:01:12.720
But, but also the things that require calculations requires so much calculation that our idea of
link |
01:01:18.880
the universe of a computer is absurd because every single little bit of it takes all the
link |
01:01:23.680
computation in the universe to figure out.
link |
01:01:26.480
That's a weird kind of computer.
link |
01:01:27.920
You know, you say the simulation is running in the computer, which has, by definition,
link |
01:01:32.880
infinite computation.
link |
01:01:34.400
Not infinite.
link |
01:01:35.200
Oh, you mean if the universe is infinite?
link |
01:01:37.600
Yeah, well, every little piece of our universe seems to take infinite computation and figure out.
link |
01:01:43.120
Just a lot.
link |
01:01:44.080
Well, a lot, some pretty big number.
link |
01:01:45.920
Computer's little teeny spot takes all the mass in the local one light year by one light
link |
01:01:52.560
year space.
link |
01:01:53.280
It's close enough to infinite.
link |
01:01:54.720
Oh, it's a heck of a computer if it is one.
link |
01:01:56.480
I know it's, it's, it's a weird, it's a weird description because the simulation description
link |
01:02:01.520
seems to break when you look closely at it.
link |
01:02:04.720
But the rules of the universe seem to imply something's up.
link |
01:02:08.640
That seems a little arbitrary.
link |
01:02:10.800
The whole, the universe, the whole thing, the laws of physics.
link |
01:02:14.320
Yeah.
link |
01:02:14.800
It just seems like, like, how did it come out to be the way it is?
link |
01:02:20.000
Well, lots of people talk about that.
link |
01:02:21.200
It's, you know, it's, like I said, the two smartest groups of humans are working on the
link |
01:02:24.960
same problem.
link |
01:02:26.000
From different aspects.
link |
01:02:26.720
From different aspects, and they're both complete failures.
link |
01:02:29.920
So that's, that's kind of cool.
link |
01:02:32.480
They might succeed eventually.
link |
01:02:35.120
Well, after 2000 years, the trend isn't good.
link |
01:02:38.000
Oh, 2000 years is nothing in the span of the history of the universe.
link |
01:02:41.360
Not for sure.
link |
01:02:41.920
We have some time.
link |
01:02:43.120
But the next 1000 years doesn't look good either.
link |
01:02:45.200
So that's what everybody says at every stage.
link |
01:02:48.800
But with Moore's law, as you've just described, not being dead, the exponential
link |
01:02:53.600
growth of technology, the future seems pretty incredible.
link |
01:02:57.520
Well, it'll be interesting.
link |
01:02:58.400
That's for sure.
link |
01:02:59.440
That's right.
link |
01:03:00.240
So what are your thoughts on Ray Kurzweil's sense that exponential improvement and
link |
01:03:05.120
technology will continue indefinitely?
link |
01:03:07.200
That, is that how you see Moore's law?
link |
01:03:09.840
Do you see Moore's law more broadly in the sense that technology of all kinds has a way
link |
01:03:16.560
of stacking S curves on top of each other, where it'll be exponential, and then we'll
link |
01:03:22.320
see all kinds of.
link |
01:03:23.280
What does an exponential of a million mean?
link |
01:03:26.320
That's a pretty amazing number.
link |
01:03:28.080
And that's just for a local little piece of silicon.
link |
01:03:30.800
Now, let's imagine you, say, decided to get 1000 tons of silicon to collaborate in one
link |
01:03:38.480
computer at a million times the density.
link |
01:03:42.880
Now you're talking, I don't know, 10 to the 20th more computation power than our
link |
01:03:48.720
current already unbelievably fast computers.
link |
01:03:52.480
Nobody knows what that's going to mean.
link |
01:03:54.080
The sci fi guy is called competronium.
link |
01:03:56.880
Like when a local civilization turns the nearby star into a computer.
link |
01:04:02.240
I don't know if that's true.
link |
01:04:04.720
So just even when you shrink a transistor, that's only one dimension.
link |
01:04:12.480
The ripple effects of that.
link |
01:04:14.160
People tend to think about computers as a cost problem, right?
link |
01:04:17.520
So computers are made out of silicon and minor amounts of metals.
link |
01:04:21.920
And you know, this and that, none of those things cost any money.
link |
01:04:26.800
Like there's plenty of sand.
link |
01:04:29.760
Like you could just turn the beach and a little bit of ocean water into computers.
link |
01:04:33.280
So all the cost is in the equipment to do it.
link |
01:04:36.560
And the trend on equipment is once you figure out how to build the equipment,
link |
01:04:40.480
the trend of cost is zero.
link |
01:04:41.680
Elon said, first you figure out what configuration you want the atoms in
link |
01:04:47.360
and then how to put them there.
link |
01:04:49.680
Right?
link |
01:04:50.160
Yeah.
link |
01:04:50.720
Because well, what you hear that, you know, his, his great insight is people are how constrained.
link |
01:04:56.320
I have this thing.
link |
01:04:57.040
I know how it works.
link |
01:04:58.560
And then little tweaks to that will generate something as opposed to what
link |
01:05:03.200
do I actually want and then figure out how to build it.
link |
01:05:06.880
It's a very different mindset and almost nobody has it, obviously.
link |
01:05:12.720
Well, let me ask on that topic.
link |
01:05:15.600
You were one of the key early people in the development of autopilot,
link |
01:05:20.000
at least in the hardware side.
link |
01:05:22.320
Elon Musk believes that autopilot and vehicle autonomy, if you just look at that problem,
link |
01:05:26.560
can follow this kind of exponential improvement in terms of the hot,
link |
01:05:30.800
the how question that we're talking about.
link |
01:05:32.400
There's no reason why you can't.
link |
01:05:34.560
What are your thoughts on this particular space of vehicle autonomy?
link |
01:05:39.920
And you're a part of it and Elon Musk's and Tesla's vision for the computer you need to build
link |
01:05:46.640
was straightforward.
link |
01:05:48.640
And you could argue, well, doesn't need to be two times faster or five times or 10 times.
link |
01:05:54.400
But that's just a matter of time or price in the short run.
link |
01:05:58.320
So that's, that's not a big deal.
link |
01:06:00.080
You don't have to be especially smart to drive a car.
link |
01:06:03.120
So it's not like a super hard problem.
link |
01:06:05.520
I mean, the big problem of safety is attention, which computers are really good at, not skills.
link |
01:06:12.800
Well, let me push back on one.
link |
01:06:15.120
You say everything you said is correct, but we as humans tend to,
link |
01:06:23.040
tend to take for granted how, how incredible our vision system is.
link |
01:06:26.800
So you can drive a car with 2050 vision and you can train a neural network to extract the
link |
01:06:33.120
distance of any object in the shape of any surface from a video and data.
link |
01:06:38.640
But that's really simple.
link |
01:06:40.080
No, it's not simple.
link |
01:06:42.000
That's a simple data problem.
link |
01:06:44.320
It's not, it's not simple.
link |
01:06:46.480
It's because you, because it's not just detecting objects.
link |
01:06:50.320
It's understanding the scene and it's being able to do it in a way that doesn't make errors.
link |
01:06:55.520
So the, the beautiful thing about the human vision system and our entire brain around the
link |
01:07:00.400
whole thing is we're able to fill in the gaps.
link |
01:07:04.240
It's not just about perfectly detecting cars.
link |
01:07:06.800
It's inferring the occluded cars.
link |
01:07:08.560
It's trying to, it's, it's understanding the statistics.
link |
01:07:11.360
I think that's mostly a data problem.
link |
01:07:13.360
So you think what data would compute with improvement of computation with improvement
link |
01:07:19.040
in collection?
link |
01:07:19.600
Well, there's a, you know, when you're driving a car and somebody cuts you off,
link |
01:07:22.400
your brain has theories about why they did it.
link |
01:07:24.720
You know, they're a bad person, they're distracted, they're dumb.
link |
01:07:28.560
You know, you can listen to yourself, right?
link |
01:07:31.360
So, you know, if you think that narrative is important to be able to successfully drive
link |
01:07:36.800
a car, then current autopilot systems can't do it.
link |
01:07:40.320
But if cars are ballistic things with tracks and probabilistic changes of speed and direction
link |
01:07:45.680
and roads are fixed and given, by the way, they don't change dynamically, right?
link |
01:07:51.920
Right, you can map the world really thoroughly.
link |
01:07:56.240
You can place every object really thoroughly, right?
link |
01:08:01.360
You can calculate trajectories of things really thoroughly, right?
link |
01:08:06.400
But everything you said about really thoroughly has a different degree of difficulty.
link |
01:08:12.480
So.
link |
01:08:12.960
And you could say at some point, computer autonomous systems will be way better at
link |
01:08:18.240
things that humans are lousy at.
link |
01:08:19.920
Like, they'll be better at attention.
link |
01:08:22.320
They'll always remember there was a pothole on the road that humans keep forgetting about.
link |
01:08:27.120
They'll remember that this set of roads has these weirdo lines on it that the computers
link |
01:08:31.920
figured out once and especially if they get updates so that somebody changes a given.
link |
01:08:37.840
Like, the key to robots and stuff somebody said is to maximize the givens.
link |
01:08:43.760
Right, right.
link |
01:08:45.040
So having a robot pick up this bottle cap is way easier to put a red dot on the top because
link |
01:08:50.960
then you'll have to figure out, you know, and if you want to do a certain thing with it,
link |
01:08:54.320
you know, maximize the givens is the thing.
link |
01:08:56.960
And autonomous systems are happily maximizing the givens.
link |
01:09:00.880
Like humans, when you drive someplace new, you remember it because you're processing it the
link |
01:09:06.160
whole time and after the 50th time you drove to work, you get to work, you don't know how you got
link |
01:09:10.240
there, right?
link |
01:09:11.280
You're on autopilot, right?
link |
01:09:13.760
Autonomous cars are always on autopilot, but the cars have no theories about why they got cut off
link |
01:09:20.160
or why they're in traffic.
link |
01:09:22.000
So that's never stopped paying attention.
link |
01:09:24.640
Right.
link |
01:09:25.200
So I tend to believe you do have deaf theories met the models of other people,
link |
01:09:29.760
especially with pedestrian cyclists, but also with other cars.
link |
01:09:32.560
So everything you said is like, is actually essential to driving.
link |
01:09:38.800
Driving is a lot more complicated than people realize.
link |
01:09:41.600
I think so sort of to push back slightly, but to cut into traffic, right?
link |
01:09:46.320
Yeah.
link |
01:09:46.880
You can't just wait for a gap.
link |
01:09:48.240
You have to be somewhat aggressive.
link |
01:09:50.080
You'll be surprised how simple the calculation for that is.
link |
01:09:53.600
I may be on that particular point, but there's a, maybe I should have to push back.
link |
01:10:00.160
I would be surprised.
link |
01:10:01.520
You know what?
link |
01:10:01.840
Yeah, I'll just say where I stand.
link |
01:10:02.880
I would be very surprised, but I think it's, you might be surprised how complicated it is.
link |
01:10:08.320
I'd say I tell people like progress disappoints in the short run surprises in the long run.
link |
01:10:13.680
It's very possible.
link |
01:10:14.720
Yeah, I suspect in 10 years it'll be just like taken for granted.
link |
01:10:18.880
Yeah, probably.
link |
01:10:19.680
But you're probably right.
link |
01:10:21.360
It's going to be a $50 solution that nobody cares about.
link |
01:10:24.800
It's like GPS is like, wow, GPS.
link |
01:10:26.960
We have satellites in space that tell you where your location is.
link |
01:10:30.880
It was a really big deal.
link |
01:10:31.840
Now everything has a GPS in it.
link |
01:10:33.360
Yeah, that's true.
link |
01:10:33.920
But I do think that systems that involve human behavior are more complicated than we give them
link |
01:10:40.160
credit for.
link |
01:10:40.720
So we can do incredible things with technology that don't involve humans.
link |
01:10:44.800
But when you...
link |
01:10:45.360
I think humans are less complicated than people, you know, frequently
link |
01:10:49.280
subscribe.
link |
01:10:50.400
Maybe I feel...
link |
01:10:51.040
We tend to operate out of large numbers of patterns and just keep doing it over and over.
link |
01:10:55.600
But I can't trust you because you're a human.
link |
01:10:57.920
That's something, something a human would say.
link |
01:11:00.640
But my hope is on the point you've made is even if, no matter who's right,
link |
01:11:08.080
there, I'm hoping that there's a lot of things that humans aren't good at that machines are
link |
01:11:12.720
definitely good at.
link |
01:11:13.280
Like you said, attention and things like that.
link |
01:11:15.520
Well, there'll be so much better that the overall picture of safety and autonomy will be
link |
01:11:21.280
obviously cars will be safer, even if they're not as good.
link |
01:11:24.000
No, I'm a big believer in safety.
link |
01:11:26.160
I mean, there are already the current safety systems like cruise control that doesn't let
link |
01:11:30.960
you run into people and lane keeping.
link |
01:11:33.120
There are so many features that you just look at the Pareto of accidents and knocking off
link |
01:11:38.240
like 80% of them is super doable.
link |
01:11:42.240
Just to linger on the autopilot team and the efforts there, the...
link |
01:11:47.840
It seems to be that there is a very intense scrutiny by the media and the public in terms
link |
01:11:53.440
of safety, the pressure, the bar put before autonomous vehicles.
link |
01:11:57.760
What are your, sort of, as a person there working on the hardware and trying to build
link |
01:12:03.200
a system that builds a safe vehicle and so on, what was your sense about that pressure?
link |
01:12:08.800
Is it unfair?
link |
01:12:09.680
Is it expected of new technology?
link |
01:12:12.080
Yeah, it seems reasonable.
link |
01:12:13.360
I was interested.
link |
01:12:14.000
I talked to both American and European regulators and I was worried that the regulations would
link |
01:12:21.280
write into the rules, technology solutions like modern brake systems imply hydraulic brakes.
link |
01:12:29.840
So if you read the regulations to meet the letter of the law for brakes, it sort of has
link |
01:12:35.600
to be hydraulic, right?
link |
01:12:37.680
And the regulator said they're interested in the use cases like a head on crash, an offset
link |
01:12:43.600
crash, don't hit pedestrians, don't run into people, don't leave the road, don't run a red
link |
01:12:49.280
light or a stoplight. They were very much into the scenarios and they had all the data about
link |
01:12:55.760
which scenarios injured or killed the most people and for the most part those conversations were
link |
01:13:03.520
like what's the right thing to do to take the next step.
link |
01:13:08.560
Now Elon's very interested in also in the benefits of autonomous driving or freeing
link |
01:13:13.280
people's time and attention as well as safety. And I think that's also an interesting thing but
link |
01:13:22.960
building autonomous systems so they're safe and safer than people seemed.
link |
01:13:27.280
Since the goal is to be 10x safer than people, having the bar to be safer than people and
link |
01:13:32.560
scrutinizing accidents seems philosophically correct.
link |
01:13:39.120
So I think that's a good thing.
link |
01:13:40.160
What are, it's different than the things that you worked at in teleMD,
link |
01:13:48.480
Apple with autopilot chip design and hardware design. What are interesting or challenging
link |
01:13:54.800
aspects of building this specialized kind of computing system in the automotive space?
link |
01:14:00.080
I mean there's two tricks to building like an automotive computer. One is the software team,
link |
01:14:05.360
the machine learning team is developing algorithms that are changing fast. So as you're building the
link |
01:14:12.720
accelerator you have this you know worry or intuition that the algorithms will change enough
link |
01:14:18.400
that the accelerator will be the wrong one, right? And there's the generic thing which is if you
link |
01:14:25.040
build a really good general purpose computer say its performance is one and then GPU guys will
link |
01:14:31.440
deliver about 5x to performance for the same amount of silicon because instead of discovering
link |
01:14:36.800
parallelism you're given parallelism. And then special accelerators get another 2 to 5x on top
link |
01:14:44.000
of a GPU because you say I know the math is always 8 bit integers into 32 bit accumulators
link |
01:14:50.960
and the operations are the subset of mathematical possibilities. So auto you know AI accelerators
link |
01:14:58.240
have a claimed performance benefit over GPUs because in the narrow math space
link |
01:15:04.960
you're nailing the algorithm. Now you still try to make it programmable but the AI field is changing
link |
01:15:11.920
really fast. So there's a you know there's a little creative tension there of I want the
link |
01:15:17.600
acceleration afforded by specialization without being over specialized so that the new algorithm is
link |
01:15:23.920
so much more effective that you would have been better off on a GPU. So there's a tension there
link |
01:15:29.840
to build a good computer for an application like automotive. There's all kinds of sensor inputs
link |
01:15:36.080
and safety processors and a bunch of stuff. So one of Elon's goals to make it super affordable.
link |
01:15:42.080
So every car gets an autopilot computer. So some of the recent startups you look at and
link |
01:15:46.400
they have a server and a trunk because they're saying I'm going to build this autopilot computer
link |
01:15:50.640
replaces the driver. So their cost budget is $10,000 or $20,000 and Elon's constraint was I'm going
link |
01:15:57.040
to put one every in every car whether people buy auto ton of striping or not. So the cost
link |
01:16:02.480
constraint he had in mind was great. Right. And to hit that you had to think about the system design
link |
01:16:08.240
that's complicated. It's fun. You know it's like it's like it's Crestman's work like a violin maker
link |
01:16:13.840
right. You can say Strativarius is this incredible thing. The musicians are incredible but the guy
link |
01:16:18.800
making the violin you know picked wood and sanded it and then he cut it you know and he glued it
link |
01:16:25.280
you know and he waited for the right day so that when he put the finish on it it didn't you know
link |
01:16:30.320
do something dumb. That's Crestman's work right. You may be a genius craftsman because you have
link |
01:16:35.760
the best techniques and you discover a new one but most engineers Crestman's work and humans
link |
01:16:42.320
really like to do that. Smart humans. No everybody. All humans. I don't know. I used to I dug ditches
link |
01:16:48.960
when I was in college. I got really good at it. Satisfying. Digging ditches is also craftsman
link |
01:16:55.120
work. Yeah of course. So there's an expression called complex mastery behavior. So when you're
link |
01:17:01.200
learning something that's fine because you're learning something. When you do something it's
link |
01:17:04.880
wrote and simple. It's not that satisfying but if the steps that you have to do are complicated
link |
01:17:09.680
and you're good at them it's satisfying to do them. And then if you're intrigued by it all as
link |
01:17:16.880
you're doing them you sometimes learn new things that you can raise your game but Crestman's work
link |
01:17:22.160
is good. And engineers like engineering is complicated enough that you have to learn a lot
link |
01:17:27.920
of skills and then a lot of what you do is then craftsman's work which is fun.
link |
01:17:33.280
Autonomous driving building a very resource constrained computer. So a computer has to be cheap enough
link |
01:17:39.360
that put in every single car. That's essentially boils down to craftsman's work. It's engineering.
link |
01:17:46.000
You know there's thoughtful decisions and problems to solve and tradeoffs to make. You
link |
01:17:50.560
need 10 camera imports or eight. You know you're building for the current car or the next one.
link |
01:17:55.840
You know how do you do the safety stuff. You know there's a whole bunch of details
link |
01:17:59.680
but it's fun. But it's not like I'm building a new type of neural network which has a new
link |
01:18:05.200
mathematics and a new computer to work. You know that's like there's more invention than that.
link |
01:18:12.240
But the rejection to practice once you pick the architecture you look inside and what do you see.
link |
01:18:16.960
Adders and multipliers and memories and you know the basics. So computers is always this weird set
link |
01:18:24.320
of abstraction layers of ideas and thinking that reduction to practice is transistors and wires and
link |
01:18:32.000
you know pretty basic stuff and that's an interesting phenomena. By the way that like factory work
link |
01:18:38.640
like lots of people think factory work is road assembly stuff. I've been on the assembly line
link |
01:18:44.080
like the people work there really like it. It's a really great job. It's really complicated putting
link |
01:18:48.800
cars together is hard right and the cars moving and the parts are moving and sometimes the parts are
link |
01:18:54.320
damaged and you have to coordinate putting all the stuff together and people are good at it.
link |
01:18:58.880
They're good at it. And I remember one day I went to work and the line was shut down for some reason
link |
01:19:03.840
and some of the guys sitting around were really bummed because they had reorganized a bunch of
link |
01:19:08.720
stuff and they were going to hit a new record for the number of cars built that day and they
link |
01:19:12.880
were all gun hoe to do it and these are big tough buggers and you know but what they did was complicated
link |
01:19:19.040
and you couldn't do it. Yeah and I mean well after a while you could but you'd have to work your way
link |
01:19:23.840
up because you know like putting the bright what's called the brights the trim on a car
link |
01:19:30.880
on a moving assembly line where it has to be attached 25 places in a minute and a half
link |
01:19:35.360
is unbelievably complicated and human beings can do it. It's really good. I think that's
link |
01:19:42.720
harder than driving a car by the way. Putting together working on a factory. Two smart people
link |
01:19:50.080
can disagree. Yay. I think driving a car. We'll get you in the factory someday and then we'll see
link |
01:19:56.800
how you do it. No not for us humans driving a car is easy. I'm saying building a machine
link |
01:20:00.880
that drives a car is not easy. Okay. Okay. Driving a car is easy for humans because
link |
01:20:07.680
we've been evolving for billions of years. Drive cars. Yeah I noticed the pale with the cars are
link |
01:20:14.560
super cool. No. Now you join the rest of the unit and mocking me. Okay. I wasn't mocking you. I was
link |
01:20:22.240
just intrigued by your you know your anthropology. Yeah. I have to go dig into that. There's some
link |
01:20:29.200
inaccuracies there. Yes. Okay. But in general what have you learned in terms of thinking about
link |
01:20:40.960
passion, craftsmanship, tension, chaos. Jesus. The whole mess of it. What have you learned
link |
01:20:51.840
and have taken away from your time working with Elon Musk working at Tesla which is known to be
link |
01:20:58.560
a place of chaos, innovation, craftsmanship and all those things. I really like the way you thought.
link |
01:21:05.280
Like you think you have an understanding about what first principles of something is and then
link |
01:21:10.160
you talk to Elon about it and you didn't scratch the surface. You know he has a deep
link |
01:21:16.240
belief that no matter what you do is a local maximum. Right. I had a friend he invented a
link |
01:21:22.320
better electric motor and it was like a lot better than what we were using and one day he came by
link |
01:21:27.760
he said you know I'm a little disappointed because you know this is really great and you didn't seem
link |
01:21:32.480
that impressed and I said you know when the super intelligent aliens come are they gonna be looking
link |
01:21:37.840
for you. Like where is he? The guy built the motor. Yeah. Probably not you know like but doing
link |
01:21:47.520
interesting work that's both innovative and let's say craftsman's work on the current thing is really
link |
01:21:52.160
satisfying and it's good and that's cool and then Elon was good at taking everything apart
link |
01:21:58.880
and like what's the deep first principle. Oh no what's really no what's really you know you know
link |
01:22:04.400
you know that that you know ability to look at it without assumptions and and how constraints
link |
01:22:11.600
is super wild. You know he built rocket ship and electric car and you know everything
link |
01:22:19.840
and that's super fun and he's into it too. Like when they first landed two SpaceX rockets to Tesla
link |
01:22:25.920
we had a video projector in the big room and like 500 people came down and when they landed
link |
01:22:30.320
everybody cheered and some people cried it was so cool. All right but how did you do that? Well
link |
01:22:38.080
it was super hard and then people say well it's chaotic really to get out of all your
link |
01:22:43.920
assumptions you think that's not going to be unbelievably painful and it was Elon tough
link |
01:22:49.920
yeah probably the people look back on it and say boy I'm really happy I had that experience to go
link |
01:22:57.680
take apart that many layers of assumptions sometimes super fun sometimes painful.
link |
01:23:05.200
So it could be emotionally and intellectually painful that whole process is just stripping away
link |
01:23:09.520
assumptions. Yeah imagine 99% of your thought process is protecting your self conception
link |
01:23:16.400
and 98% of that's wrong. Yeah. Now you got the math right. How do you think you're feeling when
link |
01:23:23.600
you get back into that one bit that's useful and now you're open and you have the ability to do
link |
01:23:28.720
something different. I don't know if I got the math right it might be 99.9 but it ain't 50.
link |
01:23:38.560
Imagining it the 50% is hard enough. Yeah. Now for a long time I've suspected you could get better
link |
01:23:48.240
like you can think better you can think more clearly you can take things apart
link |
01:23:51.680
and there's lots of examples of that people who do that.
link |
01:23:58.240
So and Elon is an example of that apparently you are an example so I don't know if I am
link |
01:24:04.320
I'm fun to talk to certainly I've learned a lot of stuff right well here's the other thing is
link |
01:24:10.000
like I joke like like I read books and people think oh you read books well no I've read a
link |
01:24:15.440
couple of books a week for 55 years well maybe 50 because I didn't read learn read until I was
link |
01:24:22.880
eight or something and it turns out when people write books they often take 20 years of their life
link |
01:24:31.120
where they passionately did something reduce it to 200 pages that's kind of fun and then
link |
01:24:38.160
you go online and you can find out who wrote the best books and who like you know that's kind of
link |
01:24:42.880
wild so there's this wild selection process and then you can read it and for the most part understand
link |
01:24:48.160
it and then you can go apply it like I went to one company I thought I haven't managed much before
link |
01:24:54.880
so I read 20 management books and I started talking to him basically compared to all the
link |
01:25:00.000
VPs running around I'd run night read 19 more management books than anybody else
link |
01:25:05.040
wasn't even that hard yeah and half the stuff worked like first time it wasn't even rocket science
link |
01:25:13.360
but at the core of that is questioning the assumptions or sort of entering the thinking
link |
01:25:19.840
first person principles thinking sort of looking at the reality of the situation and using using
link |
01:25:26.720
that knowledge applying that knowledge so yeah so I would say my brain has this idea that you can
link |
01:25:31.920
question first assumptions and but I can go days at a time and forget that and you have to kind of
link |
01:25:39.120
like circle back that observation because it is because it's hard well it's hard to just keep it
link |
01:25:46.400
front and center because you know you're you operate on so many levels all the time and
link |
01:25:51.040
you know getting this done takes priority or you know being happy takes priority or you know
link |
01:25:57.120
you know screwing around takes priority like like like how you go through life is complicated
link |
01:26:02.880
and then you remember oh yeah I could really uh think first principles you know shit that's
link |
01:26:07.120
that's tiring you know but you do for a while and that's kind of cool
link |
01:26:12.640
so just as a last question in your sense from the big picture from the first principles
link |
01:26:19.440
do you think you kind of answered already but do you think autonomous driving is something
link |
01:26:24.880
we can solve on a timeline of years so one two three five ten years as opposed to a century
link |
01:26:33.680
yeah definitely just to linger on it a little longer where's the confidence coming from is it
link |
01:26:40.480
the fundamentals of the problem the fundamentals of building the hardware and the software
link |
01:26:45.600
where as a computational problem understanding ballistics roles topography it seems pretty
link |
01:26:54.960
solvable I mean and you can see this you know like like speech recognition for a long time
link |
01:27:00.160
people are doing you know frequency and domain analysis and and all kinds of stuff and that
link |
01:27:04.400
didn't work for at all right and then they did deep learning about it and it worked great
link |
01:27:09.920
and it took multiple iterations and you know autonomous driving is way past the frequency
link |
01:27:18.640
analysis point you know use radar don't run into things and the data gathering is going up and the
link |
01:27:25.440
computation is going up and the algorithm understanding is going up and there's a whole
link |
01:27:29.280
bunch of problems getting solved like that the data side is really powerful but I disagree with
link |
01:27:34.560
both you and you and I'll tell you on once again as I did before that that when you add human beings
link |
01:27:41.200
into the picture the it's no longer a ballistics problem it's something more complicated but I
link |
01:27:48.240
could be very well proven wrong and cars are highly damped in terms of rate of change like the
link |
01:27:54.080
steering and the steering system is really slow compared to a computer the acceleration the acceleration
link |
01:27:59.360
is really slow yeah on a certain time scale on a ballistics time scale but human behavior I don't
link |
01:28:05.120
know yet I I shouldn't say the beans are really slow too we weirdly we operate you know half a
link |
01:28:11.520
second behind reality I'll be really understands that one either it's pretty funny yeah yeah so
link |
01:28:19.760
yeah I would be with very well could be surprised and I think with the rate of improvement in all
link |
01:28:25.440
aspects on both the compute and the the software and the hardware there's going to be pleasant
link |
01:28:30.400
surprises all over the place yeah speaking of unpleasant surprises many people have worries
link |
01:28:37.760
about a singularity in the development of AI forgive me for such questions yeah when AI improves
link |
01:28:45.200
the exponential and reaches a point of superhuman level general intelligence you know beyond the
link |
01:28:51.600
point there's no looking back do you share this worry of existential threats from artificial
link |
01:28:56.640
intelligence from computers becoming superhuman level intelligent no not really you know like we
link |
01:29:04.640
already have a very stratified society and then if you look at the whole animal kingdom of capabilities
link |
01:29:10.240
and abilities and interests and you know smart people have their niche and you know normal people
link |
01:29:16.880
have their niche and craftsmen's have their niche and you know animals have their niche I
link |
01:29:22.880
suspect that the domains of interest for things that you know astronomically different like the
link |
01:29:29.680
whole something got 10 times smarter than us and wanted to track us all down because what
link |
01:29:34.480
we like to have coffee at Starbucks like it doesn't seem plausible no is there an existential problem
link |
01:29:40.560
that how do you live in a world where there's something way smarter than you and you based
link |
01:29:44.960
your kind of self esteem on being the smartest local person well there's what 0.1 of the population
link |
01:29:51.200
who thinks that because the rest of the population has been dealing with it since they were born
link |
01:29:56.640
so the the breath of possible experience that can be interesting is really big
link |
01:30:04.800
and you know superintelligence seems likely although we still don't know if we're magical
link |
01:30:13.040
but I suspect we're not and it seems likely that will create possibilities that are interesting
link |
01:30:19.520
for us and its its interests will be interesting for that for whatever it is it's not obvious why
link |
01:30:28.080
its interests would somehow want to fight over some square foot of dirt or you know whatever
link |
01:30:34.800
you know the usual fears are about so you don't think you'll inherit some of the darker aspects
link |
01:30:40.160
of human nature depends on how you think reality is constructed so for for whatever reason human
link |
01:30:48.240
beings are in let's say creative tension and opposition with both our good and bad forces
link |
01:30:55.200
like there's lots of philosophical understanding of that right I don't know why that would be
link |
01:31:01.680
different so you think the evil is is necessary for the good I mean the the tension I don't know
link |
01:31:08.320
about evil but like we live in a competitive world where your good is somebody else's
link |
01:31:14.960
you know evil you know there's there's the malignant part of it but that seems to be
link |
01:31:21.760
self limiting although occasionally it's it's super horrible
link |
01:31:27.280
but yes there's a debate over ideas and some people have different beliefs and that that
link |
01:31:32.880
debate itself is a process so that at arriving at something yeah and why wouldn't that continue
link |
01:31:39.200
yeah just you but you don't think that whole process will leave humans behind in a way that's
link |
01:31:45.280
painful emotionally painful yes for the one for the point one percent they'll be you know why isn't
link |
01:31:51.280
it already painful for a large percentage of the population and it is I mean society does have a
link |
01:31:55.920
lot of stress in it about the one percent and the about to this and about to that but you know
link |
01:32:01.120
everybody has a lot of stress in a life about what they find satisfying and
link |
01:32:06.240
and you know know yourself seems to be the proper dictum and pursue something that makes
link |
01:32:12.400
your life meaningful seems proper and there's so many avenues on that like there's so much
link |
01:32:19.920
unexplored space at every single level you know I'm I'm somewhat of my nephew called me a jaded
link |
01:32:28.480
optimist and you know so it's there's a beautiful tension that in that label but if you were to
link |
01:32:39.040
look back at your life and could relive a moment a set of moments because there were the happiest
link |
01:32:47.600
times of your life outside of family what would that be I don't want to relive any moments I like
link |
01:32:56.960
that I like that situation where you have some amount of optimism and then the the anxiety of the
link |
01:33:04.000
unknown so you love the unknown the the mystery of it I don't know about the mystery you sure get
link |
01:33:11.600
your blood pumping what do you think is the meaning of this whole thing of life on this
link |
01:33:18.800
pale blue dot it seems to be what it does like the universe for whatever reason makes atoms
link |
01:33:30.000
which makes us which we do stuff and we figure out things and we explore things and that's
link |
01:33:38.080
just what it is it's not just yeah it is you know Jim I don't think there's a better place to end
link |
01:33:46.480
it it's a huge honor and uh well that was super fun thank you so much for talking today all right
link |
01:33:52.640
great thanks for listening to this conversation and thank you to our presenting sponsor cash app
link |
01:33:59.200
download it use code lex podcast you'll get $10 and $10 will go to first a stem education
link |
01:34:05.600
nonprofit that inspires hundreds of thousands of young minds to become future leaders and
link |
01:34:10.880
innovators if you enjoy this podcast subscribe on youtube give it five stars an apple podcast
link |
01:34:16.880
follow on spotify support on patreon or simply connect with me on twitter and now let me leave
link |
01:34:23.360
you with some words of wisdom from gordon more if everything you try works you aren't trying hard
link |
01:34:30.240
enough thank you for listening and hope to see you next time