back to indexJim Keller: The Future of Computing, AI, Life, and Consciousness | Lex Fridman Podcast #162
link |
The following is a conversation with Jim Keller,
link |
his second time in the podcast.
link |
Jim is a legendary microprocessor architect
link |
and is widely seen as one of the greatest
link |
engineering minds of the computing age.
link |
In a peculiar twist of space time in our simulation,
link |
Jim is also a brother in law of Jordan Peterson.
link |
We talk about this and about computing,
link |
artificial intelligence, consciousness, and life.
link |
Quick mention of our sponsors.
link |
Athletic Greens all in one nutrition drink,
link |
Brooklyn and Sheets, ExpressVPN,
link |
and Bell Campo grass fed meat.
link |
Click the sponsor links to get a discount
link |
and to support this podcast.
link |
As a side note, let me say that Jim is someone
link |
who on a personal level inspired me to be myself.
link |
There was something in his words on and off the mic
link |
or perhaps that he even paid attention to me at all
link |
that almost told me, you're all right kid.
link |
A kind of pat on the back that can make the difference
link |
between the mind that flourishes
link |
and a mind that is broken down
link |
by the cynicism of the world.
link |
So I guess that's just my brief few words
link |
of thank you to Jim and in general,
link |
gratitude for the people who have given me a chance
link |
on this podcast and my work and in life.
link |
If you enjoy this thing, subscribe on YouTube,
link |
review it on Apple podcast, follow on Spotify,
link |
support on Patreon or connect with me
link |
on Twitter, Alex Friedman.
link |
And now here's my conversation with Jim Keller.
link |
What's the value and effectiveness of theory
link |
versus engineering, this dichotomy
link |
in building good software or hardware systems?
link |
Well, it's good designs both.
link |
I guess that's pretty obvious.
link |
But engineering, do you mean, you know,
link |
reduction of practice of known methods?
link |
And then science is the pursuit of discovering things
link |
that people don't understand
link |
or solving unknown problems.
link |
Definitions are interesting here,
link |
but I was thinking more in theory,
link |
constructing models that kind of generalize
link |
about how things work.
link |
And engineering is actually building stuff,
link |
the pragmatic like, okay, we have these nice models,
link |
but how do we actually get things to work?
link |
Maybe economics is a nice example.
link |
Like economists have all these models
link |
of how the economy works
link |
and how different policies will have an effect.
link |
But then there's the actual, okay,
link |
let's call it engineering
link |
of like actually deploying the policies.
link |
So computer design is almost all engineering
link |
and reduction of practice of known methods.
link |
Now, because of the complexity of the computers we build,
link |
you know, you could think you're,
link |
well, we'll just go write some code
link |
and then we'll verify it and then we'll put it together.
link |
And then you find out that the combination
link |
of all that stuff is complicated.
link |
And then you have to be inventive
link |
to figure out how to do it, right?
link |
So that's, that's definitely happens a lot.
link |
And then every so often some big idea happens,
link |
but it might be one person.
link |
And that idea is in what,
link |
in the space of engineering or is it in the space?
link |
Well, I'll give you an example.
link |
So one of the limits of computer performance
link |
is branch prediction.
link |
So, and there's a whole bunch of ideas
link |
about how good you could predict a branch.
link |
And people said, there's a limit to it.
link |
It's an asymptotic curve.
link |
And somebody came up with a better way
link |
to do branch prediction.
link |
That was a lot better.
link |
And he published a paper on it
link |
and every computer in the world now uses it.
link |
And it was one idea.
link |
So the engineers who build branch prediction hardware
link |
were happy to drop the one kind of training array
link |
and put it in another one.
link |
So it was, it was a real idea.
link |
And branch prediction is,
link |
is one of the key problems underlying all of sort of
link |
the lowest level of software it boils down to branch prediction.
link |
It boils down to uncertainty.
link |
Computers are limited by, you know,
link |
single thread computers limited by two things.
link |
The predictability of the path of the branches
link |
and the predictability of the locality of data.
link |
So we have predictors that now predict
link |
both of those pretty well.
link |
So memory is, you know, a couple hundred cycles away.
link |
Local cash is a couple of cycles away.
link |
When you're executing fast,
link |
virtually all the data has to be in the local cash.
link |
So a simple program says, you know,
link |
add one to every element in an array.
link |
It's really easy to see what the stream of data will be.
link |
But you might have a more complicated program that's, you know,
link |
says get a, get an element of this array,
link |
look at something, make a decision,
link |
go get another element, it's kind of random.
link |
And you can think that's really unpredictable.
link |
And then you make this big predictor
link |
that looks at this kind of pattern.
link |
And you realize, well, if you get this data
link |
and this data, then you probably want that one.
link |
And if you get this one and this one
link |
and this one, you probably want that one.
link |
And is that theory or is that engineering?
link |
Like the paper that was written, was it asymptotic
link |
kind of discussion, or is it more like,
link |
here's a hack that works well?
link |
It's a little bit of both.
link |
Like there's information theory in it, I think somewhere.
link |
So it's actually trying to prove some kind of stuff.
link |
But once you know the method,
link |
implementing it is an engineering problem.
link |
Now there's a flip side of this,
link |
which is in a big design team,
link |
what percentage of people think their plan
link |
or their life's work is engineering
link |
versus inventing things.
link |
So lots of companies will reward you for filing patents.
link |
Some many big companies get stuck
link |
because to get promoted, you have to come up
link |
with something new.
link |
And then what happens is everybody's trying
link |
to do some random new thing, 99% of which doesn't matter.
link |
And the basics get neglected.
link |
And, or they get to, there's a dichotomy, they think,
link |
like the cell library and the basic CAD tools,
link |
or basic software validation methods.
link |
That's simple stuff.
link |
They wanna work on the exciting stuff.
link |
And then they spend lots of time trying to figure out
link |
how to patent something.
link |
And that's mostly useless.
link |
But the breakthroughs are on simple stuff.
link |
No, no, you have to do the simple stuff really well.
link |
If you're building a building out of bricks,
link |
you want great bricks.
link |
So you go to two places to sell bricks.
link |
So one guy says, yeah, they're over there in an ugly pile.
link |
And the other guy is like lovingly tells you
link |
about the 50 kinds of bricks and how hard they are
link |
and how beautiful they are and how square they are.
link |
And, you know, which one are you gonna buy bricks from?
link |
Which is gonna make a better house.
link |
So you're talking about the craftsman,
link |
the person who understands bricks,
link |
who loves bricks, who loves the variety.
link |
That's a good word.
link |
You know, good engineering is great craftsmanship.
link |
And when you start thinking engineering is about invention
link |
and you set up a system that rewards invention,
link |
the craftsmanship gets neglected.
link |
Okay, so maybe one perspective is the theory.
link |
The science overemphasizes invention
link |
and engineering emphasizes craftsmanship.
link |
And therefore, like, so if you,
link |
it doesn't matter what you do, theory.
link |
But everybody does, like read the tech rags.
link |
They're always talking about some breakthrough
link |
or innovation and everybody thinks
link |
that's the most important thing.
link |
But the number of innovative ideas
link |
is actually relatively low.
link |
We need them, right?
link |
And innovation creates a whole new opportunity.
link |
Like when some guy invented the internet, right?
link |
Like that was a big thing.
link |
The million people that wrote software against that
link |
were mostly doing engineering and software writing.
link |
So the elaboration of that idea was huge.
link |
I don't know if you know Brandon and I,
link |
he wrote JavaScript in 10 days.
link |
That's an interesting story.
link |
It makes me wonder, and it was, you know,
link |
famously for many years considered
link |
to be a pretty crappy programming language.
link |
It's been improving sort of consistently.
link |
But the interesting thing about that guy is,
link |
you know, he doesn't get any awards.
link |
You don't get a Nobel Prize or a Fields Medal
link |
or a crappy piece of, you know, software code.
link |
That is currently the number one programming language
link |
in the world and runs now is increasingly running
link |
the back end of the internet.
link |
Well, does he know why everybody uses it?
link |
Like that would be an interesting thing.
link |
Was it the right thing at the right time?
link |
Cause like when stuff like JavaScript came out,
link |
like there was a move from, you know,
link |
writing C programs and C++ to,
link |
let's call what they call managed code frameworks
link |
where you write simple code, it might be interpreted,
link |
it has lots of libraries, productivity is high,
link |
and you don't have to be an expert.
link |
So, you know, Java was supposed to solve
link |
all the world's problems, it was complicated.
link |
JavaScript came out, you know,
link |
after a bunch of other scripting languages.
link |
I'm not an expert on it, but was it the right thing
link |
at the right time or was there something, you know,
link |
clever cause he wasn't the only one.
link |
There's a few elements.
link |
And maybe if he figured out what it was,
link |
then he'd get a prize.
link |
Yeah, you know, maybe his problem is he hasn't defined this.
link |
Or he just needs a good promoter.
link |
Well, I think there was a bunch of blog posts
link |
written about it, which is like wrong is right,
link |
which is like doing the crappy thing fast,
link |
just like hacking together the thing
link |
that answers some of the needs
link |
and then iterating over time, listening to developers,
link |
like listening to people who actually use the thing.
link |
This is something you can do more in software,
link |
but the right time, like you have to sense,
link |
you have to have a good instinct
link |
of when is the right time for the right tool
link |
and make it super simple and just get it out there.
link |
The problem is this is true with hardware,
link |
this is less true with software,
link |
is there's a backward compatibility
link |
that just drags behind you as, you know,
link |
as you try to fix all the mistakes of the past.
link |
But the timing was good.
link |
There's something about that.
link |
It wasn't accidental.
link |
You have to like give yourself over to the,
link |
you have to have this like broad sense
link |
of what's needed now, both scientifically
link |
and like the community and just like this.
link |
It was obvious that there was no,
link |
the interesting thing about JavaScript
link |
is everything that ran in the browser at the time,
link |
like Java and I think other like scheme,
link |
other programming languages,
link |
they were all in a separate external container.
link |
And then JavaScript was literally
link |
just injected into the webpage.
link |
It was the dumbest possible thing
link |
running in the same thread as everything else.
link |
And like it was inserted as a comment.
link |
So JavaScript code is inserted as a comment in the HTML code.
link |
And it was, I mean, there's, it's either genius
link |
or super dumb, but it's like.
link |
Right, so it had no apparatus for like a virtual machine
link |
It just executed in the framework of the program
link |
that's already running.
link |
And it was, and then because something
link |
about that accessibility, the ease of its use
link |
resulted in then developers innovating
link |
of how to actually use it.
link |
I mean, I don't even know what to make of that,
link |
but it does seem to echo across different software,
link |
like stories of different software.
link |
PHP has the same story, really crappy language.
link |
They just took over the world.
link |
Well, let's have a joke that the random length instructions,
link |
variable length instructions, that's always one,
link |
even though they're obviously worse.
link |
Like nobody knows why x86 is,
link |
or you'd be the worst architecture, you know,
link |
on the planet is one of the most popular ones.
link |
Well, I mean, isn't that also the story of risk
link |
versus, I mean, is that simplicity?
link |
There's something about simplicity
link |
that us in this evolutionary process is valued.
link |
If it's simple, it spreads faster, it seems like.
link |
Or is that not always true?
link |
That's not always true.
link |
Yeah, it could be simple as good, but too simple as bad.
link |
So why did risk win, you think, so far?
link |
In the long archivist tree.
link |
So who's gonna win?
link |
What's risk, what's risk,
link |
and who's gonna win in that space in these instruction sets?
link |
AI software's gonna win,
link |
but there'll be little computers that run little programs
link |
like normal all over the place.
link |
But we're going through another transformation, so.
link |
But you think instruction sets underneath it all will change?
link |
Yeah, they evolve slowly.
link |
They don't matter very much.
link |
They don't matter very much, okay.
link |
I mean, the limits of performance are, you know,
link |
predictability of instructions and data.
link |
I mean, that's the big thing.
link |
And then the usability of it is some, you know,
link |
quality of design, quality of tools, availability.
link |
Like right now, X86 is proprietary with Intel and AMD,
link |
but they can change it any way they want independently.
link |
Right, ARM is proprietary to ARM,
link |
and they won't let anybody else change it.
link |
So it's like a sole point.
link |
And RISC 5 is open source, so anybody can change it,
link |
which is super cool.
link |
But that also might mean it gets changed
link |
in too many random ways,
link |
that there's no common subset of it that people can use.
link |
Do you like open or do you like closed?
link |
Like if you were to bet all your money
link |
on one or the other, RISC 5 versus it?
link |
It's case dependent?
link |
Well, X86 oddly enough,
link |
when Intel first started developing it,
link |
they licensed it like seven people.
link |
So it was the open architecture.
link |
And then they moved faster than others
link |
and also bought one or two of them.
link |
But there was seven different people making X86,
link |
because at the time there was 6502 and Z80s and 8086.
link |
And you could argue everybody thought
link |
Z80 was the better instruction set,
link |
but that was proprietary to one place.
link |
So there's like four or five different microprocessors.
link |
Intel went open, got the market share
link |
because people felt like they had multiple sources from it,
link |
and then over time it narrowed down to two players.
link |
So why, you as a historian, why did Intel win
link |
for so long with their processors?
link |
Their process development was great.
link |
So it's just looking back to JavaScript and Brandenike
link |
is Microsoft and Netscape and all these internet browsers.
link |
Microsoft won the browser game
link |
because they aggressively stole other people's ideas.
link |
Like right after they did it.
link |
You know, I don't know
link |
if Intel was stealing other people's ideas.
link |
They started making.
link |
In a good way, stealing them,
link |
because we're just a clarify.
link |
They started making RAMs, random access memories.
link |
And then at the time when the Japanese manufacturers came up,
link |
you know, they were getting out competed on that
link |
and they pivoted the microprocessors
link |
and they made the first, you know,
link |
integrated microprocessor grant programs.
link |
It was the 4004 or something.
link |
Who was behind that pivot?
link |
That's a hell of a pivot.
link |
That's a hell of a pivot.
link |
And then they led semiconductor industry.
link |
Like they were just a little company, IBM,
link |
all kinds of big companies had boatloads of money
link |
and they out innovated everybody.
link |
Out of the innovative.
link |
So it's not like marketing.
link |
It's not, you know, stuff.
link |
Their processor designs were pretty good.
link |
I think the, you know, Core 2 was probably the first one
link |
I thought was great.
link |
It was a really fast processor
link |
and then Haswell was great.
link |
What makes a great processor?
link |
Oh, if you just look at its performance
link |
versus everybody else, it's, you know,
link |
the size of it, you know, usability of it.
link |
So it's not specific, some kind of element
link |
that makes you beautiful.
link |
It's just like literally just raw performance.
link |
Is that how you think of bioprocessors?
link |
It's just like raw performance.
link |
It's like a horse race.
link |
The fastest one wins.
link |
You don't care how.
link |
Well, there's the fastest in the environment.
link |
Like for years, you made the fastest one you could
link |
and then people started to have power limits.
link |
So then you made the fastest at the right PowerPoint.
link |
And then, and then when we started doing multiprocessors,
link |
like if you could scale your processors more
link |
than the other guy, you could be 10% faster
link |
on like a single thread, but you have more threads.
link |
So there's lots of variability.
link |
And then ARM really explored, like, you know,
link |
they have the A series and the R series
link |
and the M series, like a family of processors
link |
for all these different design points
link |
from like unbelievably small and simple.
link |
And so then when you're doing the design,
link |
it's sort of like this big palette of CPUs.
link |
Like they're the only ones with a credible,
link |
you know, top to bottom palette.
link |
What do you mean a credible top to bottom?
link |
Well, there's people that make microcontrollers
link |
that are small, but they don't have a fast one.
link |
There's people who make fast processors,
link |
but don't have a little, a medium one or a small one.
link |
Is that hard to do that full palette?
link |
That seems like a, it's a lot of different.
link |
So what's the difference in the ARM folks and Intel,
link |
in terms of the way they're approaching this problem?
link |
Well, Intel, almost all their processors or designs
link |
were, you know, very custom high end,
link |
you know, for the last 15, 20 years.
link |
It's the fastest horse possible.
link |
Yeah. And the architecture that are really good,
link |
but the company itself was fairly insular
link |
to what's going on in the industry with CAD tools and stuff.
link |
And there's this debate about custom design versus synthesis
link |
and how do you approach that?
link |
I'd say Intel was slow on the cutting to synthesize processors.
link |
ARM came in from the bottom and they generated IP,
link |
which went to all kinds of customers.
link |
So they had very little say in how the customer
link |
implemented their IP.
link |
So ARM is super friendly to the synthesis IP environment.
link |
Whereas Intel said,
link |
we're going to make this great client chip or server chip
link |
with our own CAD tools, with our own process,
link |
with our own, you know, other supporting IP
link |
and everything only works with our stuff.
link |
So is that, is ARM winning the mobile platform space
link |
in terms of process?
link |
And so in that way you're describing is why they're winning.
link |
Well, they had lots of people doing lots
link |
of different experiments.
link |
So they controlled the processor architecture and IP,
link |
but they let people put in lots of different chips.
link |
And there was a lot of variability in what happened there.
link |
Whereas Intel, when they made their mobile,
link |
their foray into mobile,
link |
they had one team doing one part, right?
link |
So it wasn't 10 experiments.
link |
And then their mindset was PC mindset,
link |
Microsoft software mindset,
link |
and that brought a whole bunch of things along
link |
that the mobile world, the embedded world don't do.
link |
You think it was possible for Intel to pivot hard
link |
and win the mobile market?
link |
That's a hell of a difficult thing to do, right?
link |
For a huge company to just pivot.
link |
I mean, it's so interesting to,
link |
because we'll talk about your current work.
link |
It's like, it's clear that PCs were dominating
link |
for several decades, like desktop computers.
link |
And then mobile, it's unclear.
link |
It's a leadership question.
link |
Like Apple under Steve Jobs, when he came back,
link |
they pivoted multiple times.
link |
They built iPads and iTunes and phones and tablets
link |
and great Macs, like who knew computers
link |
should be made out of aluminum?
link |
That they're great, it's super fun.
link |
Yeah, Steve Jobs, like they pivoted multiple times.
link |
And the old Intel, they did that multiple times.
link |
They made DRAMs and processors and processes
link |
and I got to ask this,
link |
what was it like working with Steve Jobs?
link |
I didn't work with him.
link |
Did you interact with him?
link |
I said hi to him twice in the cafeteria.
link |
You said, hey fellas, he was friendly.
link |
He was wandering around and with somebody,
link |
you couldn't find the table
link |
because the cafeteria was packed and I gave my table.
link |
But I worked for Mike Colbert who talked to,
link |
like Mike was the unofficial CTO of Apple
link |
and a brilliant guy and he worked for Steve for 25 years,
link |
maybe more and he talked to Steve multiple times a day.
link |
And he was one of the people that could put up with Steve's,
link |
let's say brilliance and intensity.
link |
And Steve really liked him and Steve trusted Mike
link |
to translate the shit he thought up
link |
into engineering products at work
link |
and then Mike ran a group called Platform Architecture
link |
and I was in that group.
link |
So many times I'd be sitting with Mike
link |
and the phone would ring and it'd be Steve
link |
and Mike would hold the phone like this
link |
because Steve would be yelling about something or other.
link |
And then he would translate.
link |
And he translated and then he would say,
link |
Steve wants us to do this.
link |
Was Steve a good engineer or no?
link |
He was a great idea guy.
link |
And he's a really good selector for talent.
link |
Yeah, that seems to be one of the key elements
link |
of leadership, right?
link |
And then he was a really good first principles guy.
link |
Like somebody say something couldn't be done
link |
and he would just think that's obviously wrong, right?
link |
But maybe it's hard to do, maybe it's expensive to do,
link |
maybe we need different people.
link |
There's like a whole bunch of,
link |
if you want to do something hard,
link |
maybe it takes time, maybe you have to iterate.
link |
There's a whole bunch of things that you could think about
link |
but saying it can't be done is stupid.
link |
How would you compare?
link |
So it seems like Elon Musk is more engineering centric
link |
I think he considered himself a designer too.
link |
He has a design mind.
link |
Steve Jobs feels like he's much more idea space,
link |
design space versus engineering.
link |
Just make it happen.
link |
Like the world should be this way, just figure it out.
link |
But he used computers.
link |
You know, he had computer people talk to him all the time.
link |
Like Mike was a really good computer guy.
link |
He knew what computers could do.
link |
Computer meaning computer hardware,
link |
like hardware software, all the pieces and then he would,
link |
you know, have an idea about what could we do with this next
link |
that was grounded in reality.
link |
It wasn't like he was, you know,
link |
just finger painting on the wall
link |
and wishing somebody would interpret it.
link |
Like, so he had this interesting connection
link |
because, you know, he wasn't a computer architecture designer,
link |
but he had an intuition from the computers we had
link |
to what could happen and.
link |
It's interesting to say intuition because it seems
link |
like he was pissing off a lot of engineers in his intuition
link |
about what can and can't be done.
link |
Those, like the, what is all these stories
link |
about like floppy disks and all that kind of stuff.
link |
So in Steve, the first round, like he'd go into a lab
link |
and look at what's going on and hate it
link |
and fire people or assembly in the elevator,
link |
what they're doing for Apple and, you know, not be happy.
link |
When he came back, my impression was is he surrounded himself
link |
with this relatively small group of people.
link |
And didn't really interact outside of that as much.
link |
And then the joke was, you'd see like somebody moving
link |
up prototype through the quad with a black blanket over it.
link |
And that was because it was secret, you know,
link |
partly from Steve because they didn't want Steve
link |
to see it until it was ready.
link |
Yeah, the dynamic with Johnny Ive and Steve is interesting.
link |
It's like you don't want to, he ruins as many ideas
link |
It's a dangerous kind of line to walk.
link |
If you have a lot of ideas, like,
link |
like Gordon Bell was famous for ideas, right?
link |
And it wasn't that the percentage of good ideas
link |
was way higher than anybody else.
link |
It was, he had so many ideas and he was also good
link |
at talking to people about it and getting the filters, right?
link |
And, you know, seeing through stuff.
link |
Whereas Elon was like, hey, I want to build rockets.
link |
So Steve was hired bunch of rocket guys
link |
and Elon would go read rocket manuals.
link |
So Elon is a better engineer, a sense like,
link |
or like more like a love and passion for the manuals.
link |
The data and the understanding.
link |
The craftsmanship too, right?
link |
Well, I guess Steve had craftsmanship too,
link |
but of a different kind.
link |
What do you make of the, just the standard
link |
for just a little longer, what do you make of like
link |
the anger and the passion and all that,
link |
the firing and the mood swings and the madness,
link |
the, you know, being emotional and all that.
link |
That's Steve and I guess Elon too.
link |
So what, is that a bug or a feature?
link |
So there's a graph, which is Y axis productivity.
link |
X axis at zero is chaos and infinity is complete order.
link |
So as you go from the, you know, the origin,
link |
as you improve order, you improve productivity.
link |
And at some point productivity peaks
link |
and then it goes back down again.
link |
Too much order, nothing can happen.
link |
But the question is, is the,
link |
how close to the chaos is that?
link |
Is once you start moving the direction of order,
link |
the force factor to drive you towards order is unstoppable.
link |
And every organization will move to the place
link |
where their productivity is stymied by order.
link |
So the question is, who's the counter force?
link |
Like, and cause it also feels really good.
link |
As you get more organized and productivity goes up,
link |
the organization feels it, they orient towards it, right?
link |
To hire more people.
link |
They got more guys who couldn't run process,
link |
you get bigger, right?
link |
And then inevitably, the organization gets captured
link |
by the bureaucracy that manages all the processes.
link |
All right. And then humans really like that.
link |
And so if you just walk into a room and say,
link |
guys, love what you're doing,
link |
but I need you to have less order.
link |
If you don't have some force behind that,
link |
nothing will happen.
link |
I can't tell you on how many levels that's profound.
link |
So that's why I'd say it's a feature.
link |
Now, could you be nicer about it?
link |
I don't know any good examples of being nicer about it.
link |
Well, the funny thing is to get stuff done.
link |
You need people who can manage stuff
link |
and manage people because humans are complicated.
link |
They need lots of care and feeding
link |
and you need to tell them they look nice
link |
and they're doing good stuff and pat them on the back, right?
link |
Do you tell me, is that needed?
link |
Do humans need that?
link |
I had a friend, he started a manager group and he said,
link |
You have to praise them before they do anything.
link |
I was waiting till they were done
link |
and they were always mad at me.
link |
Now we tell them what a great job they're doing
link |
while they're doing it.
link |
But then you get stuck in that trap
link |
because then when they're not doing something,
link |
how do you confront these people?
link |
I think a lot of people that had trauma
link |
in their childhood would disagree with you.
link |
Successful people that you just first do the rough stuff
link |
and then be nice later.
link |
Okay, but you know, engineering companies are full
link |
of adults who had all kinds of range of childhoods.
link |
You know, most people had okay childhoods.
link |
Well, I don't know if...
link |
I know lots of people only work for praise, which is weird.
link |
You mean like everybody.
link |
I'm not that interested in it, but...
link |
Well, you're probably looking
link |
for somebody's approval, even still.
link |
I should think about that.
link |
Maybe somebody who's no longer with us kind of thing.
link |
I used to call up my dad and tell him what I was doing.
link |
He was very excited about engineering and stuff.
link |
You got his approval?
link |
Like he decided I was smart and unusual as a kid
link |
and that was okay when I was really young.
link |
So when I did poorly in school, I was dyslexic.
link |
I didn't read until I was third or fourth grade.
link |
My parents were like, oh, he'll be fine.
link |
Is he still with us?
link |
Sure, yeah, he had Parkinson's and then cancer.
link |
His last 10 years were tough.
link |
Killing a man like that's hard.
link |
Well, it was pretty good.
link |
Parkinson's causes slow dementia
link |
and the chemotherapy, I think, accelerated it.
link |
But it was like hallucinogenic dementia.
link |
So he was clever and funny and interesting
link |
and it was pretty unusual.
link |
Do you remember conversations from that time?
link |
Like what, do you have fond memories of the guy?
link |
Anything come to mind?
link |
A friend told me one time I could draw a computer
link |
on the way forward faster than anybody had ever met.
link |
And I said, you should meet my dad.
link |
Like when I was a kid, he'd come home and say,
link |
I was driving by this bridge and I was thinking about it.
link |
And he pulled out a piece of paper
link |
and he'd draw the whole bridge.
link |
He was a mechanical engineer.
link |
And he would just draw the whole thing
link |
and then he would tell me about it
link |
and then tell me how he would have changed it.
link |
And he had this idea that he could understand
link |
and conceive anything.
link |
And I just grew up with that, so that was natural.
link |
So if, you know, like when I interview people,
link |
I ask them to draw a picture of something
link |
they did on the whiteboard.
link |
And it's really interesting.
link |
Like some people will draw a little box, you know,
link |
and then they'll say, and then this talks to this
link |
and I'll be like, that's just frustrating.
link |
And then I had this other guy come in one time.
link |
He says, well, I designed a floating point in this chip
link |
but I'd really like to tell you how the whole thing works
link |
and then tell you how the floating point works inside of it.
link |
Do you mind if I do that?
link |
He covered two whiteboards in like 30 minutes.
link |
Like, he was great.
link |
There's craftsmen.
link |
I mean, that's the craftsmanship to that.
link |
Yeah, but also the mental agility
link |
to understand the whole thing.
link |
Put the pieces in context, you know,
link |
real view of the balance of how the design worked.
link |
Because if you don't understand it properly,
link |
when you start to draw it,
link |
you'll fill up half the whiteboard
link |
with like a little piece of it and, you know,
link |
like your ability to lay it out in an understandable way
link |
takes a lot of understanding.
link |
So. And be able to zoom in to the detail
link |
and then zoom out to the picture.
link |
Zoom out really fast.
link |
What about the impossible thing?
link |
You see, your dad believed that you can do anything.
link |
That's a weird feature for a craftsman.
link |
It seems that that echoes in your own behavior.
link |
Like, that's the...
link |
Well, it's not that anybody can do anything right now.
link |
It's that if you work at it, you can get better at it.
link |
And there might not be a limit.
link |
And they did funny things like,
link |
like he always wanted to play piano.
link |
So at the end of his life, he started playing the piano.
link |
When he had Parkinson's, I mean, he was terrible.
link |
But he thought if he really worked out it in this life,
link |
maybe the next life, he'd be better at it.
link |
He might be onto something.
link |
He enjoyed doing it.
link |
So that's pretty funny.
link |
Do you think the perfect is the enemy of the good
link |
in hardware and software engineering?
link |
It's like we were talking about JavaScript a little bit
link |
and the messiness of the 10 day building process.
link |
It's, you know, creative tension, right?
link |
So creative tension is you have two different ideas
link |
that you can't do both.
link |
And the, but the fact that you want to do both causes you
link |
to go try to solve that problem.
link |
That's the creative part.
link |
So if you're building computers, like some people say,
link |
we have the schedule and anything that doesn't fit
link |
in the schedule, we can't do, right?
link |
And so they throw out the perfect cause I have a schedule.
link |
Then there's other people to say,
link |
we need to get this perfectly right.
link |
And no matter what, you know, more people, more money, right?
link |
And there's a really clear idea about what you want.
link |
And some people are really good at articulating it, right?
link |
So let's call that the perfect.
link |
But that's also terrible cause they never ship anything.
link |
They never hit any goals.
link |
So now you have the, now you have your framework.
link |
You can't throw out stuff cause you can't get it done today.
link |
Cause maybe you get it done tomorrow with the next project.
link |
You can't, so you have to, I work with a guy that I really
link |
like working with, but he over filters his ideas.
link |
He'd start thinking about something.
link |
And as soon as he figured out what's wrong with it,
link |
he'd throw it out.
link |
And then I start thinking about it.
link |
And I, you know, you come up with an idea and then you find
link |
out what's wrong with it.
link |
And then you give it a little time to set.
link |
Cause sometimes, you know, you figure out how to tweak it
link |
or maybe that idea helps some other idea.
link |
So idea generation is really funny.
link |
So you have to give your idea space.
link |
Like spaciousness of mind is key,
link |
but you also have to execute programs and get shit done.
link |
And then it turns out computer engineering is fun
link |
because it takes, you know, a hundred people to build a
link |
computer, 200 to 300, whatever the number is.
link |
And people are so variable about, you know, temperament and,
link |
you know, skill sets and stuff that in a big organization,
link |
you find that the people who love the perfect ideas and the
link |
people that want to get stuffed on yesterday and people like
link |
that come up with ideas and people like the, let's say,
link |
And it takes the whole, it takes a large group of people.
link |
So some are good at generating ideas.
link |
Some are good at filtering ideas.
link |
And in that giant mess, you're somehow, I guess the goal is
link |
for that giant mess of people to find the perfect path
link |
through the tension, the creative tension.
link |
But like, how do you know when you said there's some people
link |
good at articulating what perfect looks like,
link |
what a good design is?
link |
Like if you're sitting in a room and you have a set of ideas
link |
about like how to design a better processor,
link |
how do you know this is something special here?
link |
This is a good idea.
link |
So if you ever brainstormed idea with a couple of people
link |
that were really smart and you kind of go into it
link |
and you don't quite understand it and you're working on it.
link |
And then you start, you know, talking about it,
link |
putting it on the whiteboard, maybe it takes days or weeks.
link |
And then your brain starts to kind of synchronize.
link |
It's really weird.
link |
Like you start to see what each other is thinking and it starts to work.
link |
Like you can see work, like my talent in computer design
link |
is I can see how computers work in my head like really well.
link |
And I know other people can do that too.
link |
And when you're working with people that can do that,
link |
like it is kind of an amazing experience.
link |
And then every once in a while you get to that place
link |
and then you find the flaw and it was just kind of funny
link |
because you can fool yourself.
link |
The two of you kind of drifted along in the direction that was useless.
link |
Yeah, that happens too.
link |
Like you have to, because, you know, the nice thing about computer design
link |
is always reduction of practice.
link |
Like you come up with your good ideas.
link |
And I've noticed some architects who really love ideas
link |
and then they work on them and they put it on the shelf
link |
and they go work on the next idea and put it on the shelf
link |
and they never reduce it to practice.
link |
So they find out what's good and bad
link |
because almost every time I've done something really new,
link |
by the time it's done, like the good parts are good,
link |
but I know all the flaws, like...
link |
Yeah, would you say your career, just your own experience?
link |
Is your career defined mostly by flaws or by successes?
link |
Again, there's great attention between those.
link |
If you haven't tried hard, right, and done something new,
link |
right, then you're not gonna be facing the challenges
link |
when you build it, then you find out all the problems with it.
link |
But when you look back, do you see problems or...
link |
Oh, when I look back, I think earlier in my career,
link |
like EV5 was the second Alpha chip,
link |
I was so embarrassed about the mistakes,
link |
I could barely talk about it.
link |
And it was in the Guinness Book of Rolls records
link |
and it was the fastest processor on the planet.
link |
So it was, and at some point I realized
link |
that was really a bad mental framework to deal with,
link |
like doing something new, we did a bunch of new things
link |
and some of them worked out great and some were bad.
link |
And we learned a lot from it and then the next one,
link |
That also, EV6 also had some really cool things in it.
link |
I think the proportion of good stuff went up,
link |
but it had a couple of fatal flaws in it
link |
that were painful.
link |
You learned to channel the pain into like pride.
link |
Not pride really, just realization
link |
about how the world works or how that kind of idea set works.
link |
Life is suffering, that's the reality.
link |
I know the Buddhists have that and a couple of other people
link |
No, it's, you know, there's just kind of weird combination
link |
of good and bad and light and darkness
link |
that you have to deal with.
link |
Yeah, there's definitely lots of suffering in the world.
link |
Depends on the perspective.
link |
It seems like there's way more darkness,
link |
but that makes the light part really nice.
link |
Computing hardware or just any kind of even software design,
link |
are you defined beautiful from your own work,
link |
from other people's work?
link |
That you're just...
link |
We were just talking about the kind of software
link |
that you're just... We were just talking about the battleground
link |
of flaws and mistakes and errors,
link |
but things that were just beautifully done.
link |
Is there something that pops to mind?
link |
Well, when things are beautifully done,
link |
usually there's a well thought out set of abstraction layers.
link |
So the whole thing works in unison nicely.
link |
And when I say abstraction layer,
link |
that means two different components
link |
when they work together, they work independently.
link |
They don't have to know what the other one is doing.
link |
So that decoupling.
link |
Yeah, so the famous one was the network stack.
link |
Like there's a seven layer network stack,
link |
you know, data transport and protocol and all the layers.
link |
And the innovation was is when they really got that right.
link |
Because networks before that didn't define those very well.
link |
The layers could innovate independently
link |
and occasionally the layer boundary would...
link |
The interface would be upgraded.
link |
And that let, you know, the design space breathe.
link |
You could do something new in layer seven
link |
without having to worry about how layer four worked.
link |
And so good design does that.
link |
And you see it in processor designs.
link |
When we did the Zen design at AMD,
link |
we made several components very modular.
link |
And, you know, my insistence at the top was
link |
I wanted all the interfaces defined
link |
before we wrote the RTL for the pieces.
link |
One of the verification leads said,
link |
if we do this right, I can test the pieces
link |
so well independently.
link |
When we put it together,
link |
we won't find all these interaction bugs
link |
because the floating point knows how the cache works.
link |
And I was a little skeptical, but he was mostly right.
link |
That the modularity design greatly improved the quality.
link |
Is that universally true in general?
link |
Would you say about good designs,
link |
the modularity is like usually modularity?
link |
Well, we talked about this before.
link |
Humans are only so smart.
link |
And we're not getting any smarter, right?
link |
But the complexity of things is going up.
link |
So, you know, a beautiful design
link |
can't be bigger than the person doing it.
link |
It's just, you know, their piece of it.
link |
Like the odds of you doing a really beautiful design
link |
of something that's way too hard for you is low, right?
link |
If it's way too simple for you,
link |
it's not that interesting.
link |
It's like, well, anybody could do that.
link |
But when you get the right match of your expertise
link |
and, you know, mental power to the right design size,
link |
that's cool, but that's not big enough
link |
to make a meaningful impact in the world.
link |
So now you have to have some framework
link |
to design the pieces so that the whole thing
link |
is big and harmonious.
link |
But, you know, when you put it together,
link |
it's, you know, sufficiently interesting to be used.
link |
And, you know, so that's like a beautiful design is.
link |
Matching the limits of that human cognitive capacity
link |
to the module you can create
link |
and creating a nice interface between those modules.
link |
And thereby, do you think there's a limit
link |
to the kind of beautiful complex systems
link |
we can build with this kind of modular design?
link |
It's like, you know, if we build increasingly more complicated,
link |
you can think of like the internet, okay, let's scale it down.
link |
No, you can think of like social network,
link |
like Twitter as one computing system.
link |
And, but those are the little modules, right?
link |
Well, it's built on, it's built on so many components
link |
nobody at Twitter even understands.
link |
So, so, so if an alien showed up and looked at Twitter,
link |
he wouldn't just see Twitter as a beautiful,
link |
simple thing that everybody uses, which is really big.
link |
You would see the network, it runs on the fiber optics,
link |
the data is transported to computers.
link |
The whole thing is so bloody complicated,
link |
nobody at Twitter understands it.
link |
And so that's what the alien would see.
link |
So yeah, if an alien showed up and looked at Twitter
link |
or looked at the various different network systems
link |
that you can see on earth.
link |
So imagine they were really smart
link |
that could comprehend the whole thing.
link |
And then they sort of, you know, evaluated the human
link |
and thought, this is really interesting.
link |
No human on this planet comprehends the system they built.
link |
No individual, well, would they even see individual humans?
link |
That's interesting, like we humans are very human centric,
link |
And so we think of us as the central organism
link |
and the networks as just the connection of organisms,
link |
but from a perspective of an alien,
link |
from an outside perspective, it seems like.
link |
Yeah, I get it, we're the answer to the ant colony.
link |
The ant colony, yeah.
link |
Or the result of production of the ant colony,
link |
which is like cities and it's,
link |
it's a, in that sense, humans are pretty impressive.
link |
The modularity that we're able to and the,
link |
and how robust we are to noise and mutation,
link |
all that kind of stuff.
link |
Well, that's cause it's stress tested all the time.
link |
You know, you build all these cities with buildings
link |
and you get earthquakes occasionally and.
link |
You know, wars, earthquakes.
link |
Viruses every once in a while.
link |
Changes in business plans for, you know,
link |
like shipping or something.
link |
Like, as long as there's all stress tested,
link |
then it keeps adapting to the situation.
link |
So that's a curious phenomena.
link |
Well, let's go, let's talk about Moore's Law a little bit.
link |
It's a, at the broad view of Moore's Law,
link |
where it's just exponential improvement of computing
link |
capability, like OpenAI, for example,
link |
recently published this kind of papers looking
link |
at the exponential improvement in the training efficiency
link |
of neural networks.
link |
For like ImageNet and all that kind of stuff,
link |
we just got better on this,
link |
this is purely software side,
link |
just figuring out better tricks and algorithms
link |
for training neural networks.
link |
And that seems to be improving significantly faster
link |
than the Moore's Law prediction, you know?
link |
So that's in the software space.
link |
Like, what do you think if Moore's Law continues
link |
or if the general version of Moore's Law continues,
link |
do you think that comes mostly from the hardware,
link |
from the software, some mix of the two,
link |
some interesting totally,
link |
so not the reduction of the size of the transistor
link |
kind of thing, but more in the totally interesting
link |
kinds of innovations in the hardware space,
link |
all that kind of stuff.
link |
Well, there's like half a dozen things going on
link |
So one is there's initial innovations
link |
that had a lot of headroom to be exploited.
link |
So, you know, the efficiency of the networks
link |
has improved dramatically.
link |
And then the decomposability of those and the use,
link |
you know, they started running on one computer,
link |
then multiple computers and then multiple GPUs
link |
and then arrays of GPUs and they're up to thousands.
link |
And at some point, so it's sort of like,
link |
they were consumed, they were going from
link |
like a single computer application
link |
to a thousand computer application.
link |
So that's not really a Moore's Law thing.
link |
That's an independent vector.
link |
How many computers can I put on this problem?
link |
Because the computers themselves are getting better
link |
on like a Moore's Law rate,
link |
but their ability to go from one to 10 to 100 to a thousand
link |
you know, was something.
link |
And then multiplied by, you know,
link |
the amount of computers it took to resolve
link |
like AlexNet, to ResNet, to transformers.
link |
It's been quite, you know, steady improvements.
link |
But those are like S cores, aren't they?
link |
That's the exactly kind of S cores
link |
that are underlying Moore's Law from the very beginning.
link |
So what's the biggest, what's the most productive,
link |
rich source of S curves in the future, do you think?
link |
Is it hardware or is it software?
link |
So hardware is going to move along relatively slowly.
link |
Like, you know, double performance every two years.
link |
There's still, I like how you call that slow.
link |
You know, it's the slow version.
link |
The snail's pace of Moore's Law.
link |
Maybe we should, we should trade mark that one.
link |
Whereas the scaling by number of computers,
link |
you know, can go much faster.
link |
You know, I'm sure at some point, Google had a,
link |
you know, their initial search engine
link |
was running on a laptop, you know, like,
link |
and at some point they really worked on scaling that.
link |
And then they factored the indexer from, you know,
link |
this piece and this piece and this piece
link |
and they spread the data on more and more things.
link |
And, you know, they did a dozen innovations.
link |
But as they scaled up the number of computers on that,
link |
it kept breaking, finding new bottlenecks in their software
link |
and their schedulers and made them rethink.
link |
Like, it seems insane to do a scheduler
link |
across a thousand computers who schedule parts of it
link |
and then send the results to one computer.
link |
But if you want to schedule a million searches,
link |
that makes perfect sense.
link |
So there's, the scaling by just quantity
link |
is probably the richest thing.
link |
But then as you scale quantity, like a network
link |
that was great on a hundred computers,
link |
maybe completely the wrong one,
link |
you may pick a network that's 10 times slower
link |
on 10,000 computers, like per computer.
link |
But if you go from a hundred to 10,000, that's a hundred times.
link |
So that's one of the things that happened
link |
when we did internet scaling.
link |
This efficiency went down, not up.
link |
The future of computing is inefficiency, not efficiency.
link |
But scales, inefficient scale.
link |
It's scaling faster than inefficiency by two.
link |
And as long as there's, you know, dollar value there,
link |
like scaling costs lots of money.
link |
But Google showed, Facebook showed, everybody showed
link |
that the scale was where the money was at.
link |
It was worth the financial.
link |
Do you think, is it possible
link |
that like basically the entirety of Earth
link |
will be like a computing surface?
link |
Like this table will be doing computing.
link |
This hedgehog will be doing computing.
link |
Like everything really inefficient
link |
done computing will be lover.
link |
Science fiction books, they call it computronium.
link |
We turn everything into computing.
link |
Well, most of the elements aren't very good for anything.
link |
Like you're not gonna make a computer out of iron.
link |
Like, you know, silicon and carbon
link |
have like nice structures.
link |
You know, we'll see what you can do with the rest of it.
link |
People talk about, well, maybe you can turn the sun
link |
into a computer, but it's hydrogen.
link |
And a little bit of helium, so.
link |
What I mean is more like actually
link |
just adding computers to everything.
link |
I thought you were just converting all the mass
link |
of the universe into computer.
link |
To be ironic from the simulation point of view
link |
is like the simulator build mass, the simulate.
link |
Yeah, I mean, yeah.
link |
So, I mean, ultimately this is all heading
link |
towards the simulation.
link |
Yeah, well, I think I might have told you this story.
link |
At Tesla, they were deciding,
link |
so they wanna measure the current coming out of the battery
link |
and they decided between putting a resistor in there
link |
and putting a computer with a sensor in there.
link |
And the computer was faster than the computer
link |
I worked on in 1982.
link |
And we chose the computer
link |
because it was cheaper than the resistor.
link |
So, sure, this hedgehog, you know, costs $13
link |
and we can put an AI that's as smart as you
link |
in there for five bucks.
link |
You know, so computers will be everywhere.
link |
I was hoping it wouldn't be smarter than me
link |
Well, everything's gonna be smarter than you.
link |
But you were saying it's inefficient.
link |
I thought it was better to have a lot of dumb things.
link |
Well, Moore's Law will slowly compact that stuff.
link |
So even the dumb things will be smarter than us.
link |
The dumb things are gonna be smart
link |
or they're gonna be smart enough to talk to something
link |
that's really smart.
link |
You know, it's like, well, just remember,
link |
like a big computer chip, you know,
link |
it's like an inch by an inch and, you know,
link |
40 microns thick, it doesn't take very much,
link |
very many atoms to make a high power computer.
link |
And 10,000 of them can fit in the shoebox.
link |
But, you know, you have the cooling and power problems,
link |
but, you know, people are working on that.
link |
But they still can't write compelling poetry or music
link |
or understand what love is
link |
or have a fear of mortality.
link |
So we're still winning.
link |
Neither can most of humanity, so...
link |
Well, they can write books about it.
link |
So, but speaking about this walk along the path
link |
of innovation towards the dumb things
link |
being smarter than humans,
link |
you are now the CTO of Ten Storent as of two months ago.
link |
They built hardware for deep learning.
link |
How do you build scalable and efficient deep learning?
link |
This is such a fascinating space.
link |
Yeah, yeah, so it's interesting.
link |
So up until recently,
link |
I thought there was two kinds of computers.
link |
There are serial computers that run like C programs,
link |
and then there's parallel computers.
link |
So the way I think about it is, you know,
link |
parallel computers have given parallelism.
link |
Like GPUs are great cause you have a million pixels.
link |
And modern GPUs run a program on every pixel.
link |
They call it the shader program, right?
link |
So, or like finite element analysis.
link |
You built something, you know,
link |
you make this into little tiny chunks.
link |
You give each chunk to a computer.
link |
So you're given all these chunks of parallelism like that.
link |
But most C programs, you write this linear narrative
link |
and you have to make it go fast.
link |
To make it go fast, you predict all the branches,
link |
all the data fetches and you run that more in parallel,
link |
but that's found parallelism.
link |
AI is, I'm still trying to decide how fundamental this is.
link |
It's a given parallelism problem.
link |
But the way people describe the neural networks
link |
and then how they write them in PyTorch, it makes graphs.
link |
That might be fundamentally different than the GPU kind of.
link |
Parallelism, yeah, it might be.
link |
Because when you run the GPU program on all the pixels,
link |
you're running, you know, depends, you know,
link |
this group of pixels say it's background blue
link |
and it runs a really simple program.
link |
This pixel is, you know, some patch of your face.
link |
So you have some really interesting shader program
link |
to give you the impression of translucency.
link |
But the pixels themselves don't talk to each other.
link |
There's no graph, right?
link |
So you do the image
link |
and then you do the next image and you do the next image
link |
and you run 8 million pixels, 8 million programs every time
link |
and modern GPUs have like 6,000 thread engines in them.
link |
So, you know, to get 8 million pixels,
link |
each one runs a program on, you know, 10 or 20 pixels.
link |
And that's how they work, there's no graph.
link |
But you think graph might be a totally new way
link |
to think about hardware.
link |
So, Roger Gattori and I have been having this good conversation
link |
about given versus found parallelism.
link |
And then the kind of walk,
link |
as we got more transistors, like, you know,
link |
computers way back when did stuff on scalar data.
link |
Then we did on vector data, famous vector machines.
link |
Now we're making computers that operate on matrices, right?
link |
And then the category we said that was next was spatial.
link |
Like imagine you have so much data that, you know,
link |
you want to do the compute on this data.
link |
And then when it's done,
link |
it says send the result to this pile of data
link |
on some software on that.
link |
And it's better to think about it spatially
link |
than to move all the data to a central processor
link |
and do all the work.
link |
So, spatially, I mean, moving in the space of data
link |
as opposed to moving the data.
link |
Yeah, you have a petabyte data space
link |
spread across some huge array of computers.
link |
And when you do a computation somewhere,
link |
you send the result of that computation
link |
or maybe a pointer to the next program
link |
to some other piece of data and do it.
link |
But I think a better word might be graph
link |
and all the AI neural networks are graphs.
link |
Do some computations and the result here,
link |
do another computation, do a data transformation,
link |
do emerging, do a pooling, do another computation.
link |
Is it possible to compress and say
link |
how we make this thing efficient,
link |
this whole process efficient, this different?
link |
So first, the fundamental elements in the graphs
link |
are things like metrics, multiplies, convolutions,
link |
data manipulations, and data movements.
link |
So GPUs emulate those things with their little singles,
link |
basically running a single threaded program.
link |
And then there's an NVIDIA calls it a warp
link |
where they group a bunch of programs
link |
that are similar together for efficiency
link |
and instruction use.
link |
And then at a higher level, you take this graph
link |
and you say this part of the graph is a matrix multiplier
link |
which runs on these 30 G threads.
link |
But the model at the bottom was built
link |
for running programs on pixels, not executing graphs.
link |
So it's emulation, ultimately.
link |
So is it possible to build something
link |
that natively runs graphs?
link |
Yes, so it's what Ten Storent did.
link |
So where are we on that?
link |
How, like in the history of that effort,
link |
are we in the early days?
link |
Ten Storent started by a friend of mine,
link |
Labisha Bajek, and I was his first investor.
link |
So I've been kind of following him
link |
and talking to him about it for years
link |
and in the fall when I was considering things to do.
link |
I decided, you know, we held a conference last year
link |
with a friend to organize it.
link |
And we wanted to bring in thinkers
link |
and two of the people were Andre Carpathi and Chris Latner.
link |
And Andre gave this talk, it's on YouTube,
link |
called software 2.0, which I think is great.
link |
Which is, we went from programmed computers,
link |
where you write programs to data program computers.
link |
You know, like the futures of software
link |
is data programs, the networks.
link |
And I think that's true.
link |
And then Chris has been working,
link |
he worked on LLVM, the low level virtual machine,
link |
which became the intermediate representation
link |
for all compilers.
link |
And now he's working on another project called MLIR,
link |
which is mid level intermediate representation,
link |
which is essentially under the graph
link |
about how do you represent that kind of computation
link |
and then coordinate large numbers
link |
of potentially heterogeneous computers.
link |
And I would say technically 10 storents,
link |
you know, two pillars of those two ideas,
link |
software 2.0 and mid level representation.
link |
But it's in service of executing graph programs.
link |
The hardware is designed to do that.
link |
So that's including the hardware piece.
link |
And then the other cool thing is,
link |
for a relatively small amount of money,
link |
they did a test chip and two production chips.
link |
So it's like a super effective team.
link |
And unlike some AI startups,
link |
where if you don't build the hardware
link |
to run the software that they really want to do,
link |
then you have to fix it by writing lots more software.
link |
So the hardware naturally does,
link |
matrix multiply, convolution, the data manipulations,
link |
and the data movement between processing elements
link |
that you can see in the graph,
link |
which I think is all pretty clever.
link |
And that's what I'm working on now.
link |
So I think it's called the grace call processor
link |
introduced last year.
link |
It's, you know, there's a bunch of measures
link |
of performance we're talking about, horses.
link |
It seems to outperform 368 trillion operations per second.
link |
It seems to outperform NVIDIA's Tesla T4 system.
link |
So these are just numbers.
link |
What do they actually mean in real world performance?
link |
Like what are the metrics for you that you're chasing
link |
in your horse racing?
link |
What do you care about?
link |
Well, first, so the native language of,
link |
you know, people who write AI network programs
link |
is PyTorch now, PyTorch TensorFlow.
link |
There's a couple others.
link |
The PyTorch is one over TensorFlow,
link |
this is just, I'm not an expert on that.
link |
I know many people have switched from TensorFlow
link |
And there's technical reasons for it.
link |
I use both, both are still awesome.
link |
Both are still awesome.
link |
But the deepest love is for PyTorch currently.
link |
Yeah, there's more love for that.
link |
And that may change.
link |
So the first thing is when they write their programs
link |
can the hardware execute it pretty much as it was written.
link |
So PyTorch turns into a graph.
link |
We have a graph compiler that makes that graph.
link |
Then it fractions the graph down.
link |
So if you have big matrix multiply,
link |
we turn it into right size chunks
link |
to run on the processing elements.
link |
It hooks all the graph up, it lays out all the data.
link |
There's a couple of mid level representations of it
link |
that are also simulatable.
link |
So that if you're writing the code,
link |
you can see how it's gonna go through the machine,
link |
which is pretty cool.
link |
And then at the bottom it's scheduled kernels
link |
like math, data manipulation, data movement kernels,
link |
which do this stuff.
link |
So we don't have to run, write a little program
link |
to do matrix multiply.
link |
Because we have a big matrix multiplier.
link |
Like there's no SIMT program for that.
link |
But there is scheduling for that, right?
link |
So one of the goals is if you write a piece of PyTorch code
link |
that looks pretty reasonable,
link |
you should be able to compile it, run it on the hardware
link |
without having to tweak it
link |
and do all kinds of crazy things to get performance.
link |
There's not a lot of intermediate steps.
link |
It's running directly as written.
link |
Like on a GPU, if you write a large matrix multiply
link |
naively, you'll get 5% to 10%
link |
there's a peak performance of the GPU, right?
link |
And then there's a bunch of people published papers on this
link |
and I read them about what steps do you have to do?
link |
And it goes from pretty reasonable,
link |
well transpose one of the matrices.
link |
So you do rotor, not column ordered, you know, block it
link |
so that you can put a block of the matrix
link |
on different SMs, you know, groups of threads.
link |
But some of it gets into little details
link |
like you have to schedule it just so
link |
so you don't have register conflicts.
link |
So the, they call them CUDA ninjas.
link |
CUDA ninjas, I love it.
link |
To get to the optimal point,
link |
you either write a pre, use a pre written library
link |
which is a good strategy for some things
link |
or you have to be an expert
link |
in micro architecture to program it.
link |
Right, so the optimization step
link |
is way more complicated with the GPU.
link |
So our goal is, if you write PyTorch
link |
that's good PyTorch, you can do it.
link |
Now there's, as the networks are evolving, you know
link |
they've changed from convolutional to matrix multiply.
link |
The people are talking about conditional graphs,
link |
they're talking about very large matrices,
link |
they're talking about sparsity.
link |
They're talking about problems
link |
that scale across many, many chips.
link |
So the native, you know, data item is a packet.
link |
Like so you send the packet to a processor,
link |
it gets processed, it does a bunch of work
link |
and then it may send packets to other processors
link |
and they execute like a data flow graph kind of methodology.
link |
Got it. We have a big network on chip
link |
and then that second chip has 16 ethernet ports
link |
to help lots of them together
link |
and it's the same graph compiler across multiple chips.
link |
So that's where the scale comes in.
link |
So it's built to scale naturally.
link |
Now, my experience with scaling is as you scale
link |
you run into lots of interesting problems.
link |
So scaling is the amount of the climb.
link |
Yeah. So the hardware is built to do this
link |
and then we're in the process of...
link |
Is there a software part to this?
link |
With ethernet and all that?
link |
Well, the protocol at the bottom, you know, we send,
link |
it's an ethernet phi, but the protocol basically says
link |
send the packet from here to there.
link |
It's all point to point.
link |
The header bit says which processor to send it to
link |
and we basically take a packet off our on chip network,
link |
put an ethernet header on it, send it to the other end,
link |
strip the header off and send it to the local thing.
link |
It's pretty straightforward.
link |
Human to human interaction is pretty straightforward too
link |
but when you get a million of us
link |
we're just some crazy stuff together.
link |
Yeah, it can be fun.
link |
So is that the goal is scale?
link |
So like, for example, I've been recently doing a bunch
link |
of robots at home for my own personal pleasure.
link |
Am I going to ever use 10 storey or is this more for...
link |
There's all kinds of problems.
link |
Like there's small inference problems
link |
or small training problems or big training problems.
link |
What's the big goal?
link |
Is it the big training problems
link |
or the small training problems?
link |
There's one of the goals is to scale
link |
from 100 milliwatts to a megawatt.
link |
So like really have some range on the problems
link |
and the same kind of AI programs work
link |
at all different levels.
link |
The natural, since the natural data item is a packet
link |
that we can move around, it's built to scale
link |
but so many people have small problems.
link |
But like I said, that phone is a small problem to solve.
link |
So do you see 10 storey potentially being inside a phone?
link |
Well, the power efficiency of local memory,
link |
local computation and the way we built it is pretty good.
link |
And then there's a lot of efficiency
link |
on being able to do conditional graphs in sparsity.
link |
I think it's for complicated networks
link |
that want to go in a small factor, it's going to be quite good.
link |
But we have to prove that that's a fun problem.
link |
And that's the early days of the company, right?
link |
It's a couple of years, you said.
link |
But you think, you invested, you think they're legit
link |
as you join, well, that's...
link |
Well, it's also, it's a really interesting place to be.
link |
Like the AI world is exploding, you know?
link |
And I looked at some other opportunities
link |
like build a faster processor, which people want.
link |
But that's more on an incremental path
link |
than what's going to happen in AI in the next 10 years.
link |
So this is kind of an exciting place to be part of.
link |
The revolutions will be happening in the very space that's...
link |
And then lots of people are working on it,
link |
but there's lots of technical reasons why some of them,
link |
you know, aren't going to work out that well.
link |
And that's interesting.
link |
And there's also the same problem about getting the basics right.
link |
Like we've talked to customers about exciting features.
link |
And at some point, we realized that each of the networks,
link |
realizing they want to hear first about memory bandwidth,
link |
local bandwidth, compute intensity, programmability.
link |
They want to know the basics, power management,
link |
how the network ports work.
link |
What are the basics?
link |
Do all the basics work?
link |
Because it's easy to say we've got this great idea
link |
of the, you know, the crack GPT3.
link |
But the people we talk to want to say,
link |
So we have a piece of express card with our chip on it.
link |
If you buy the card, you plug it in your machine
link |
to download the driver.
link |
How long does it take me to get my network to run?
link |
You know, that's a real question.
link |
It's a very basic question.
link |
Is there an answer to that yet?
link |
Or is it's trying to get to it?
link |
Our goal is like an hour.
link |
When can I buy a test for it?
link |
For my, for the small case training.
link |
Yeah, pretty soon.
link |
I love the idea of you inside a room
link |
with the Carpathian, under Carpathian, Chris Ladner.
link |
Very, very interesting, very brilliant people,
link |
very out of the box thinkers,
link |
but also like first principles thinkers.
link |
Well, they both get stuff done.
link |
They only get stuff done to get their own projects done.
link |
They talk about it clearly.
link |
They educate large numbers of people
link |
and they've created platforms for other people
link |
to go do their stuff on.
link |
The clear thinking that's able to be communicated
link |
is kind of impressive.
link |
It's kind of remarkable to, yeah, I'm a fan.
link |
because I talked to Chris actually a lot these days.
link |
He's been a, one of the, just to give him a shout out
link |
and he's been so supportive as a human being.
link |
So everybody's quite different.
link |
Like great engineers are different,
link |
but he's been like sensitive to the human element
link |
in a way that's been fascinating.
link |
Like he was one of the early people
link |
on this stupid podcast that I do to say like,
link |
don't quit this thing and also talk to whoever
link |
the hell you want to talk to.
link |
That kind of from a legit engineer
link |
to get like props and be like, you can do this.
link |
That was, I mean, that's what a good leader does, right?
link |
It's just kind of let a little kid do his thing.
link |
Like go do it, let's see what turns out.
link |
That's a pretty powerful thing.
link |
But what do you, what's your sense about,
link |
he used to be, no, I think stepped away from Google, right?
link |
He said, sci fi, I think.
link |
What's really impressive to you
link |
about the things that Chris has worked on?
link |
As we mentioned, the optimization,
link |
the compiler design stuff, the LLVM,
link |
then there's, he's also at Google work, the TPU stuff.
link |
He's obviously worked on Swift,
link |
so the programming language side,
link |
talking about people that work in the entirety of the stack.
link |
What, from your time interacting with Chris
link |
and knowing the guy, what's really impressive to you?
link |
It just inspires you.
link |
Well, like LLVM became the de facto platform
link |
for compilers, it's amazing.
link |
And it was good code quality, good design choices.
link |
He hit the right level of abstraction.
link |
There's a little bit of the right time and the right place.
link |
And then he built a new programming language called Swift,
link |
which after, let's say some adoption resistance
link |
became very successful.
link |
I don't know that much about his work at Google,
link |
although I know that, that was a typical,
link |
they started TensorFlow stuff and they,
link |
it was new, they wrote a lot of code
link |
and then at some point it needed to be refactored to be,
link |
because it's development slowed down,
link |
why PyTorch started a little later and then passed it.
link |
So he did a lot of work on that.
link |
And then his idea about MLIR,
link |
which is what people started to realize
link |
is the complexity of the software stack
link |
above the low level IR was getting so high
link |
that forcing the features of that into low level
link |
was putting too much of a burden on it.
link |
So he's splitting that into multiple pieces.
link |
And that was one of the inspirations for our software stack
link |
where we have several intermediate representations
link |
that are all executable.
link |
And you can look at them and do transformations on them
link |
before you lower the level.
link |
So that was, I think we started before MLIR
link |
really got far enough along to use,
link |
but we're interested in that.
link |
He's really excited about MLIR.
link |
That's just like little baby.
link |
So he, and there seems to be some profound ideas on that
link |
that are really useful.
link |
So each one of those things has been,
link |
as the world of software gets more and more complicated,
link |
how do we create the right abstraction levels
link |
to simplify it in a way that people can now work independently
link |
on different levels of it?
link |
So I would say all three of those projects, LLVM, Swift,
link |
and MLIR did that successfully.
link |
So I'm interested in what he's going to do next
link |
in the same kind of way.
link |
On either the TPU or maybe the NVIDIA GPU side,
link |
how does TensorFlow, you think, or the ideas
link |
underlying it doesn't have to be TensorFlow.
link |
Just this kind of graph focused, graph centric hardware
link |
deep learning centric hardware beat NVIDIAs.
link |
Do you think it's possible for it to basically overtake NVIDIA?
link |
What's that process look like?
link |
What's that journey look like, do you think?
link |
Well, GPUs were built around shader programs
link |
on millions of pixels, not to run graphs.
link |
So there's a hypothesis that says,
link |
the way the graphs are built is going
link |
to be really interesting to be efficient on computing this.
link |
And then the primitives is not a SIMD program.
link |
It's a matrix multiply convolution.
link |
And then the data manipulations are fairly extensive
link |
about how do you do a fast transpose with a program.
link |
I don't know if you've ever written a transpose program.
link |
They're ugly and slow, but in hardware you can do really well.
link |
I've got to give you an example.
link |
So when GPU accelerators started doing triangles,
link |
so you have a triangle which maps on a set of pixels.
link |
So it's very easy, straightforward
link |
to build a hardware engine that will find all those pixels.
link |
And it's kind of weird because you walk along the triangle
link |
to get to the edge.
link |
And then you have to go back down to the next row
link |
And then you have to decide on the edge
link |
if the line of the triangle is like half on the pixel.
link |
What's the pixel color?
link |
Because it's half of this pixel and half the next one.
link |
That's called rasterization.
link |
And you're saying that could be done in hardware?
link |
No, that's an example of that operation
link |
as a software program is really bad.
link |
I've written a program that did rasterization.
link |
The hardware that does it has actually less code
link |
than the software program that does it.
link |
And it's way faster.
link |
So there are certain times when the abstraction you have
link |
rasterize a triangle, execute a graph,
link |
components of a graph, the right thing
link |
to do in the hardware software boundary
link |
is for the hardware to naturally do it.
link |
And so the GPU is really optimized
link |
for the rasterization of triangles.
link |
Well, like in a modern, that's a small piece of modern GPUs.
link |
What they did is that they still rasterized triangles
link |
when you're running a game.
link |
But for the most part, most of the computation
link |
in the area of the GPU is running shader programs.
link |
But there's single threaded programs on pixels, not graphs.
link |
To be honest, let's say I don't actually
link |
know the math behind shader, shading and lighting
link |
and all that kind of stuff.
link |
I don't know what.
link |
They look like little simple floating point programs
link |
or complicated ones.
link |
You can have 8,000 instructions in a shader program.
link |
But I don't have a good intuition
link |
why it could be parallelized so easily.
link |
No, it's because you have 8 million pixels in every single.
link |
So when you have a light that comes down, the amount of light,
link |
like say this is a line of pixels across this table,
link |
the amount of light on each pixel is subtly different.
link |
And each pixel is responsible for figuring out what it is.
link |
Figuring it out. So that pixel says, I'm this pixel.
link |
I know the angle of the light.
link |
I know the occlusion, I know the color I am.
link |
Like every single pixel here is a different color.
link |
Every single pixel gets a different amount of light.
link |
Every single pixel has a subtly different translucency.
link |
So to make it look realistic, the solution
link |
was you run a separate program on every pixel.
link |
See, but I thought there's a reflection from all over the place.
link |
Is it every pixel?
link |
Yeah, but there is.
link |
So you build a reflection map, which also
link |
has some pixelated thing.
link |
And then when the pixel is looking at the reflection map,
link |
it has to calculate what the normal of the surface is.
link |
And it does it per pixel.
link |
By the way, there's bull loads of hacks on that.
link |
You may have a lower resolution light map, reflection map.
link |
There's all these tax they do.
link |
But at the end of the day, it's per pixel computation.
link |
And it's so happening that you can map graph light computation
link |
onto this pixel central computation.
link |
You can do floating point programs
link |
on convolution and matrices.
link |
And NVIDIA invested for years in CUDA, first for HPC.
link |
And then they got lucky with the AI trend.
link |
But do you think they're going to essentially not
link |
be able to hardcore pivot out of their hole?
link |
That's always interesting.
link |
How often do big companies hardcore pivot occasionally?
link |
How much do you know about NVIDIA, folks?
link |
Well, I'm curious as well, who's ultimately as a?
link |
Well, they've innovated several times.
link |
But they've also worked really hard on mobile.
link |
They've worked really hard on radios.
link |
They're fundamentally a GPU company.
link |
Well, they tried to pivot.
link |
It's an interesting little game and play in autonomous vehicles,
link |
with or semi autonomous, like playing with Tesla and so on,
link |
and seeing that's dipping a toe into that kind of pivot.
link |
They came out with this platform, which
link |
is interesting technically.
link |
But it was like a $3,000 GPU platform.
link |
I don't know if it's interesting technically.
link |
It's interesting philosophically.
link |
I technically, I don't know if it's
link |
the execution the craftsmanship is there.
link |
I didn't get a sense.
link |
I think they were repurposing GPUs for an automotive solution.
link |
It's not a real pivot.
link |
They didn't build a ground up solution.
link |
Like the chips inside Tesla are pretty cheap.
link |
Like Mobileye has been doing this.
link |
They're doing the classic work from the simplest thing.
link |
They were building 40 square millimeter chips.
link |
And NVIDIA, their solution, had 800 millimeter chips
link |
and 200 millimeter chips.
link |
And like boatloads are really expensive DRAMs.
link |
And it's a really different approach.
link |
So Mobileye fit the, let's say, automotive cost and form
link |
And then they added features as it was economically viable.
link |
NVIDIA said, take the biggest thing
link |
and we're going to go make it work.
link |
And that's also influenced like Waymo.
link |
There's a whole bunch of autonomous startups
link |
where they have a 5,000 watt server in their trunk.
link |
And but that's because they think, well, 5,000 watts
link |
and $10,000 is OK because it's replacing a driver.
link |
Elon's approach was that port has
link |
to be cheap enough to put it in every single Tesla,
link |
whether they turn on autonomous driving or not.
link |
And Mobileye was like, we need to fit in the bomb
link |
and cost structure that car companies do.
link |
So they may sell you a GPS for 1,500 bucks.
link |
But the bomb for that's like $25.
link |
Well, and for Mobileye, it seems like neural networks
link |
were not first class citizens, like the computation.
link |
They didn't start out as a.
link |
Yeah, it was a CB problem.
link |
And they did classic CB and found stop lights and lines.
link |
And they were really good at it.
link |
Yeah, and they never, I mean, I don't know what's happening
link |
now, but they never fully pivoted.
link |
I mean, it's like, it's the NVIDIA thing.
link |
And then as opposed to, so if you look at the new Tesla work,
link |
it's like neural networks from the ground up, right?
link |
Yeah, and even Tesla started with a lot of CB stuff in it.
link |
And Andre's basically been eliminating it.
link |
Move everything into the network.
link |
So without, this isn't like confidential stuff,
link |
but you sitting on a porch looking over the world,
link |
looking at the work that Andre is doing,
link |
that Elon's doing with Tesla autopilot.
link |
Do you like the trajectory of where things are going
link |
on the hardware side?
link |
Well, they're making serious progress.
link |
I like the videos of people driving the beta stuff.
link |
Like it's taking some pretty complicated intersections
link |
and all that, but it's still an intervention per drive.
link |
I mean, I have autopilot, the current autopilot, my Tesla.
link |
I use it every day.
link |
Do you have full stop driving beta or no?
link |
So you like where this is going?
link |
They're making progress.
link |
It's taking longer than anybody thought.
link |
You know, my wonder was, you know, hardware three,
link |
is it enough computing?
link |
Off by two, off by five, off by 10, off by 100.
link |
And I thought it probably wasn't enough,
link |
but they're doing pretty well with it now.
link |
And one thing is, the data set gets bigger,
link |
the training gets better.
link |
And then there's this interesting thing is,
link |
you sort of train and build an arbitrary size network
link |
that solves the problem.
link |
And then you refactor the network down to the thing
link |
that you can afford to ship, right?
link |
So the goal isn't to build a network that fits in the phone.
link |
It's to build something that actually works.
link |
And then how do you make that most effective
link |
on the hardware you have?
link |
And they seem to be doing that much better
link |
than a couple of years ago.
link |
Well, the one really important thing is also
link |
what they're doing well is how to iterate that quickly,
link |
which means like it's not just about one time
link |
deployment, one building, it's constantly
link |
iterating the network and trying to automate
link |
as many steps as possible, right?
link |
And that's actually the principles of the software 2.0
link |
I can mention with Andre is it's not just,
link |
I mean, I don't know what the actual,
link |
his description of software 2.0 is.
link |
If it's just high level, full software, there's specifics.
link |
But the interesting thing about what that actually looks
link |
in the real world is it's that, what I think Andre
link |
calls the data engine.
link |
It's like, it's the iterative improvement of the thing.
link |
You have a neural network that does stuff,
link |
fails on a bunch of things and learns from it
link |
over and over and over.
link |
So you constantly discovering edge cases.
link |
So it's very much about like data engineering,
link |
like figuring out, it's kind of what you were talking about
link |
with TensorFlow is you have the data landscape.
link |
You have to walk along that data landscape in a way
link |
that it's constantly improving the neural network.
link |
And that feels like that's the central piece
link |
And there's two pieces of it, like you find edge cases
link |
that don't work and then you define something
link |
that goes get you data for that.
link |
But then the other constraint is whether you have
link |
to label it or not.
link |
Like the amazing thing about like the GPT3 stuff
link |
is it's unsupervised.
link |
So there's essentially infinite amount of data.
link |
Now there's obviously infinite amount of data available
link |
from cars of people who are successfully driving.
link |
But the current pipelines are mostly running on labeled data,
link |
which is human limited.
link |
So when that becomes unsupervised, it'll
link |
create unlimited amount of data, which is on a scale.
link |
Now the networks that may use that data
link |
might be way too big for cars, but then there'll
link |
be the transformation from now we have unlimited data.
link |
I know exactly what I want.
link |
Now can I turn that into something that fits in the car?
link |
And that process is going to happen all over the place.
link |
Every time you get to the place where you have unlimited data,
link |
that's what software 2.0 is about, unlimited data training
link |
networks to do stuff without humans writing code to do it.
link |
And ultimately also trying to discover
link |
like you're saying the self supervised formulation
link |
of the problem, so the unsupervised formulation
link |
Like in driving, there's this really interesting thing,
link |
which is you look at a scene that's before you,
link |
and you have data about what a successful human driver did
link |
in that scene one second later.
link |
It's a little piece of data that you can use just
link |
like with GPT3 as training.
link |
Currently, even though Tesla says they're using that,
link |
it's an open question to me.
link |
How far can you, can you saw all of the driving
link |
with just that self supervised piece of data?
link |
Well, that's what Common AI is doing.
link |
That's what Common AI is doing, but the question
link |
is how much data, so what Common AI doesn't have
link |
is as good of a data engine, for example, as Tesla does.
link |
That's where the organization of the data.
link |
I mean, as far as I know, I haven't talked to George,
link |
but they do have the data.
link |
The question is how much data is needed?
link |
Because we say infinite very loosely here.
link |
And then the other question, which you said,
link |
I don't know if you think it's still an open question,
link |
is are we on the right order of magnitude
link |
for the compute necessary?
link |
That is this, is it like what Elon said,
link |
this chip that's in there now is enough
link |
to do full self driving, or do we need
link |
another order of magnitude?
link |
I think nobody actually knows the answer to that question.
link |
I like the confidence that Elon has, but.
link |
There's another funny thing is you don't learn to drive
link |
with infinite amounts of data.
link |
You learn to drive with an intellectual framework
link |
that understands physics and color and horizontal surfaces
link |
and laws and roads and all your experience
link |
from manipulating your environment.
link |
There's so many factors go into that.
link |
And then when you learn to drive,
link |
driving is a subset of this conceptual framework
link |
And so with self driving cars right now,
link |
we're teaching them to drive with driving data.
link |
You never teach a human to do that.
link |
You teach a human all kinds of interesting things,
link |
like language, like don't do that, watch out.
link |
There's all kinds of stuff going on.
link |
This is where you, I think, previous time
link |
we talked about where you poetically disagreed
link |
with my naive notion about humans.
link |
I just think that humans will make
link |
this whole driving thing really difficult.
link |
I said, humans don't move that slow.
link |
It's a ballistics problem.
link |
It's a ballistics, humans are a ballistics problem,
link |
which is like poetry to me.
link |
It's very possible that in driving,
link |
they're indeed purely a ballistics problem.
link |
And I think that's probably the right way to think about it,
link |
but I still, they still continue to surprise me
link |
with those damp pedestrians, the cyclists,
link |
other humans in other cars and.
link |
Yeah, but it's going to be one of these compensating things.
link |
So like when you're driving, you have an intuition
link |
about what humans are going to do,
link |
but you don't have 360 cameras and radars
link |
and you have an attention problem.
link |
So the self driving car comes in with no attention problems,
link |
360 cameras, a bunch of other features.
link |
So they'll wipe out a whole class of accidents, right?
link |
And emergency braking with radar,
link |
and especially as it gets AI enhanced,
link |
will eliminate collisions, right?
link |
But then you have the other problems
link |
of these unexpected things where,
link |
you think your human intuition is helping,
link |
but then the cars also have a set of hardware features
link |
that you're not even close to.
link |
And the key thing of course is,
link |
if you wipe out a huge number of kind of accidents,
link |
then it might be just way safer than a human driver,
link |
even though, even if humans are still a problem,
link |
that's hard to figure out.
link |
Yeah, that's probably what will happen.
link |
So autonomous cars will have a small number of accidents,
link |
humans would have avoided,
link |
but they'll wipe, they'll get rid of the bulk of them.
link |
What do you think about like Tesla's dojo efforts,
link |
or it can be bigger than Tesla in general?
link |
It's kind of like the tense torrent,
link |
trying to innovate, like this is the economy,
link |
like should a company try to from scratch
link |
build its own neural network training hardware?
link |
Well, first I think it's great.
link |
So we need lots of experiments, right?
link |
And there's lots of startups working on this
link |
and they're pursuing different things.
link |
Now I was there when we started Dojo,
link |
and it was sort of like,
link |
what's the unconstrained computer solution
link |
to go do very large training problems.
link |
And then there's fun stuff like,
link |
we said, well, we have this 10,000 watt board to cool.
link |
Well, you go talk to guys at SpaceX,
link |
and they think 10,000 watts is a really small number,
link |
And there's brilliant people working on it.
link |
I'm curious to see how it'll come out.
link |
I couldn't tell you,
link |
I know it pivoted a few times since I left, so.
link |
So the cooling does seem to be a big problem.
link |
I do like what Elon said about it,
link |
which is like, we don't want to do the thing
link |
unless it's way better than the alternative,
link |
whatever the alternative is.
link |
So it has to be way better than like racks or GPUs.
link |
Yeah, and the other thing is just like,
link |
you know, the Tesla autonomous driving hardware,
link |
it was only serving one software stack.
link |
And the hardware team and the software team
link |
were tightly coupled.
link |
You know, if you're building a general purpose AI solution,
link |
then you know, there's so many different customers
link |
with so many different needs.
link |
Now, something Andre said is,
link |
I think this is amazing, 10 years ago,
link |
like vision, recommendation, language
link |
were completely different disciplines.
link |
I said, the people literally couldn't talk to each other.
link |
And three years ago, it was all neural networks,
link |
but the very different neural networks.
link |
And recently it's converging on one set of networks.
link |
They vary a lot in size, obviously,
link |
they vary in data, vary in outputs,
link |
but the technology has converged a good bit.
link |
Yeah, these transformers behind GPT3,
link |
it seems like they could be applied to video,
link |
they could be applied to a lot of,
link |
and it's like, and they're all really simple.
link |
And it was like to literally replace letters with pixels.
link |
It does vision, it's amazing.
link |
So, and then size actually improves the thing.
link |
So the bigger it gets,
link |
the more compute you throw at it, the better it gets.
link |
And the more data you have, the better it gets.
link |
So, so then you start to wonder,
link |
well, is that a fundamental thing,
link |
or is this just another step
link |
to some fundamental understanding
link |
about this kind of computation,
link |
which is really interesting.
link |
Us humans don't want to believe
link |
that that kind of thing will achieve
link |
conceptual understandings you were saying,
link |
like you'll figure out physics, but maybe it will.
link |
Maybe. Probably will.
link |
Well, it's worse than that.
link |
It'll understand physics in ways that we can't understand.
link |
I liked your Stephen Wolfram talk,
link |
where he said, you know,
link |
there's three generations of physics.
link |
There was physics by reasoning,
link |
well, big things should fall faster than small things,
link |
right, that's reasoning.
link |
And then there's physics by equations, like, you know,
link |
but the number of programs in a world that are solved
link |
with the single equations relatively low,
link |
almost all programs have, you know,
link |
more than one line of code,
link |
maybe a hundred million lines of code.
link |
So he said, now we're going to physics by equation,
link |
which is his project, which is cool.
link |
I might point out that there was two generations of physics
link |
before reasoning, habit.
link |
Like all animals, you know,
link |
no things fall and, you know, birds fly and, you know,
link |
predators know how to, you know,
link |
solve a differential equation to cut off a accelerating,
link |
you know, curving animal path.
link |
And then there was, you know, the gods did it, right?
link |
So, right, so there was, you know, there's five generations.
link |
Now, software 2.0 says programming things
link |
is not the last step, data.
link |
So there's going to be a physics
link |
past Stevens, Wolfram's comp.
link |
That's not explainable to us humans.
link |
And actually, there's no reason that I can see
link |
while that, even that's the limit, like,
link |
there's something beyond that.
link |
I mean, they're usually, like,
link |
usually when you have this hierarchy, it's not like,
link |
well, if you have this step and this step and this step
link |
and they're all qualitatively different,
link |
conceptually different, it's not obvious why, you know,
link |
six is the right ant number of hierarchy steps
link |
and not seven or eight or...
link |
Well, then it's probably impossible for us
link |
to comprehend something that's beyond
link |
the thing that's not explainable.
link |
Yeah, but the thing that, you know,
link |
understands the thing that's not explainable to us,
link |
Wolfram's conceives the next one and, like,
link |
I'm not sure why there's a limit to it.
link |
Looks like your brain hurts.
link |
That's the sad story.
link |
If we look at our own brain,
link |
which is an interesting, illustrative example,
link |
in your work with TestTorrent
link |
and trying to design deep learning architectures,
link |
do you think about the brain at all?
link |
Maybe from a hardware designer perspective,
link |
if you could change something about the brain,
link |
what would you change?
link |
Like, how would you do that?
link |
So your brain is really weird.
link |
Like, you know, your cerebral cortex,
link |
where we think we do most of our thinking,
link |
is what, like six or seven neurons thick?
link |
Like, that's weird.
link |
Like, all the big networks are way bigger than that.
link |
So that seems odd.
link |
And then, you know, when you're thinking,
link |
if the input generates a result you can lose,
link |
it goes really fast.
link |
But if it can't, that generates an output
link |
that's interesting, which turns into an input,
link |
and then your brain,
link |
to the point where you mull things over for days,
link |
and how many trips through your brain is that, right?
link |
Like, it's, you know, 300 milliseconds or something,
link |
and you get through seven levels of neurons.
link |
I forget the number exactly.
link |
But then it does it over and over and over as it searches.
link |
And the brain clearly looks like some kind of graph
link |
because you have a neuron with, you know, connections,
link |
and it talks to other ones.
link |
And it's locally very computationally intense,
link |
but it also does sparse computations
link |
across a pretty big area.
link |
There's a lot of messy biological type of things,
link |
and it's meaning like, first of all,
link |
there's mechanical, chemical, and electrical signals.
link |
It's all that's going on.
link |
Then there's the acynchronicity of signals.
link |
And there's like, there's just a lot of variability
link |
that seems continuous and messy
link |
and just a mess of biology.
link |
And it's unclear whether that's a good thing
link |
or it's a bad thing.
link |
Because if it's a good thing,
link |
then we need to run the entirety of the evolution.
link |
Well, we're gonna have to start with basic bacteria
link |
to create something.
link |
Imagine you could build a brain with 10 layers.
link |
Would that be better or worse?
link |
Or more connections or less connections,
link |
or we don't know to what level our brains are optimized.
link |
But if I was changing things,
link |
like you can only hold like seven numbers in your head.
link |
Like why not a hundred or a million?
link |
There was a lot of that.
link |
And why can't we have like a floating point processor
link |
that can compute anything we want
link |
and see it all properly?
link |
Like that would be kind of fun.
link |
And why can't we see in four or eight dimensions?
link |
Like three D's kind of a drag.
link |
Like all the hard mass transforms
link |
are up in multiple dimensions.
link |
So there's, you could imagine a brain architecture
link |
that you could enhance with a whole bunch of features
link |
that would be really useful for thinking about things.
link |
It's possible that the limitations you're describing
link |
are actually essential for like the constraints
link |
are essential for creating like the depth of intelligence.
link |
Like that, the ability to reason, you know.
link |
Yeah, it's hard to say
link |
because like your brain is clearly a parallel processor.
link |
You know, 10 billion neurons talking to each other
link |
at a relatively low clock rate.
link |
But it produces something that looks like
link |
a serial thought process.
link |
It's a serial narrative in your head.
link |
But then there are people famously who are visual thinkers.
link |
Like I think I'm a relatively visual thinker.
link |
I can imagine any object and rotate it in my head
link |
And there are people who say they don't think that way at all.
link |
And recently I read an article about people
link |
who say they don't have a voice in their head.
link |
They can talk, but when they, you know, it's like,
link |
well, what are you thinking though?
link |
They'll describe something that's visual.
link |
So that's curious.
link |
Now, if you're saying, if we dedicated more hardware
link |
to holding information like, you know, 10 numbers
link |
or a million numbers, like would that distract us
link |
from our ability to form this kind of singular identity?
link |
Like it dissipates somehow.
link |
But maybe, you know, future humans will have many identities
link |
that have some higher level organization,
link |
but can actually do lots more things in parallel.
link |
Yeah, there's no reason, if we're thinking modularly,
link |
there's no reason we can have multiple consciousnesses
link |
And maybe there's some way to make it faster
link |
so that the, you know, the area of the computation
link |
could still have a unified feel to it
link |
while still having way more ability
link |
to do parallel stuff at the same time.
link |
Could definitely be improved.
link |
Could be improved?
link |
Well, it's pretty good right now.
link |
Actually, people don't give it enough credit.
link |
The thing is pretty nice that, you know,
link |
the fact that the ride ends seem to be,
link |
give a nice like spark of beauty to the whole experience.
link |
So I don't know, I don't know if it can be improved easily.
link |
It could be more beautiful.
link |
I don't know how, what do you mean?
link |
What do you mean how?
link |
All the ways you can't imagine.
link |
No, but that's the whole point.
link |
I wouldn't be able to imagine,
link |
the fact that I can imagine ways
link |
in which it could be more beautiful means.
link |
So do you know, you know, Ian Banks, his stories.
link |
So the super smart AIs, they're live,
link |
mostly live in the world of what they call infinite fun
link |
because they can create arbitrary worlds.
link |
So they interact and, you know, the story has it.
link |
They interact in the normal world
link |
and they're very smart and they can do all kinds of stuff.
link |
And, you know, given mine can, you know,
link |
talk to a million humans at the same time
link |
because we're very slow and for reasons,
link |
you know, artificial, the story,
link |
they're interested in people and doing stuff,
link |
but they mostly live in this other land of thinking.
link |
My inclination is to think that the ability
link |
to create infinite fun will not be so fun.
link |
There's so many things to do.
link |
Imagine being able to make a star, move planets around.
link |
But because we can imagine that as wildlife is fun,
link |
if we actually were able to do it, it'd be a slippery slope
link |
where fun, we'd even have a meeting
link |
because we just consistently desensitize ourselves
link |
by the infinite amounts of fun we're having.
link |
And the sadness, the dark stuff is what makes it fun, I think.
link |
That could be the Russian.
link |
It could be the fun makes it fun
link |
and the sadness makes it bittersweet.
link |
Yeah, that's true.
link |
Fun could be the thing that makes it fun.
link |
So what do you think about the expansion,
link |
not through the biology side,
link |
but through the BCI, the brain computer interfaces?
link |
Yeah, you got a chance to check out the Neuralink stuff.
link |
It's super interesting.
link |
Like humans, like our thoughts manifest as action.
link |
Like as a kid, shooting a rifle was super fun,
link |
driving a mini bike, doing things.
link |
And then computer games, I think,
link |
for a lot of kids became the thing where they can do
link |
what they want, they can fly a plane,
link |
they can do this, they can do this, right?
link |
But you have to have this physical interaction.
link |
Now imagine, you know, you could just imagine stuff
link |
and it happens, right?
link |
Like really richly and interestingly.
link |
Like we kind of do that when we dream.
link |
Like dreams are funny because like,
link |
if you have some control or awareness in your dreams,
link |
like it's very realistic looking
link |
or not realistic, it depends on the dream.
link |
But you can also manipulate that.
link |
And you know, what's possible there is odd
link |
in the fact that nobody understands it's hilarious, but.
link |
Do you think it's possible to expand
link |
that capability through computing?
link |
Is there some interesting,
link |
so from a hardware designer perspective,
link |
is there, do you think it'll present totally new challenges
link |
in the kind of hardware that required that like,
link |
so this hardware isn't standalone computing.
link |
So just take it from this, so today,
link |
computer games are rendered by GPUs, right?
link |
So, but you've seen the GAN stuff, right?
link |
Where train neural networks render realistic images,
link |
but there's no pixels, no triangles, no shaders,
link |
no light maps, no nothing.
link |
So the future of graphics is probably AI, right?
link |
Now that AI is heavily trained by lots of real data, right?
link |
So if you have an interface with a AI renderer, right?
link |
So if you say render a cat,
link |
it won't say, well, how tall is the cat
link |
and how big, you know, it'll render a cat.
link |
You might say, oh, a little bigger, a little smaller,
link |
you know, make it a tabby, shorter hair, you know,
link |
like you could tweak it.
link |
Like the amount of data you'll have to send
link |
to interact with a very powerful AI renderer could be low.
link |
But the question is, for brain computer interfaces,
link |
we need to render not onto a screen,
link |
but render onto the brain.
link |
And like directly, so there's a bandwidth.
link |
Well, we could do it both ways.
link |
I mean, our eyes are really good sensors.
link |
They could render onto a screen
link |
and we could feel like we're participating in it.
link |
You know, they're gonna have, you know,
link |
like the Oculus kind of stuff.
link |
It's gonna be so good when a projection to your eyes,
link |
you think it's real.
link |
You know, they're slowly solving those problems.
link |
And I suspect when the renderer of that information
link |
into your head is also AI mediated,
link |
they'll be able to give you the cues that, you know,
link |
you really want for depth and all kinds of stuff.
link |
Like your brain is partly faking your visual field, right?
link |
Like your eyes are twitching around,
link |
but you don't notice that.
link |
Occasionally they blank, you don't notice that.
link |
You know, there's all kinds of things.
link |
Like you think you see over here,
link |
but you don't really see there.
link |
It's all fabricated.
link |
Yeah, peripheral vision is fascinating.
link |
So if you have an AI renderer that's trained
link |
to understand exactly how you see
link |
and the kind of things that enhance the realism
link |
of the experience could be super real, actually.
link |
So I don't know what the limits that are,
link |
but obviously if we have a brain interface
link |
that goes in inside your, you know, visual cortex
link |
in a better way than your eyes do, which is possible.
link |
It's a lot of neurons.
link |
Maybe that'll be even cooler.
link |
Well, the really cool thing is it has to do
link |
with the infinite fun that you were referring to,
link |
which is our brains have to be very limited.
link |
And like you said, computations.
link |
It's also very plastic.
link |
Very plastic, yeah.
link |
So it's a interesting combination.
link |
The interesting open question is the limits
link |
of that neuroplasticity.
link |
Like how flexible is that thing?
link |
Because we haven't really tested it.
link |
We know about that experience where they put
link |
like a pressure pad on somebody's head
link |
and had a visual transducer pressurize it
link |
and somebody slowly learned to see.
link |
Especially at a young age, if you throw a lot at it,
link |
like what can it completely,
link |
so can you like arbitrarily expand it with computing power?
link |
So connected to the internet directly somehow.
link |
Yeah, the answer is probably yes.
link |
So the problem with biology and ethics is like,
link |
there's a mess there.
link |
Like us humans are perhaps unwilling to take risks
link |
into directions that are full of uncertainty.
link |
90% of the population's unwilling to take risks.
link |
The other 10% is rushing into the risks
link |
unaided by any infrastructure whatsoever.
link |
And that's where all the fun happens in this society.
link |
There's been huge transformations
link |
in the last couple of thousand years.
link |
I got the chance to interact with this Matthew Johnson
link |
from Johns Hopkins.
link |
He's doing this large scale study of psychedelics.
link |
It's becoming more and more.
link |
I've gotten a chance to interact
link |
with that community of scientists working on psychedelics.
link |
But because of that, that opened the door to me
link |
to all these, what are they called?
link |
Psychonauts, the people who, like you said, the 10%.
link |
Like I don't care.
link |
I don't know if there's a science behind this.
link |
I'm taking the spaceship to, if I'm be the first on Mars,
link |
I'll be, you know, psychedelics interesting in the sense
link |
that in another dimension, like you said,
link |
it's a way to explore the limits of the human mind.
link |
Like what is this thing capable of doing?
link |
Cause you kind of, like when you dream,
link |
you detach it, I don't know exactly
link |
in your science of it,
link |
but you detach your like reality from what your mind,
link |
the images your mind is able to conjure up
link |
and your mind goes into weird places.
link |
And like entities appear,
link |
somehow Freudian type of like trauma
link |
is probably connected in there somehow.
link |
You start to have like these weird, vivid worlds that like.
link |
So do you actively dream?
link |
I have like six hours of dreams
link |
and it's like real useful time.
link |
I don't, I don't for some reason, I just knock out
link |
and I have sometimes like anxiety inducing kind of like
link |
a very pragmatic like nightmare type of dreams,
link |
but not nothing fun, nothing.
link |
I try, I unfortunately have mostly have fun
link |
in the waking world,
link |
which is very limited in the amount of fun you can have.
link |
It's not that limited either.
link |
Yeah, that's why we'll have to talk.
link |
Yeah, and your instructions.
link |
There's like a manual for that.
link |
I'll ask you on what, what did you dream?
link |
You know, years ago when I read about, you know,
link |
like, you know, a book about how to have, you know,
link |
become aware in your dreams.
link |
I worked on it for a while.
link |
Like there's this trick about, you know,
link |
imagine you can see your hands and look out
link |
and I got somewhat good at it.
link |
Like, but my mostly when I'm thinking about things
link |
or working on problems,
link |
I prep myself before I go to sleep.
link |
It's like, I pull into my mind all the things
link |
I wanna work on or think about.
link |
And then that, let's say, greatly improves the chances
link |
that I'll work on that while I'm sleeping.
link |
And then I also, you know, basically asked to remember it.
link |
And I often remember very detailed.
link |
Or outside the dream.
link |
Well, to bring it up in my dreaming
link |
and then to remember it when I wake up.
link |
It's just, it's more of a meditative practice.
link |
You say, you know, to prepare yourself to do that.
link |
Like if you go to, you know, to sleep,
link |
still gnashing your teeth about some random thing
link |
that happened that you're not that really interested
link |
in your dream about it.
link |
That's really interesting.
link |
But you can direct your dreams somewhat by prepping.
link |
Yeah, I'm gonna have to try that.
link |
It's really interesting.
link |
Like the most important, the interesting,
link |
not like what did this guy send an email
link |
kind of like stupid worry stuff,
link |
but like fundamental problems you're actually concerned about.
link |
And interesting things you're worried about.
link |
Or most of your reading or, you know,
link |
some great conversation you had
link |
or some adventure you want to have.
link |
Like there's a lot of space there and it seems to work
link |
that, you know, my percentage of interesting dreams
link |
and memories went up.
link |
Is there, is that the source of,
link |
if you were able to deconstruct like where
link |
some of your best ideas came from,
link |
is there a process that's at the core of that?
link |
Like, so some people, you know, walk and think,
link |
some people like in the shower, the best ideas hit them.
link |
If you talk about like Newton,
link |
Apple hitting them on the head.
link |
No, I found out a long time ago,
link |
I process things somewhat slowly.
link |
So like in college, I had friends that could study it
link |
the last minute and get an A next day.
link |
I can't do that at all.
link |
So I always front loaded all the work.
link |
Like I do all the problems early, you know,
link |
for finals, like the last three days,
link |
I wouldn't look at a book because I want, you know,
link |
cause like a new fact day before finals may screw up
link |
my understanding of what I thought I knew.
link |
So my goal was to always get it in
link |
and give it time to soak.
link |
And I used to, you know,
link |
I remember when we were doing like 3D calculus,
link |
I would have these amazing dreams of 3D surfaces
link |
with normal, you know, calculating the gradient.
link |
And just like all come up.
link |
So it was like really fun, like very visual.
link |
And if I got cycles of that, that was useful.
link |
And the other is just don't over filter your ideas.
link |
Like I like that process of brainstorming
link |
where lots of ideas can happen.
link |
I like people who have lots of ideas.
link |
And then you just let them sit.
link |
Then there's a, yeah, I'll let them sit
link |
and let it breathe a little bit.
link |
And then reduce it to practice.
link |
Like at some point you really have to, does it really work?
link |
Like, you know, is this real or not?
link |
Right, but you have to do both.
link |
There's creative tension there.
link |
Like how do you be both open and, you know, precise?
link |
Have you had ideas that you just,
link |
that sit in your mind for like years before the?
link |
It's an interesting way to just generate ideas
link |
and just let them sit.
link |
Let them sit there for a while.
link |
I think I have a few of those ideas.
link |
You know, it was so funny.
link |
Yeah, I think that's, you know, creativity,
link |
this one or something.
link |
For the slow thinkers in the, in the room, I suppose.
link |
As I, some people, like you said, are just like, like the.
link |
Yeah, it's really interesting.
link |
There's so much diversity in how people think, you know,
link |
how fast or slow they are, how well they remember,
link |
don't like, you know, I'm not super good at remembering facts,
link |
but processes and methods.
link |
Like in our engineering, I went to Penn State
link |
and almost all our engineering tests were open book.
link |
I could remember the page and not the formula.
link |
As soon as I saw the formula,
link |
I could remember the whole method if I, if I'd learned it.
link |
So it's a funny, where some people could, you know,
link |
I just watched friends like flipping through the book,
link |
trying to find the formula,
link |
even knowing that they'd done just as much work.
link |
Now we just opened the book and I was on page 27.
link |
About a half, I could see the whole thing visually.
link |
And you have to learn that about yourself
link |
and figure out what the, what the function optimally.
link |
I had a friend who, he was always concerned.
link |
He didn't know how he came up with ideas.
link |
He had lots of ideas, but he said they just sort of popped up.
link |
Like he'd be working on something, he had this idea.
link |
Like, where does it come from?
link |
But you can have more awareness of it.
link |
Like, like, like, like how you,
link |
how your brain works as a little murky as you go down
link |
from the voice in your head or the obvious visualizations.
link |
Like when you visualize something, how does that happen?
link |
Yeah, that's right.
link |
You know, if I say, you know, visualize a volcano,
link |
it's easy to do, right?
link |
And what does it actually look like when you visualize it?
link |
I can visualize to the point where I don't see
link |
very much out of my eyes and I see the colors
link |
of the thing I'm visualizing.
link |
Yeah, but there's like a, there's a shape, there's a texture,
link |
there's a color, but there's also conceptual visualization.
link |
Like, what are you actually visualizing
link |
when you're visualizing a volcano?
link |
Just like with peripheral vision,
link |
you think you see the whole thing.
link |
That's a good way to say it.
link |
You know, you have this kind of almost peripheral vision
link |
of your visualizations, they're like these ghosts.
link |
But if, you know, if you, if you work on it,
link |
you can get a pretty high level of detail.
link |
And somehow you can walk along those visualizations
link |
to come up with an idea, which is weird.
link |
But when you're thinking about solving problems,
link |
like you're putting information in,
link |
you're exercising the stuff you do know,
link |
you're sort of teasing the area that's,
link |
you don't understand and don't know,
link |
but you can almost, you know, feel,
link |
you know, that process happening.
link |
You know, that's, that's how I, like,
link |
like, I know sometimes when I'm working really hard
link |
on something, like, I get really hot when I'm sleeping.
link |
And, you know, it's like, we got the blank throw,
link |
I wake up with all the blank throw on the floor.
link |
And, you know, every time it's while I wake up
link |
and think, wow, that was great, you know.
link |
Oh, you're able to reverse engineer
link |
what the hell happened there?
link |
Oh, sometimes it's vivid dreams.
link |
And sometimes it's just kind of like you say,
link |
like shadow thinking that you sort of have this feeling
link |
you're going through this stuff,
link |
but it's not that obvious.
link |
Isn't that so amazing that the mind just does
link |
all these little experiments?
link |
I never, you know, I thought, I always thought,
link |
it's like a river that you can't,
link |
you're just there for the ride, but you're right.
link |
No, it's all understandable.
link |
Meditation really helps.
link |
You gotta start figuring out, you need to learn language
link |
if you're on mind.
link |
And there's multiple levels of it, but.
link |
The abstractions again, right?
link |
It's somewhat comprehensible and observable and feelable
link |
or whatever the right word is.
link |
You know, you're not alone for the ride.
link |
I have to ask you, hardware engineer working
link |
on your own networks now, what's consciousness?
link |
What the hell is that thing?
link |
Is that, is that just some little weird quirk
link |
of our particular computing device?
link |
Or is it something fundamental that we really need
link |
to crack open it for, to build like good computers?
link |
Do you ever think about consciousness?
link |
Like why it feels like something to be?
link |
I know, it's really weird.
link |
I mean, everything about it's weird.
link |
First is to half a second behind reality, right?
link |
It's a post hoc narrative about what happened.
link |
You've already done stuff by the time you're conscious of it.
link |
And your consciousness generally is a single threaded thing,
link |
but we know your brain is 10 billion neurons running
link |
some crazy parallel thing.
link |
And there's a really big sorting thing going on there.
link |
It also seems to be really reflective in the sense
link |
that you create a space in your head, right?
link |
Like we don't really see anything, right?
link |
Like photons hit your eyes, it gets turned into signals,
link |
it goes through multiple layers of neurons.
link |
You know, like I'm so curious that, you know,
link |
that looks glassy and that looks not glassy.
link |
And like, like how the resolution of your vision is so high,
link |
you had to go through all this processing.
link |
Where for most of it, it looks nothing like vision, right?
link |
Like, like there's no theater in your mind, right?
link |
So we, we have a world in our heads.
link |
We're literally disisolated behind our sensors,
link |
but we can look at it, speculate about it,
link |
speculate about alternatives, problem solve, what if,
link |
you know, there's so many things going on,
link |
and that process is lagging reality.
link |
And it's single threaded,
link |
even though the underlying thing is like massively parallel.
link |
Yeah, so it's, it's so curious.
link |
So imagine you're building an AI computer,
link |
if you wanted to replicate humans,
link |
well, you'd have huge arrays of neural networks,
link |
and apparently only sixers have in deep, which is hilarious.
link |
They don't even remember seven numbers,
link |
but I think we can upgrade that a lot, right?
link |
And then somewhere in there,
link |
you would train the network to create basically
link |
the world that you live in, right?
link |
So like tell stories to itself about the world
link |
that it's perceiving.
link |
Well, create the world, tell stories in the world,
link |
and then have many dimensions of, you know,
link |
like side jokes to it.
link |
Like we have an emotional structure,
link |
like we have a biological structure,
link |
and that seems hierarchical too,
link |
like if you're hungry, it dominates your thinking.
link |
If you're mad, it dominates your thinking.
link |
Like, and we don't know if that's important
link |
to consciousness or not, but it certainly disrupts,
link |
you know, intrudes in the consciousness.
link |
Like so there's lots of structure to that,
link |
and we like to dwell on the past,
link |
we like to think about the future,
link |
we like to imagine, we like to fantasize, right?
link |
And the somewhat circular observation of that
link |
is the thing we call consciousness.
link |
Now, if you created a computer system
link |
that did all things, created worldviews,
link |
created future alternate histories,
link |
you know, dwelled on past events,
link |
you know, accurately or semi accurately, you know, it's...
link |
Well, consciousness just spring up like naturally.
link |
Well, would that feel, look and feel conscious to you?
link |
Like you seem conscious to me, but I don't know.
link |
Like external observer sense.
link |
Do you think a thing that looks conscious is conscious?
link |
Like, do you, again,
link |
this is like an engineering kind of question, I think,
link |
because like, if we want to engineer consciousness,
link |
is it okay to engineer something
link |
that just looks conscious?
link |
Or is there a difference between something that is...
link |
Well, we have all consciousness
link |
because it's a super effective way to manage our affairs.
link |
Yeah, yeah, this is a social element, yeah.
link |
Well, it gives us a planning system, you know,
link |
we have a huge amount of stuff.
link |
Like when we're talking,
link |
like the reason we can talk really fast is we're modeling
link |
each other a really high level of detail.
link |
And consciousness is required for that.
link |
Well, all those components together manifest consciousness.
link |
So if we make intelligent beings that we want to interact with
link |
that we're like, you know, wondering what they're thinking,
link |
you know, you know, looking forward to seeing them,
link |
you know, when they interact with them,
link |
they're interesting, surprising, you know, fascinating,
link |
you know, they will probably be feel conscious like we do
link |
and we'll perceive them as conscious.
link |
I don't know why not, but never know.
link |
Another fun question on this,
link |
because in, from a computing perspective,
link |
we're trying to create something that's human like
link |
or super human like.
link |
Let me ask you about aliens.
link |
Do you think there's intelligent alien civilizations
link |
out there and do you think their technology,
link |
their computing, their AI bots,
link |
their chips are of the same nature as ours?
link |
Yeah, I got no idea.
link |
I mean, if there's lots of aliens out there,
link |
they've been awfully quiet.
link |
You know, there's your speculation about why
link |
there seems to be more than enough planets out there.
link |
There's intelligent life on this planet
link |
that seems quite different, you know, like, you know,
link |
dolphins seem like plausibly understandable.
link |
Octopuses don't seem understandable at all.
link |
If they live longer than a year,
link |
maybe they would be running the planet.
link |
They seem really smart.
link |
And their neuro architecture is completely different than ours.
link |
Now, who knows how they perceive things.
link |
I mean, that's the question is for us intelligent beings,
link |
we might not be able to perceive other kinds of intelligence
link |
if they become sufficiently different than us.
link |
Yeah, like we live in the current constrained world,
link |
you know, it's three dimensional geometry
link |
and the geometry defines a certain amount of physics.
link |
And, you know, there's like how time work seems to work.
link |
Like there's so many things that seem like a whole bunch
link |
of the input parameters to the, you know,
link |
another conscious being are the same.
link |
Like if it's biological, biological things seem to be
link |
in a relatively narrow temperature range, right?
link |
Because, you know, organics don't aren't stable,
link |
too cold or too hot, you know, so, so there's,
link |
if you specify the list of things that input to that,
link |
but soon as we make really smart, you know, beings
link |
and they go solve about how to think about a billion numbers
link |
at the same time and then how to think in n dimensions.
link |
There's a funny science fiction book
link |
where all the society had uploaded into this matrix.
link |
And at some point, some of the beings in the matrix thought,
link |
I wonder if there's intelligent life out there.
link |
So they had to do a whole bunch of work to figure out
link |
like how to make a physical thing
link |
because their matrix was self sustaining
link |
and they made a little spaceship
link |
and they traveled to another planet.
link |
When they got there, there was like life running around,
link |
but there was no intelligent life.
link |
And then they figured out that there was these huge,
link |
you know, organic matrix all over the planet inside there
link |
where intelligent beings had uploaded themselves
link |
and into that matrix.
link |
So everywhere intelligent life was, soon as it got smart,
link |
it up leveled itself into something way more interesting
link |
than 3D geometry and...
link |
Yeah, it escaped whatever the...
link |
It's not escaped, it's...
link |
Upload was better.
link |
The essence of what we think of as an intelligent being,
link |
I tend to like the thought experiment of the organism,
link |
like humans aren't the organisms.
link |
I like the notion of like Richard Dawkins and memes
link |
that ideas themselves are the organisms,
link |
like they're just using our minds to evolve.
link |
So like we're just like meat receptacles
link |
for ideas to breed and multiply and so on.
link |
And maybe those are the aliens.
link |
So Jordan Peterson has a line that says, you know,
link |
you think you have ideas, but ideas have you.
link |
And then we know about the phenomenon of groupthink
link |
and there's so many things that constrain us.
link |
But I think you can examine all that
link |
and not be completely owned by the ideas
link |
and completely sucked into groupthink.
link |
And part of your responsibility as a human
link |
is to escape that kind of phenomena, which isn't...
link |
You know, it's one of the creative tension things again.
link |
You're constructed by it, but you can still observe it
link |
and you can think about it
link |
and you can make choices about to some level
link |
how constrained you are by it.
link |
And, you know, it's useful to do that.
link |
But at the same time, and it could be by doing that,
link |
you know, the group and society you're part of
link |
becomes collectively even more interesting.
link |
So, you know, so that the outside observer will think,
link |
wow, you know, all these lexes running around
link |
with all these really independent ideas
link |
have created something even more interesting
link |
So, so I don't know.
link |
Those are lenses to look at the situation.
link |
That'll give you some inspiration,
link |
but I don't think they're constrained.
link |
As a small little quirk of history,
link |
it seems like you're related to Jordan Peterson,
link |
like you mentioned.
link |
He's going through some rough stuff now.
link |
Is there some comment you can make
link |
about the roughness of the human journey,
link |
Well, I became an expert in Benz withdrawal.
link |
Like, which is, you took Benz as the aspenes
link |
and at some point they interact with GABA circuits,
link |
you know, to reduce anxiety and do a hundred other things.
link |
Like, there's actually no known list of everything they do
link |
because they interact with so many parts of your body.
link |
And then once you're on them, you habituate to them
link |
and you have a dependency.
link |
It's not like you're a drug dependency.
link |
We're trying to get high.
link |
It's a metabolic dependency.
link |
And then if you discontinue them,
link |
there's a funny thing called kindling,
link |
which is if you stop them and then go,
link |
you know, you'll have a horrible withdrawal symptoms.
link |
If you go back on them at the same level,
link |
you won't be stable.
link |
And that unfortunately happened to him.
link |
Because it's so deeply integrated
link |
into all the kinds of systems in the body?
link |
It literally changes the size and numbers
link |
of neurotransmitter sites in your brain.
link |
So there's a process called the Ashton protocol
link |
where you taper it down slowly over two years.
link |
The people go through that, go through unbelievable hell.
link |
And what Jordan went through seemed to be worse
link |
because on advice of doctors, you know,
link |
we'll stop taking these and take this.
link |
It was a disaster and he got some.
link |
Yeah, it was pretty tough.
link |
He seems to be doing quite a bit better intellectually.
link |
You can see his brain clicking back together.
link |
I spent a lot of time with him.
link |
I've never seen anybody suffer so much.
link |
Well, his brain is also like this powerhouse, right?
link |
So I wonder, does a brain that's able to think deeply
link |
about the world suffer more to these kinds of withdrawals?
link |
Like, I don't know.
link |
I've watched videos of people going through withdrawal.
link |
They all seem to suffer unbelievably.
link |
And, you know, my heart goes out to everybody.
link |
And there's some funny math about this.
link |
Some doctors said as best you can tell, you know,
link |
there's the standard recommendations
link |
don't take them for more than a month
link |
and then taper over a couple of weeks.
link |
Many doctors prescribe them endlessly,
link |
which is against the protocol, but it's common, right?
link |
And then something like 75% of people,
link |
when they taper it's, you know,
link |
half the people have difficulty,
link |
but 75% get off okay.
link |
20% have severe difficulty
link |
and 5% have life threatening difficulty.
link |
And if you're one of those, it's really bad.
link |
And the stories that people have on this
link |
is heartbreaking and tough.
link |
So you put some of the fault at the doctors.
link |
They just not know what the hell they're doing.
link |
Oh, that was hard to say.
link |
It's one of those commonly prescribed things.
link |
Like one doctor said, what happens is
link |
if you're prescribed them for a reason
link |
and then you have a hard time getting off,
link |
the protocol basically says you're either crazy
link |
or dependent and you get kind of pushed
link |
into a different treatment regime.
link |
You're a drug addict or a psychiatric patient.
link |
And so like one doctor said, you know,
link |
I prescribed them for 10 years thinking
link |
I was helping my patients
link |
and I realized I was really harming them.
link |
And, you know, the awareness of that is slowly coming up.
link |
The fact that they're casually prescribed to people
link |
is horrible and it's bloody scary.
link |
And some people are stable on them,
link |
but they're on them for life.
link |
Like once you, you know,
link |
it's another one of those drugs that,
link |
but Benzo's long range have real impacts
link |
on your personality.
link |
People talk about the Benzo bubble
link |
where you get disassociated from reality
link |
and your friends a little bit.
link |
It's really terrible.
link |
The mind is terrifying.
link |
We were talking about how the infinite possibility of fun,
link |
but like it's the infinite possibility of suffering too,
link |
which is one of the dangers of like expansion
link |
of the human mind.
link |
It's like, I wonder if all the possible human experiences
link |
that intelligent computer can have,
link |
is it mostly fun or is it mostly suffering?
link |
So like if you brute force expand the set of possibilities
link |
like are you going to run into some trouble
link |
in terms of like torture and suffering and so on?
link |
Maybe our human brain is just protecting us
link |
from much more possible pain and suffering.
link |
Maybe the space of pain is like much larger
link |
than we could possibly imagine and that.
link |
The world's in a balance.
link |
You know, all the literature on religion and stuff is,
link |
you know, the struggle between good and evil
link |
is balanced for very finely tuned
link |
for reasons that are complicated.
link |
But that's a long philosophical conversation.
link |
Speaking of balance that's complicated,
link |
I wonder because we're living through one
link |
of the more important moments in human history
link |
with this particular virus,
link |
it seems like pandemics have at least the ability
link |
to kill off most of the human population at their worst.
link |
And there's just fascinating
link |
because there's so many viruses in this world.
link |
There's so many, I mean viruses basically run the world
link |
in the sense that they've been around for a very long time.
link |
They're everywhere.
link |
They seem to be extremely powerful
link |
and they're distributed kind of way,
link |
but at the same time they're not intelligent
link |
and they're not even living.
link |
Do you have like high level thoughts about this virus
link |
that like in terms of you being fascinated
link |
or terrified or somewhere in between?
link |
So I believe in frameworks, right?
link |
So like one of them is evolution.
link |
Like we're evolved creatures, right?
link |
And one of the things about evolution
link |
is it's hyper competitive.
link |
And it's not competitive out of a sense of evil.
link |
It's competitive in a sense of there's endless variation
link |
and variations that work better when.
link |
And then over time, there's so many levels
link |
of that competition, like multi cellular life
link |
partly exists because of the competition
link |
between different kinds of life forms.
link |
And we know sex partly exists to scramble our genes
link |
so that we have genetic variation
link |
against the invasion of the bacteria and the viruses.
link |
Like I read some funny statistic,
link |
like the density of viruses and bacteria in the ocean
link |
And one third of the bacteria die every day
link |
because the virus is invading them.
link |
Like one third of them.
link |
Like I don't know if that number is true,
link |
but it was like there's like the amount of competition
link |
and what's going on is stunning.
link |
And there's a theory as we age,
link |
we slowly accumulate bacterias and viruses
link |
and as our immune system kind of goes down,
link |
that's what slowly kills us.
link |
It just feels so peaceful from a human perspective
link |
when we sit back and they're able to have a relaxed
link |
conversation and there's wars going on out there.
link |
Like right now you're harboring how many bacteria
link |
and the ones, many of them are parasites on you.
link |
And some of them are helpful.
link |
And some of them are modifying your behavior.
link |
And some of them are, you know, it's just really wild.
link |
But you know, this particular manifestation is unusual.
link |
You know, in the demographic, how it hit
link |
and the political response that it engendered
link |
and the healthcare response it engendered
link |
and the technology it engendered, it's kind of wild.
link |
Yeah, the communication on Twitter
link |
that it led to all that kind of stuff,
link |
at every single level, yeah.
link |
But what usually kills is life.
link |
The big extinctions are caused by meteors and volcanoes.
link |
That's the one you're worried about,
link |
as opposed to human created bombs that we launch.
link |
Solar flares are another good one.
link |
You know, occasionally solar flares hit the planet.
link |
Yeah, it's all pretty wild.
link |
On another historic moment, this is perhaps outside,
link |
but perhaps within your space of frameworks
link |
so you think about that just happened,
link |
I guess a couple of weeks ago is,
link |
I don't know if you're paying attention at all,
link |
it's the game stop and Wall Street bets.
link |
It's a lot of fun.
link |
So it's really fascinating.
link |
There's kind of a theme to this conversation today
link |
because it's like neural networks,
link |
it's cool how there's a large number of people
link |
in a distributed way, almost having a kind of fun,
link |
were able to take on the powerful elite hedge funds,
link |
centralized powers and overpower them.
link |
Do you have thoughts on this whole saga?
link |
I don't know enough about finance,
link |
but it was like the Elon, you know,
link |
Robin Hood guy when they talked.
link |
Yeah, what'd you think about that?
link |
Well, Robin Hood guy didn't know
link |
how the finance system worked.
link |
That was clear, right?
link |
He was treating like the people who settled
link |
the transactions as a black box.
link |
And suddenly somebody called him up
link |
and say, hey, black box calling you,
link |
your transaction volume means you need
link |
to put out $3 billion right now.
link |
And he's like, I don't have $3 billion.
link |
Like I don't even make any money on these trades.
link |
Why do I have $3 billion while you're sponsoring a trade?
link |
So there was a set of abstractions that,
link |
I don't think either, like now we understand it.
link |
Like this happens in chip design.
link |
Like you buy wafers from TSMC or Samsung or Intel.
link |
And they say it works like this
link |
and you do your design based on that.
link |
And then chip comes back and it doesn't work.
link |
And then suddenly you started having to open the black boxes.
link |
The transistors really work like they said,
link |
what's the real issue?
link |
So there's a whole set of things
link |
that created this opportunity and somebody spotted it.
link |
Now, people spot these kinds of opportunities all the time.
link |
So there's been flash crashes, there's been,
link |
there's always short squeezes that are fairly regular.
link |
Every CEO I know hates the shorts
link |
because they're manipulating,
link |
they're trying to manipulate their stock
link |
in a way that they make money and deprive value
link |
from both the company and the investors.
link |
So the fact that some of these stocks were so short,
link |
it's hilarious that this hasn't happened before.
link |
And I don't actually know why some serious hedge funds
link |
didn't do it to other hedge funds.
link |
And some of the hedge funds actually made a lot of money
link |
Yes, so my guess is we know 5% of what really happened
link |
and a lot of the players don't know what happened.
link |
And the people who probably made the most money
link |
aren't the people that they're talking about.
link |
Do you think there was something...
link |
I mean, this is the cool kind of Elon.
link |
You're the same kind of conversationalist,
link |
which is like first principles,
link |
questions of like what the hell happened.
link |
Just very basic questions of like,
link |
was there something shady going on?
link |
What, you know, who are the parties involved?
link |
It's the basic questions that everybody wants to know about.
link |
Yeah, so like we're in a very
link |
hyper competitive world, right?
link |
But transactions like buying and selling stock
link |
You know, I trust the company,
link |
representing themselves properly.
link |
You know, I bought the stock
link |
because I think it's gonna go up.
link |
I trust that the regulations are solid.
link |
Now, inside of that, there's all kinds of places
link |
where, you know, humans over trust.
link |
And, you know, this, this expose,
link |
let's say some weak points in the system.
link |
I don't know if it's gonna get corrected.
link |
I don't know if we have close to the real story.
link |
You know, my suspicion is we don't.
link |
And listening to that guy, he was like a little wide eyed
link |
about, and then he did this and then he did that.
link |
And I was like, I think you should know more
link |
about that spit your business than that.
link |
But again, there's many businesses when,
link |
like this layer is really stable.
link |
You stop paying attention to it.
link |
You pay attention to the stuff that's bugging you or new.
link |
You don't pay attention to the stuff
link |
that just seems to work all the time.
link |
You just, you know, the sky's blue every day, California.
link |
And we're once while, you know, it rains there.
link |
It was like, what do we do?
link |
Somebody go bring in the lawn furniture.
link |
You know, like it's getting wet.
link |
You don't know why it's getting wet.
link |
I was blue for like 100 days and now it's, you know, so.
link |
But part of the problem here with Vlad,
link |
the CEO of Robinhood is the scaling
link |
is that what we've been talking about
link |
is there's a lot of unexpected things
link |
that happen with the scaling.
link |
And you have to be, I think the scaling forces you
link |
to then return to the fundamentals.
link |
Well, it's interesting because when you buy
link |
and sell stocks, the scaling is, you know,
link |
the stocks don't only move in a certain range.
link |
And if you buy a stock, you can only lose that amount of money.
link |
On the short, short market, you can lose a lot more
link |
than you can benefit.
link |
Like it has a, it has a weird cause, you know,
link |
cost function or whatever the right word for that is.
link |
So he was trading in a market
link |
where he wasn't actually capitalized for the downside.
link |
If it got outside a certain range.
link |
Now, whether something that various has happened,
link |
I have no idea, but at some point the financial risk,
link |
both him and his customers was way outside
link |
of his financial capacity.
link |
And his understanding how the system work was clearly weak
link |
or he didn't represent himself.
link |
I don't know the person.
link |
When I listened to him, Nick,
link |
it could have been the surprise question was,
link |
like, how many of these guys called him?
link |
You know, it sounded like he was treating stuff
link |
as a black box, maybe he shouldn't have,
link |
but maybe he has a whole pile of experts
link |
somewhere else than it was going on.
link |
I don't, I don't know.
link |
Yeah, I mean, this is, this is one of the qualities
link |
of a good leader is under fire, you have to perform.
link |
And that means to think clearly and to speak clearly.
link |
And he dropped the ball on those things
link |
cause and understand the problem quickly,
link |
learn and understand the problem at like,
link |
at this like basic level, like what the hell happened.
link |
And my guess is, you know, at some level it was amateurs
link |
trading against, you know, experts slash insiders
link |
slash people with, you know, special information.
link |
Outsiders versus insiders.
link |
Yeah. And the insiders, you know,
link |
my guess is the next time this happens,
link |
we'll make money on it.
link |
The insiders always win.
link |
Well, they have more tools and more incentive.
link |
I mean, this always happens.
link |
Like the outsiders are doing this for fun.
link |
The insiders are doing this 24 stop.
link |
But there's numbers in the outsiders.
link |
This is the interesting thing.
link |
Well, there's numbers on the insiders too.
link |
Like that's different kind of numbers.
link |
But this could be a new era because, I don't know,
link |
at least I didn't expect that a bunch of Redditors could,
link |
you know, there's, you know, millions of people
link |
can get to the next one will be a surprise.
link |
But don't you think the crowd,
link |
the people are planning the next attack?
link |
But it has to be a surprise.
link |
Can't be the same game.
link |
And so the insiders, like it could be,
link |
there's a very large number of games to play
link |
and they can be agile about it.
link |
I don't know, I'm not an expert.
link |
Right. That's a good question.
link |
The space of games, how restricted is it?
link |
Yeah. And the system is so complicated,
link |
it could be relatively unrestricted.
link |
And also, like, you know,
link |
during the last couple of financial crashes,
link |
you know, what set it off was, you know,
link |
sets of derivative events where, you know,
link |
Nesim Talib's, you know, saying is,
link |
they're trying to lower volatility in the short run
link |
by creating tail events.
link |
And systems always evolve towards that.
link |
And then they always crash.
link |
Like S curve is the, you know,
link |
start low, ramp, plateau, crash.
link |
It's 100% effective.
link |
In the long run, let me ask you some advice
link |
to put on your profound hat.
link |
There's a bunch of young folks who listen to this thing
link |
for no good reason whatsoever.
link |
Undergraduate students, maybe high school students,
link |
maybe just young folks, young at heart,
link |
looking for the next steps to taking life.
link |
What advice would you give to a young person today
link |
about life, maybe career, but also life in general?
link |
Get good at some stuff.
link |
Well, get to know yourself, right?
link |
To get good at something that you're actually interested in.
link |
You have to love what you're doing to get good at it.
link |
You really got to find that.
link |
Don't waste all your time doing stuff
link |
that's just boring or bland or numbing, right?
link |
Don't let old people screw you.
link |
Well, people get talked into doing all kinds of shit
link |
and racking up huge student, you know, student deaths.
link |
And like, there's so much crap going on, you know?
link |
And they drains your time and drains your energy.
link |
Yeah, they are quite a sign, you know, thesis that,
link |
you know, the older generation won't let go.
link |
They're trapping all the young people.
link |
I think that's some truth to that.
link |
Just because you're old doesn't mean you stop thinking.
link |
I know lots of really original old people.
link |
I'm an old person.
link |
So, but you have to be conscious about it.
link |
You can fall into the ruts and then do that.
link |
I mean, when I hear young people spouting opinions,
link |
it sounds like they come from Fox News or CNN.
link |
I think they've been captured by group thinking, memes,
link |
I supposed to think on their own.
link |
You know, so if you find yourself repeating
link |
what everybody else is saying,
link |
you're not gonna have a good life.
link |
Like, that's not how the world works.
link |
It may be, it seems safe,
link |
but it puts you at great jeopardy for
link |
well being boring or unhappy.
link |
How long did it take you to find the thing
link |
that you have fun with?
link |
I've been a fun person since I was pretty little.
link |
I've gone through a couple of periods
link |
of depression in my life.
link |
Well, good reason or for a reason
link |
that doesn't make any sense.
link |
Like, some things are hard.
link |
Like, you go through mental transitions in high school.
link |
I was really depressed for a year.
link |
And I think I had my first midlife crisis at 26.
link |
I kind of thought, is this all there is?
link |
Like, I was working at a job that I loved
link |
and but I was going to work and all my time is consumed.
link |
What's the escape out of that depression?
link |
What's the answer to is, is this all there is?
link |
Well, a friend of mine, I asked him
link |
because he was working in his ass off.
link |
I said, what's your work life balance?
link |
Like, there's work, friends, family, personal time.
link |
Are you balancing in that?
link |
And he said, work 80%, family 20%.
link |
And I tried to find some time to sleep.
link |
Like, there's no personal time.
link |
There's no passionate time.
link |
Like, you know, the young people are often passionate
link |
So I was sort of like that.
link |
But you need to have some space in your life
link |
for different things.
link |
And that's, that creates, that makes you resistant
link |
to the whole, the deep dips into depression kind of thing.
link |
Yeah. Well, you have to get to know yourself too.
link |
Some physical, something physically intense helps.
link |
Like the weird places your mind goes kind of thing.
link |
Like, and why does it happen?
link |
Why do you do what you do?
link |
Like triggers, like the things that cause your mind
link |
to go to different places kind of thing.
link |
Or like events, like.
link |
You're upbringing for better or worse,
link |
whether your parents are great people or not,
link |
you come into adulthood with all kinds of emotional burdens.
link |
And you can see some people are so bloody stiff
link |
and restrained and they think, you know,
link |
the world's fundamentally negative.
link |
Like you maybe, you have unexplored territory.
link |
Or you're afraid of something.
link |
Definitely afraid of quite a few things.
link |
Then you got to go face them.
link |
Like what's the worst thing that can happen?
link |
You're going to die, right?
link |
Like that's inevitable.
link |
You might as well get over that, like a hundred percent.
link |
Like people are worried about the virus,
link |
but you know, the human condition is pretty deadly.
link |
There's something about embarrassment.
link |
That's, I've competed a lot in my life.
link |
And I think the, if I'm too introspected,
link |
the thing I'm most afraid of is being like humiliated.
link |
I think nobody cares about that.
link |
Like you're the only person on the planet
link |
that cares about you being humiliated.
link |
So it can really useless thought.
link |
It's like, you're all humiliated.
link |
Something happened in a room full of people
link |
and they walk out and they didn't think about it.
link |
Or maybe somebody told a funny story
link |
to somebody else and then it dissipated throughout.
link |
No, I know it too.
link |
I mean, I've been really embarrassed about shit
link |
that nobody cared about myself.
link |
It's a funny thing.
link |
So the worst thing ultimately is just, yeah.
link |
But that's a cage and then you have to get out of it.
link |
Like once you, here's the thing.
link |
Once you find something like that,
link |
you have to be determined to break it.
link |
Cause otherwise you'll just, you know,
link |
so you accumulate that kind of junk
link |
and then you die as a, you know, a mess.
link |
So the goal, I guess it's like a cage within a cage.
link |
I guess the goal is to die in the biggest possible cage.
link |
Well, ideally you'd have no cage.
link |
You know, people do get enlightened.
link |
There's a few out there?
link |
Of course there are.
link |
Either that or they have, you know,
link |
it's a great sales pitch.
link |
There's like enlightened people,
link |
write books and do all kinds of stuff.
link |
It's a good way to sell a book.
link |
I'll give you that.
link |
You've never met somebody you just thought,
link |
they just kill me.
link |
Like they just, like mental clarity, humor.
link |
No, 100%, but I just feel like they're living
link |
They have their own.
link |
You still think there's a cage?
link |
There's still a cage.
link |
You secretly suspect there's always a cage.
link |
There's no, there's nothing outside the universe.
link |
There's nothing outside the cage.
link |
You work, you work, you work at a bunch of companies.
link |
You work, you work at a bunch of companies.
link |
You led a lot of amazing teams.
link |
I don't, I'm not sure if you've ever been like
link |
at the early stages of a startup,
link |
but do you have advice for somebody that wants to
link |
do a startup or build a company,
link |
like build a strong team of engineers that are passionate.
link |
Just want to solve a big problem.
link |
Like, is there a more specifically on that point?
link |
You have to be really good at stuff.
link |
If you're going to lead and build a team,
link |
you better be really interested in how people work and think.
link |
The people or the solution to the problem.
link |
So there's two things, right?
link |
One is how people work and the other is the fund.
link |
Actually, there's quite a few successful startups
link |
that's really clear.
link |
The founders don't know anything about people.
link |
Like the idea was so powerful that it propelled them.
link |
But I suspect somewhere early,
link |
they hired some people who understood people
link |
because people really need a lot of care
link |
and feeding the collaborate and work together
link |
and feel engaged and work hard.
link |
Like startups are all about out producing other people.
link |
Like you're nimble because you don't have any legacy.
link |
You don't have a bunch of people who are depressed
link |
about life just showing up.
link |
So startups have a lot of advantages that way.
link |
Do you like the, Steve Jobs talked about this idea of A players
link |
I don't know if you know this formulation.
link |
Organizations that get taken over by B player leaders
link |
often really underperform their HRC players.
link |
That said, in big organizations,
link |
there's so much work to do.
link |
And there's so many people who are happy to do what,
link |
like the leadership or the big idea people
link |
would consider menial jobs.
link |
And you need a place for them,
link |
but you need an organization that both values and rewards them,
link |
but doesn't let them take over the leadership of it.
link |
So you need to have an organization that's resistant to that.
link |
But in the early days,
link |
the notion with Steve was that one B player in a room
link |
of A players will be destructive to the whole.
link |
I've seen that happen.
link |
I don't know if it's always true,
link |
like you run into people who are clearly B players,
link |
but they think they're A players.
link |
And so they have a loud voice at the table
link |
and they make lots of demands for that.
link |
But there's other people are like, I know I am.
link |
I just want to work with cool people on cool shit
link |
and just tell me what to do and I'll go get it done.
link |
So you have to, again, this is like people skills,
link |
like what kind of person is it?
link |
I've met some really great people I love working with.
link |
That weren't the biggest ID people,
link |
the most productive ever, but they show up,
link |
You know, they create connection and community
link |
that people value.
link |
It's pretty diverse.
link |
I don't think there's a recipe for that.
link |
I got to ask you about love.
link |
I heard you into this now.
link |
Into this love thing?
link |
Do you think this is your solution to your depression?
link |
No, I'm just trying to, like you said,
link |
to delight in people on occasion trying to sell a book.
link |
I'm writing a book about love.
link |
You're writing a book about love.
link |
I'm a friend of mine.
link |
Somebody said, you should really write a book
link |
about your management philosophy.
link |
He said, it'd be a short book.
link |
Well, that one was all pretty well.
link |
What role do you think love, family, friendship,
link |
all that kind of human stuff play in a successful life?
link |
You've been exceptionally successful in the space
link |
of like running teams, building cool shit in this world,
link |
creating some amazing things.
link |
What, did love get in the way?
link |
Did love help the family get in the way?
link |
Did family help friendship?
link |
You want the engineer's answer?
link |
So, but first love is functional, right?
link |
It's functional in what way?
link |
So we habituate ourselves to the environment.
link |
And actually Jordan told me,
link |
Jordan Peterson told me this line.
link |
So you go through life and you just get used to everything,
link |
except for the things you love.
link |
Like this is really useful for, you know,
link |
like other people's children and dogs and trees.
link |
You just don't pay that much attention to them.
link |
Your own kids, you're monitoring them really closely.
link |
Like, and if they go off a little bit,
link |
because you love them, if you're smart,
link |
if you're gonna be a successful parent,
link |
you notice it right away.
link |
You don't habituate just things you love.
link |
And if you wanna be successful at work,
link |
if you don't love it,
link |
you're not gonna put the time in somebody else.
link |
It's somebody else that loves it.
link |
Like, cause it's new and interesting
link |
and that lets you go to the next level.
link |
So it's a thing, it's just a function
link |
that generates newness and novelty
link |
and surprises, you know, all those kinds of things.
link |
It's really interesting.
link |
Like, and there's people figured out lots of, you know,
link |
frameworks for this, you know, like,
link |
like humans seem to go in partnership,
link |
go through, you know, interests.
link |
Like somebody, suddenly somebody's interesting
link |
and then you're infatuated with them
link |
and then you're in love with them.
link |
And then you, you know, different people have ideas
link |
about parental love or mature love.
link |
Like you go through a cycle of that,
link |
which keeps us together and it's, you know,
link |
super functional for creating families
link |
and creating communities
link |
and making you support somebody
link |
despite the fact that you don't love them.
link |
Like, and, and it can be really enriching.
link |
You know, no, no, in the work life balance scheme,
link |
if all you do is work,
link |
you think you may be optimizing your work potential,
link |
but if you don't love your work
link |
or you don't have family and friends
link |
and things you care about,
link |
your brain isn't well balanced.
link |
Like everybody knows experience
link |
of your work's on something all week.
link |
You went home and took two days off and you came back in.
link |
The odds of you working on the thing,
link |
you picking up right where you left off is zero.
link |
Your brain refactored it, but being in love is great.
link |
It's like change is the color of the light in the room.
link |
It creates a spaciousness that's, that's different.
link |
It helps you think, it makes you strong.
link |
Bukowski had this line about love being a fog
link |
that dissipates with the first light of reality
link |
That's depressing.
link |
I think it's the other way around.
link |
It lasts, well, like you said, it's just a function.
link |
It's a thing that generates.
link |
It can be the light that actually enlivens your world
link |
and creates the interest and the power and the strengths
link |
to go do something.
link |
It's like, that sounds like, you know,
link |
there's like physical love, emotional love,
link |
intellectual love, spiritual love, right?
link |
Isn't it all the same thing?
link |
You should differentiate that.
link |
Maybe that's your problem.
link |
In your book, you should refine that a little bit.
link |
The different chapters?
link |
Yeah, there's different chapters.
link |
What's the, what's, these are, aren't these just different
link |
layers of the same thing or the stack?
link |
People, people, some people are addicted to physical love
link |
and they have no idea about emotional or intellectual love.
link |
I don't know if they're the same things.
link |
I think they're different.
link |
They could be different.
link |
I'd be, I guess the ultimate goal is for it to be the same.
link |
Well, if you want something to be bigger and interesting,
link |
you should find all its components and differentiate them,
link |
not clown it together.
link |
People do this all the time.
link |
Get your abstraction layers right and then you can,
link |
you have room to breathe.
link |
Well, maybe you can write the forward to my book about love.
link |
Or the afterwards.
link |
I feel like Lex has made a lot of progress with this book.
link |
Well, you have things in your life that you love.
link |
And they are, you're right.
link |
And you can have multiple things with the same person or the same thing.
link |
Depending on the moment of the day.
link |
Like what Bacowski described is that moment you go from being in love
link |
to having a different kind of love.
link |
And that's a transition.
link |
But when it happens, if you'd read the owner's manual and you believed it,
link |
you would have said, oh, this happened.
link |
It doesn't mean it's not love.
link |
It's a different kind of love.
link |
But, but maybe there's something better about that is you grow old.
link |
If all you do is regret how you used to be.
link |
You should have learned a lot of things because like who you can be in your future
link |
self is actually more interesting and possibly delightful than, you know,
link |
being a mad kid in love with the next person.
link |
Like that's super fun when it happens.
link |
That's, that's, you know, 5% of the possibility.
link |
That there's a lot more fun to be had in the long lasting stuff.
link |
Or meaning, you know, if that's your thing.
link |
Meaning, which is a kind of fun.
link |
It's a deeper kind of fun.
link |
And it's surprising, you know, that's like, like the thing I like is surprises,
link |
you know, and you just never know what's going to happen.
link |
But you have to look carefully and you have to work at it.
link |
You have to think about it.
link |
You have to see the surprises when they happen, right?
link |
You have to be looking for it from the branching perspective.
link |
You mentioned regrets.
link |
Do you have regrets about your own trajectory?
link |
Some of it's painful, but you want to hear the painful stuff.
link |
I'd say like in terms of working with people, when people did say stuff I didn't like,
link |
especially if it was a bit nefarious, I took it personally.
link |
I also felt it was personal about them.
link |
But a lot of times, like humans are, you know, most humans are a mess, right?
link |
And then they act out and they do stuff.
link |
And this psychologist I heard a long time ago said, you tend to think somebody does something to you.
link |
But really what they're doing is they're doing what they're doing while they're in front of you.
link |
It's not that much about you.
link |
And as I got more interested in, you know, when I work with people,
link |
I think about them and probably analyze them and understand them a little bit.
link |
And then when they do stuff, I'm way less surprised.
link |
And I'm way, you know, and if it's bad, I'm way less hurt.
link |
And I react way less.
link |
Like I sort of expect everybody's got their shit.
link |
And it's not about you.
link |
It's not about me that much.
link |
It's like, you know, you do something and you think you're embarrassed, but nobody cares.
link |
Like somebody's really mad at you.
link |
The odds of it being about you.
link |
Because they're getting mad the way they're doing that because of some pattern they learned.
link |
And, you know, and maybe you can help them if you care enough about it.
link |
Or you could see it coming and step out of the way.
link |
Like, I wish I was way better at that.
link |
I'm a bit of a hothead.
link |
You said with Steve, that was a feature, not a bug.
link |
Well, he was using it as the counter for orderliness that would crush his work.
link |
Well, you were doing the same.
link |
I don't think my vision was big enough.
link |
It was more like I just got pissed off and did stuff.
link |
I'm sure that's the...
link |
I don't know if it had the...
link |
It didn't have the amazing effect of creating a trillion dollar company.
link |
It was more like I just got pissed off and left.
link |
And or made enemies that he shouldn't have.
link |
Like, I didn't really understand politics until I worked at Apple where, you know, Steve
link |
was a master player of politics and his staff had to be or they wouldn't survive them.
link |
And it was definitely part of the culture.
link |
And then I've been in companies where they say it's political, but it's all, you know,
link |
fun and games compared to Apple.
link |
And it's not that the people at Apple are bad people.
link |
It's just they operate politically at a higher level.
link |
You know, it's not like, oh, somebody said something bad about somebody, somebody else,
link |
which is most politics.
link |
It's, you know, they had strategies about accomplishing their goals.
link |
Sometimes, you know, over the dead bodies of their enemies, you know, with sophistication.
link |
Yeah, more game of thrones and sophistication and like a big time factor rather than a,
link |
Well, that requires a lot of control over your emotions, I think, to have a bigger strategy
link |
in the way you behave.
link |
And it's effective in the sense that coordinating thousands of people to do really hard things
link |
where many of the people in there don't understand themselves much less how they're participating
link |
creates all kinds of, you know, drama and problems that, you know, our solution is political
link |
Like, how do you convince people?
link |
How do you leverage them?
link |
How do you motivate them?
link |
How do you get rid of them?
link |
How, you know, like, there's, there's so many layers of that that are interesting.
link |
And even though some, some of it, let's say, may be tough, it's not evil.
link |
Unless, you know, you use that skill to evil purposes, which some people obviously do.
link |
But it's a skill set that operates.
link |
You know, and I wish I'd, you know, I was interested in it, but I, you know, it was sort of like,
link |
I'm an engineer, I do my thing.
link |
And, you know, there's, there's times when I could have had a way bigger impact if I,
link |
you know, knew how to, if I paid more attention and knew more about that.
link |
About the human layer of the stack.
link |
Yeah, that, that human political power, you know, expression layer of the stack.
link |
It's just complicated.
link |
And there's lots to know about it.
link |
I mean, people are good at it or just amazing.
link |
And when they're good at it and let's say relatively kind and oriented a good direction,
link |
you can really feel, you can get lots of stuff done and coordinate things that you never
link |
But all people like that also have some pretty hard edges because, you know, it's,
link |
it's a heavy lift.
link |
And I wish I'd spent more time with that when I was younger, but maybe I wasn't ready.
link |
You know, I was a wide eyed kid for 30 years.
link |
Still a bit of a kid.
link |
What do you hope your legacy is when there's a, when there's a book like a H Hikers guy
link |
And this is like a, one sentence entry about Jim Miller from like that guy lived at some
link |
There's not many, you know, not many people would be remembered.
link |
You're one of the sparkling little human creatures that had a big impact on the world.
link |
How do you hope you'll be remembered?
link |
My daughter was trying to get, she edited my Wikipedia page to say that I was a legend
link |
in the guru, but they took it out.
link |
So she put it back and she's 15.
link |
I think, I think that was probably the best part of my legacy.
link |
She got her sister and they were all excited.
link |
They were like trying to put it in the references because there's articles in that on the top.
link |
So in the eyes of your kids, you're a legend.
link |
Well, they're pretty skeptical because they don't be better than that.
link |
They're like, dad.
link |
So yeah, that's, that's super, that kind of stuff is super fun in terms of the big legend
link |
I don't really care.
link |
You're just an engineer.
link |
They've been thinking about building a big pyramid.
link |
So I had a debate with a friend about whether pyramids or craters are cooler.
link |
And he realized that there's craters everywhere, but you know, they built a couple pyramids
link |
And they remember you for a while.
link |
We're still talking about it.
link |
I think that would be cool.
link |
Those aren't easy to build.
link |
And they don't actually know how they built them, which is great.
link |
It's either a AGI or aliens could be involved.
link |
So I think, I think you're going to have to figure out quite a few more things than just
link |
the basics of civil engineering.
link |
So I guess you hope your legacy is pyramids.
link |
That would, that would be cool.
link |
And my Wikipedia page, you know, getting updated by my daughter periodically.
link |
Like those two things would pretty much make it.
link |
Jim, it's a huge honor talking to you again.
link |
I hope we talk many more times in the future.
link |
I can't wait to see what you do with TimeStorrent.
link |
I can't wait to use it.
link |
I can't wait for you to revolutionize yet another space in computing.
link |
It's a huge honor to talk to you.
link |
Thanks for talking today.
link |
Thanks for listening to this conversation with Jim Keller.
link |
And thank you to our sponsors, Athletic Greens, all in one nutrition drink, Brooklyn and
link |
Sheets, ExpressVPN, and Bell Campo grass fed meat.
link |
Click the sponsor links to get a discount and to support this podcast.
link |
And now let me leave you with some words from Alan Turing.
link |
Those who can imagine anything can create the impossible.
link |
Thank you for listening and hope to see you next time.