back to index

Chris Lattner: The Future of Computing and Programming Languages | Lex Fridman Podcast #131


small model | large model

link |
00:00:00.000
The following is a conversation with Chris Latner,
link |
00:00:02.640
his second time in the podcast.
link |
00:00:04.640
He's one of the most brilliant engineers
link |
00:00:06.600
in modern computing, having created
link |
00:00:08.760
LLVM compiler infrastructure project,
link |
00:00:11.480
the Clang compiler, the Swift programming language,
link |
00:00:14.640
a lot of key contributions to TensorFlow and TPUs
link |
00:00:17.640
as part of Google.
link |
00:00:19.040
He served as vice president of autopilot software at Tesla,
link |
00:00:23.520
was a software innovator and leader at Apple,
link |
00:00:26.200
and now is at SyFive as senior vice president
link |
00:00:29.360
of platform engineering,
link |
00:00:30.920
looking to revolutionize chip design
link |
00:00:33.520
to make it faster, better, and cheaper.
link |
00:00:36.600
Quick mention of each sponsor,
link |
00:00:38.240
followed by some thoughts related to the episode.
link |
00:00:40.960
First sponsor is Blinkist,
link |
00:00:42.440
an app that summarizes key ideas from thousands of books.
link |
00:00:45.440
I use it almost every day to learn new things
link |
00:00:48.040
or to pick which books I want to read or listen to next.
link |
00:00:52.320
Second is Neuro,
link |
00:00:53.960
the maker of functional sugar free gum and mints
link |
00:00:56.520
that I use to supercharge my mind
link |
00:00:58.560
with caffeine, Althianine, and B vitamins.
link |
00:01:01.640
Third is Masterclass,
link |
00:01:03.280
online courses from the best people in the world
link |
00:01:06.720
on each of the topics covered from rockets,
link |
00:01:09.280
to game design, to poker, to writing, and to guitar.
link |
00:01:13.960
And finally, Cash App,
link |
00:01:15.720
the app I use to send money to friends for food,
link |
00:01:18.560
drinks, and unfortunately lost bets.
link |
00:01:21.800
Please check out the sponsors in the description
link |
00:01:23.800
to get a discount and to support this podcast.
link |
00:01:27.360
As a side note, let me say that Chris
link |
00:01:29.360
has been an inspiration to me on a human level
link |
00:01:32.600
because he is so damn good as an engineer
link |
00:01:35.280
and leader of engineers and yet he's able to stay humble,
link |
00:01:38.640
especially humble enough to hear the voices
link |
00:01:41.080
of disagreement and to learn from them.
link |
00:01:43.840
He was supportive of me and this podcast
link |
00:01:46.120
from the early days and for that, I'm forever grateful.
link |
00:01:49.560
To be honest, most of my life,
link |
00:01:51.240
no one really believed that I would amount to much.
link |
00:01:53.960
So when another human being looks at me,
link |
00:01:56.560
it makes me feel like I might be someone special,
link |
00:01:58.960
it can be truly inspiring.
link |
00:02:00.880
That's a lesson for educators.
link |
00:02:02.800
The weird kid in the corner with a dream
link |
00:02:05.680
is someone who might need your love and support
link |
00:02:08.200
in order for that dream to flourish.
link |
00:02:10.960
If you enjoy this thing, subscribe on YouTube,
link |
00:02:13.360
review it with five stars on Apple Podcast,
link |
00:02:15.520
follow us on Spotify, support on Patreon,
link |
00:02:18.000
or connect with me on Twitter at Lex Freedman.
link |
00:02:21.360
And now, here's my conversation with Chris Latner.
link |
00:02:24.800
What are the strongest qualities of Steve Jobs,
link |
00:02:27.840
Elon Musk, and the great and powerful Jeff Dean
link |
00:02:31.840
since you've gotten the chance to work with each?
link |
00:02:34.840
You're starting with an easy question there.
link |
00:02:37.240
These are three very different people.
link |
00:02:39.680
I guess you could do maybe a pairwise comparison
link |
00:02:42.920
between them instead of a group comparison.
link |
00:02:44.880
So if you look at Steve Jobs and Elon,
link |
00:02:47.160
I worked a lot more with Elon than I did with Steve.
link |
00:02:49.880
They have a lot of commonality.
link |
00:02:51.360
They're both visionary in their own way.
link |
00:02:54.000
They're both very demanding in their own way.
link |
00:02:57.400
My sense is Steve is much more human factor focused,
link |
00:03:01.240
where Elon is more technology focused.
link |
00:03:03.440
What does human factor mean?
link |
00:03:04.800
Steve's trying to build things that feel good,
link |
00:03:07.240
that people love, that affect people's lives, how they live.
link |
00:03:10.400
He's looking into the future a little bit
link |
00:03:13.440
in terms of what people want.
link |
00:03:16.240
Where I think that Elon focuses more on
link |
00:03:18.680
learning how exponentials work
link |
00:03:20.400
and predicting the development of those.
link |
00:03:22.640
Steve worked a lot of engineers.
link |
00:03:24.880
That was one of the things that reading the biography
link |
00:03:27.480
and how can a designer essentially talk to engineers
link |
00:03:32.080
and get their respect?
link |
00:03:33.880
I think so I did not work very closely with Steve.
link |
00:03:36.480
I'm not an expert at all.
link |
00:03:37.480
My sense is that he pushed people really hard,
link |
00:03:40.480
but then when he got an explanation that made sense to him,
link |
00:03:43.080
then he'd let go.
link |
00:03:44.640
And he did actually have a lot of respect for engineering,
link |
00:03:48.440
but he also knew when to push.
link |
00:03:50.440
And when you can read people well,
link |
00:03:52.840
you can know when they're holding back
link |
00:03:55.640
and when you can get a little bit more out of them.
link |
00:03:57.240
And I think he was very good at that.
link |
00:03:59.640
I mean, if you compare the other folks,
link |
00:04:02.240
so Jeff Dean, right?
link |
00:04:04.040
Jeff Dean's an amazing guy.
link |
00:04:05.040
He's super smart, as are the other guys.
link |
00:04:09.240
Jeff is a really, really, really nice guy.
link |
00:04:11.840
Well meaning, he's a classic Googler.
link |
00:04:14.040
He wants people to be happy.
link |
00:04:16.640
He combines it with brilliance
link |
00:04:18.440
so he can pull people together in a really great way.
link |
00:04:21.440
He's definitely not a CEO type.
link |
00:04:23.440
I don't think he would even want to be that.
link |
00:04:26.440
If he's still programs.
link |
00:04:28.440
Oh yeah, he definitely programs.
link |
00:04:29.440
Jeff is an amazing engineer today.
link |
00:04:31.440
And that has never changed.
link |
00:04:33.440
It's really hard to compare Jeff to either of those two.
link |
00:04:37.440
I think that Jeff leads through technology
link |
00:04:42.440
and building it himself
link |
00:04:43.440
and then pulling people in and inspiring them.
link |
00:04:45.440
And so I think that that's one of the amazing things about Jeff.
link |
00:04:49.440
But each of these people, with their pros and cons,
link |
00:04:52.440
all are really inspirational
link |
00:04:54.440
and have achieved amazing things.
link |
00:04:56.440
I've been very fortunate to get to work with these guys.
link |
00:05:00.440
For yourself, you've led large teams,
link |
00:05:03.440
you've done so many incredible,
link |
00:05:05.440
difficult technical challenges.
link |
00:05:07.440
Is there something you've picked up from them
link |
00:05:10.440
about how to lead?
link |
00:05:12.440
Yeah, I mean, I think leadership is really hard.
link |
00:05:14.440
It really depends on what you're looking for there.
link |
00:05:16.440
I think you really need to know what you're talking about.
link |
00:05:19.440
So being grounded on the product, on the technology,
link |
00:05:22.440
on the business, on the mission is really important.
link |
00:05:26.440
Being understanding what people are looking for,
link |
00:05:29.440
why they're there.
link |
00:05:30.440
One of the most amazing things about Tesla is
link |
00:05:32.440
the unifying vision.
link |
00:05:34.440
People are there because they believe in clean energy
link |
00:05:36.440
and electrification and all these kinds of things.
link |
00:05:41.440
The other is to understand what really motivates people,
link |
00:05:44.440
how to get the best people,
link |
00:05:45.440
how to build a plan that actually can be executed.
link |
00:05:48.440
There's so many different aspects of leadership
link |
00:05:50.440
and it really depends on the time, the place, the problems.
link |
00:05:54.440
There's a lot of issues that don't need to be solved
link |
00:05:56.440
and so if you focus on the right things and prioritize well,
link |
00:05:59.440
that can really help move things.
link |
00:06:01.440
Two interesting things you mentioned.
link |
00:06:03.440
One is you really have to know what you're talking about.
link |
00:06:06.440
You've worked on a lot of very challenging technical things.
link |
00:06:11.440
So I kind of assume you were born technically savvy,
link |
00:06:17.440
but assuming that's not the case,
link |
00:06:20.440
how did you develop technical expertise?
link |
00:06:24.440
Even at Google, you worked on, I don't know, how many projects,
link |
00:06:28.440
but really challenging, very varied.
link |
00:06:31.440
Compilers, TPUs, hardware, cloud stuff,
link |
00:06:34.440
a bunch of different things.
link |
00:06:36.440
The thing that I've become more comfortable with
link |
00:06:39.440
as I've gained experience is being okay with not knowing
link |
00:06:45.440
and so a major part of leadership is actually,
link |
00:06:48.440
it's not about having the right answer,
link |
00:06:50.440
it's about getting the right answer
link |
00:06:52.440
and so if you're working in a team of amazing people
link |
00:06:55.440
and many of these places, many of these companies
link |
00:06:58.440
all have amazing people,
link |
00:07:00.440
it's the question of how do you get people together,
link |
00:07:02.440
how do you build trust, how do you get people to open up,
link |
00:07:05.440
how do you get people to be vulnerable sometimes
link |
00:07:09.440
with an idea that maybe isn't good enough,
link |
00:07:11.440
but it's the start of something beautiful,
link |
00:07:13.440
how do you provide an environment where you're not just
link |
00:07:17.440
like top down, don't do the thing that I tell you to do,
link |
00:07:20.440
but you're encouraging people to be part of the solution
link |
00:07:23.440
and providing a safe space where if you're not doing the right thing,
link |
00:07:27.440
they're willing to tell you about it.
link |
00:07:29.440
So you're asking dumb questions?
link |
00:07:31.440
Yeah, dumb questions are my specialty.
link |
00:07:33.440
So I've been in the harbor room recently
link |
00:07:35.440
and I don't know much at all about how chips are designed,
link |
00:07:38.440
I know a lot about using them, I know some of the principles
link |
00:07:41.440
and the arse technical level of this,
link |
00:07:43.440
but it turns out that if you ask a lot of dumb questions,
link |
00:07:47.440
you get smarter really quick
link |
00:07:49.440
and when you're surrounded by people that want to teach
link |
00:07:51.440
and learn themselves, it can be a beautiful thing.
link |
00:07:55.440
So let's talk about programming languages if it's okay.
link |
00:07:58.440
At the highest absurd philosophical level.
link |
00:08:01.440
Don't get romantic on me, Lex.
link |
00:08:03.440
I will forever get romantic and torture you.
link |
00:08:07.440
I apologize.
link |
00:08:09.440
Why do programming languages even matter?
link |
00:08:13.440
Okay, well, thank you very much.
link |
00:08:15.440
So you're saying why should you care about any one programming language
link |
00:08:18.440
or why do we care about programming computers?
link |
00:08:20.440
No, why do we care about programming language design,
link |
00:08:24.440
creating effective programming languages,
link |
00:08:29.440
choosing a, you know, one programming languages
link |
00:08:32.440
versus another programming language,
link |
00:08:34.440
why we keep struggling and improving
link |
00:08:37.440
through the evolution of these programming languages?
link |
00:08:39.440
Sure, sure, sure.
link |
00:08:40.440
Okay, so I mean, I think you have to come back
link |
00:08:41.440
to what are we trying to do here, right?
link |
00:08:43.440
So we have these beasts called computers
link |
00:08:46.440
that are very good at specific kinds of things
link |
00:08:48.440
and we think it's useful to have them do it for us, right?
link |
00:08:51.440
Now, you have this question of how best to express that
link |
00:08:55.440
because you have a human brain still that has an idea in its head
link |
00:08:58.440
and you want to achieve something, right?
link |
00:09:00.440
So, well, there's lots of ways of doing this.
link |
00:09:03.440
You can go directly to the machine and speak assembly language
link |
00:09:06.440
and then you can express directly what the computer understands.
link |
00:09:09.440
That's fine.
link |
00:09:10.440
You can then have higher and higher and higher levels of abstraction
link |
00:09:13.440
up until machine learning
link |
00:09:15.440
and you're designing a neural net to do the work for you.
link |
00:09:17.440
The question is where along this way do you want to stop
link |
00:09:20.440
and what benefits do you get out of doing so?
link |
00:09:23.440
And so, programming languages in general, you have C,
link |
00:09:25.440
you have Fortran and Java and Ada, Pascal, Swift,
link |
00:09:30.440
you have lots of different things.
link |
00:09:32.440
They all have different tradeoffs
link |
00:09:34.440
and they're tackling different parts of the problems.
link |
00:09:36.440
Now, one of the things that most programming languages do
link |
00:09:39.440
is they're trying to make it so that you have pretty basic things
link |
00:09:42.440
like portability across different hardware.
link |
00:09:44.440
So you've got, I'm going to run on an Intel PC,
link |
00:09:47.440
I'm going to run on a RISC 5 PC, I'm going to run on an ARM phone
link |
00:09:51.440
or something like that, fine.
link |
00:09:53.440
I want to write one program and have it portable
link |
00:09:55.440
and this is something the assembly doesn't do.
link |
00:09:57.440
Now, when you start looking at the space of programming languages,
link |
00:10:00.440
this is where I think it's fun
link |
00:10:02.440
because programming languages all have tradeoffs
link |
00:10:05.440
and most people will walk up to them
link |
00:10:07.440
and they look at the surface level of syntax
link |
00:10:10.440
and say, oh, I like curly braces or I like tabs
link |
00:10:13.440
or I like, you know, semicolons or not or whatever, right?
link |
00:10:16.440
It's subjective, fairly subjective, very shallow things.
link |
00:10:20.440
But programming languages, when done right,
link |
00:10:22.440
can actually be very powerful
link |
00:10:24.440
and the benefit they bring is expression.
link |
00:10:29.440
Okay, and if you look at programming languages,
link |
00:10:32.440
there's really kind of two different levels to them.
link |
00:10:34.440
One is the down in the dirt, nuts and bolts,
link |
00:10:37.440
how do you get the computer to be efficient, stuff like that,
link |
00:10:39.440
how they work, type systems, compiler stuff, things like that.
link |
00:10:42.440
The other is the UI and the UI for programming language
link |
00:10:46.440
is really a design problem
link |
00:10:48.440
and a lot of people don't think about it that way.
link |
00:10:50.440
And the UI, you mean all that stuff with the braces and...
link |
00:10:53.440
Yeah, all that stuff's the UI and what it is
link |
00:10:55.440
and UI means user interface
link |
00:10:57.440
and so what's really going on
link |
00:10:59.440
is it's the interface between the guts and the human
link |
00:11:03.440
and humans are hard, right?
link |
00:11:05.440
Humans have feelings, they have things they like,
link |
00:11:08.440
they have things they don't like
link |
00:11:10.440
and a lot of people treat programming languages as though
link |
00:11:13.440
humans are just kind of abstract creatures that cannot be predicted
link |
00:11:17.440
but it turns out that actually there is better and worse.
link |
00:11:21.440
People can tell when a programming language is good
link |
00:11:24.440
or when it was an accident, right?
link |
00:11:26.440
And one of the things with Swift in particular
link |
00:11:29.440
is that a tremendous amount of time
link |
00:11:31.440
by a tremendous number of people
link |
00:11:33.440
have been put into really polishing and making it feel good
link |
00:11:36.440
but it also has really good nuts and bolts underneath it.
link |
00:11:39.440
You said that Swift makes a lot of people feel good.
link |
00:11:42.440
How do you get to that point?
link |
00:11:45.440
So how do you predict that tens of thousands,
link |
00:11:51.440
hundreds of thousands of people are going to enjoy
link |
00:11:53.440
using the user experience of this programming language?
link |
00:11:57.440
Well, you can look at it in terms of better and worse, right?
link |
00:11:59.440
So if you have to write lots of boilerplate
link |
00:12:01.440
or something like that, you will feel unproductive
link |
00:12:03.440
and so that's a bad thing.
link |
00:12:05.440
You can look at it in terms of safety.
link |
00:12:07.440
Safety, for example, is what's called a memory unsafe language
link |
00:12:10.440
and so you get dangling pointers and you get all these kind of bugs
link |
00:12:13.440
that then you have spent tons of time debugging
link |
00:12:15.440
and it's a real pain in the butt and you feel unproductive
link |
00:12:18.440
and so by subtracting these things from the experience
link |
00:12:20.440
you get happier people.
link |
00:12:23.440
But again, keep interrupting.
link |
00:12:25.440
I'm sorry.
link |
00:12:27.440
It's so hard to deal with.
link |
00:12:29.440
If you look at the people that are most productive on Stack Overflow
link |
00:12:34.440
they have a set of priorities that may not always correlate perfectly
link |
00:12:39.440
with the experience of the majority of users.
link |
00:12:42.440
If you look at the most upvoted, quote unquote,
link |
00:12:46.440
correct answer on Stack Overflow,
link |
00:12:48.440
it usually really prioritizes like safe code,
link |
00:12:56.440
proper code, stable code, that kind of stuff
link |
00:13:01.440
as opposed to if I want to use go to statements in my basic,
link |
00:13:06.440
I want to use go to statements.
link |
00:13:09.440
What if 99% of people want to use go to statements
link |
00:13:12.440
or use completely improper unsafe syntax?
link |
00:13:16.440
I don't think that people actually,
link |
00:13:17.440
if you boil it down and you get below the surface level
link |
00:13:19.440
people don't actually care about go tos or if statements or things like this.
link |
00:13:23.440
They care about achieving a goal.
link |
00:13:26.440
So the real question is I want to set up a web server
link |
00:13:29.440
and I want to do a thing and whatever.
link |
00:13:32.440
How quickly can I achieve that?
link |
00:13:34.440
From a programming language perspective
link |
00:13:36.440
there's really two things that matter there.
link |
00:13:38.440
One is what libraries exist
link |
00:13:41.440
and then how quickly can you put it together
link |
00:13:44.440
and what are the tools around that look like?
link |
00:13:47.440
And when you want to build a library that's missing,
link |
00:13:49.440
what do you do?
link |
00:13:51.440
This is where you see huge divergence in the force between worlds.
link |
00:13:55.440
So you look at Python, for example.
link |
00:13:57.440
Python is really good at assembling things
link |
00:13:59.440
but it's not so great at building all the libraries.
link |
00:14:02.440
And so you get, because of performance reasons,
link |
00:14:04.440
other things like this,
link |
00:14:05.440
is you get Python layered on top of C, for example.
link |
00:14:09.440
And that means that doing certain kinds of things,
link |
00:14:11.440
well, it doesn't really make sense to do in Python.
link |
00:14:13.440
Instead you do it in C and then you wrap it
link |
00:14:15.440
and then you have, you're living in two worlds
link |
00:14:17.440
and two worlds never is really great
link |
00:14:19.440
because tooling and the debugger doesn't work right
link |
00:14:21.440
and like all these kinds of things.
link |
00:14:23.440
Can you clarify a little bit what you mean by
link |
00:14:26.440
Python is not good at building libraries,
link |
00:14:28.440
meaning it doesn't make a conducive...
link |
00:14:30.440
Certain kinds of libraries.
link |
00:14:31.440
No, but it's just the actual meaning of the sentence.
link |
00:14:34.440
Meaning like it's not conducive to developers
link |
00:14:38.440
to come in and add libraries
link |
00:14:40.440
or is it the duality of the...
link |
00:14:44.440
it's a dance between Python and C and...
link |
00:14:47.440
Well, so Python's amazing.
link |
00:14:49.440
Python's a great language.
link |
00:14:50.440
I do not mean to say that Python is bad for libraries.
link |
00:14:53.440
What I meant to say is there are libraries that Python's really good at
link |
00:14:58.440
that you can write in Python,
link |
00:15:00.440
but there are other things like if you want to build a machine learning framework,
link |
00:15:03.440
you're not going to build a machine learning framework in Python
link |
00:15:05.440
because of performance, for example,
link |
00:15:07.440
or you want GPU acceleration or things like this.
link |
00:15:10.440
Instead what you do is you write a bunch of C or C++ code or something like that
link |
00:15:15.440
and then you talk to it from Python.
link |
00:15:18.440
And so this is because of decisions that were made in the Python design
link |
00:15:22.440
and those decisions have other counterbalancing forces.
link |
00:15:26.440
But the trick when you start looking at this from a programming language perspective
link |
00:15:30.440
is you start to say, okay, cool,
link |
00:15:32.440
how do I build this catalog of libraries that are really powerful?
link |
00:15:37.440
And how do I make it so that then they can be assembled into ways that feel good
link |
00:15:41.440
and they generally work the first time?
link |
00:15:43.440
Because when you're talking about building a thing,
link |
00:15:46.440
you have to include the debugging, the fixing, the turnaround cycle,
link |
00:15:50.440
the development cycle, all that kind of stuff into the process of building the thing.
link |
00:15:55.440
It's not just about pounding out the code.
link |
00:15:57.440
And so this is where things like catching bugs at compile time is valuable, for example.
link |
00:16:03.440
But if you dive into the details in this,
link |
00:16:07.440
Swift, for example, has certain things like value semantics,
link |
00:16:10.440
which is this fancy way of saying that when you treat a variable like a value,
link |
00:16:17.440
it acts like a mathematical object would.
link |
00:16:21.440
So you have used PyTorch a little bit.
link |
00:16:24.440
In PyTorch, you have tensors.
link |
00:16:26.440
Tensors are n dimensional grid of numbers, very simple.
link |
00:16:31.440
You can do plus and other operators on them.
link |
00:16:34.440
It's all totally fine.
link |
00:16:35.440
But why do you need to clone a tensor sometimes?
link |
00:16:38.440
Have you ever run into that?
link |
00:16:40.440
Yeah.
link |
00:16:41.440
Okay.
link |
00:16:42.440
And so why is that?
link |
00:16:43.440
Why do you need to clone a tensor?
link |
00:16:44.440
It's the usual object thing that's in Python.
link |
00:16:46.440
Yeah.
link |
00:16:47.440
So in Python, and just like with Java and many other languages,
link |
00:16:50.440
this isn't unique to Python.
link |
00:16:51.440
In Python, it has a thing called reference semantics,
link |
00:16:53.440
which is the nerdy way of explaining this.
link |
00:16:55.440
And what that means is you actually have a pointer do a thing instead of the thing.
link |
00:17:00.440
Okay.
link |
00:17:01.440
Now, this is due to a bunch of implementation details that you don't want to go into.
link |
00:17:06.440
But in Swift, you have this thing called value semantics.
link |
00:17:09.440
And so when you have a tensor in Swift, it is a value.
link |
00:17:12.440
If you copy it, it looks like you have a unique copy.
link |
00:17:15.440
If you change one of those copies, then it doesn't update the other one because you
link |
00:17:19.440
just made a copy of this thing.
link |
00:17:21.440
So that's like highly error prone in at least computer science math centric disciplines
link |
00:17:28.440
about Python.
link |
00:17:30.440
That like the thing you would expect to behave.
link |
00:17:34.440
Like math.
link |
00:17:35.440
Like math.
link |
00:17:36.440
It doesn't behave like math.
link |
00:17:38.440
And in fact, quietly it doesn't behave like math and then can ruin the entirety of your
link |
00:17:43.440
math.
link |
00:17:44.440
It puts you in debugging land again.
link |
00:17:46.440
Now, you just want to get something done and you're like, wait a second.
link |
00:17:49.440
Where do I need to put clone?
link |
00:17:51.440
And what level of the stack, which is very complicated, which I thought I was reusing
link |
00:17:55.440
somebody's library and now I need to understand it to know where to clone a thing.
link |
00:17:59.440
And hard to debug, by the way.
link |
00:18:01.440
Exactly.
link |
00:18:02.440
And so this is where programming languages really matter.
link |
00:18:04.440
And so in Swift, having value semantics so that both you get the benefit of math working
link |
00:18:10.440
like math.
link |
00:18:11.440
Right, but also the efficiency that comes with certain advantages there, certain implementation
link |
00:18:16.440
details there really benefit you as a programmer.
link |
00:18:18.440
Right.
link |
00:18:19.440
And clarify the values.
link |
00:18:20.440
Manics.
link |
00:18:21.440
Like how do you know that a thing should be treated like a value?
link |
00:18:23.440
Yeah.
link |
00:18:24.440
So, so Swift has a pretty strong culture and good language support for defining values.
link |
00:18:30.440
And so if you have an array.
link |
00:18:31.440
So tensors are one example that the machine learning folks are very used to.
link |
00:18:36.440
Just think about arrays.
link |
00:18:37.440
Same thing where you have an array.
link |
00:18:39.440
You put, you create an array, you put two or three or four things into it, and then you
link |
00:18:44.440
pass it off to another function.
link |
00:18:46.440
What happens if that function adds some more things to it?
link |
00:18:51.440
Well, you'll see it on the side that you pass it in, right?
link |
00:18:54.440
This is called reference semantics.
link |
00:18:56.440
Now, what if you pass an array off to a function?
link |
00:19:01.440
It scrolls it away in some dictionary or some other data structure somewhere.
link |
00:19:04.440
Right.
link |
00:19:05.440
Well, it thought that you just handed it that array.
link |
00:19:07.440
But then you return back and that reference to that array still exists in the caller.
link |
00:19:12.440
And they go and put more stuff in it.
link |
00:19:14.440
Right.
link |
00:19:15.440
The person you handed it off to may have thought they had the only reference to that.
link |
00:19:20.440
And so they didn't know what they, that this was going to change underneath the covers.
link |
00:19:23.440
And so this is where you end up having to do clone.
link |
00:19:26.440
So like, I was passed a thing.
link |
00:19:27.440
I'm not sure if I have the only version of it.
link |
00:19:30.440
So now I have to clone it.
link |
00:19:32.440
So what value semantics does is it allows you to say, hey, I have a, so in Swift, it defaults to value semantics.
link |
00:19:38.440
Also defaults to value semantics.
link |
00:19:40.440
And then because most things should end up being like this, then it makes sense for that to be the default.
link |
00:19:46.440
And one of the important things about that is that arrays and dictionaries and all these other collections that are aggregations of other things also have value semantics.
link |
00:19:53.440
And so when you pass this around to different parts of your program, you don't have to do these defensive copies.
link |
00:19:59.440
And so this is, this is great for two sides.
link |
00:20:01.440
It's great because you define away the bug, which is a big deal for productivity than the number one thing most people care about.
link |
00:20:08.440
But it's also good for performance because when you're doing a clone, so you pass the array down to the thing.
link |
00:20:13.440
It was like, I don't know if anybody else has it.
link |
00:20:15.440
I have to clone it.
link |
00:20:16.440
Well, you just did a copy of a bunch of data.
link |
00:20:18.440
It could be big.
link |
00:20:19.440
And then it could be the thing that called you is not keeping track of the old thing.
link |
00:20:23.440
So you just made a copy of it and you may not have had to.
link |
00:20:27.440
And so the way the value semantics work is in Swift is that it uses this thing called copy on right, which means that you get, you get the benefit of safety and performance.
link |
00:20:36.440
And it has another special trick because if you think certain languages like Java, for example, they have immutable strings.
link |
00:20:43.440
And so what they're trying to do is they provide value semantics by having pure immutability.
link |
00:20:48.440
Functional languages have pure immutability in lots of different places and this provides a much safer model and it provides value semantics.
link |
00:20:55.440
The problem with this is if you have immutability, everything is expensive.
link |
00:20:59.440
Everything requires a copy.
link |
00:21:01.440
For example, in Java, if you have a string X and a string Y, you append them together, we have to allocate a new string to hold X, Y.
link |
00:21:11.440
Oh, if they're immutable.
link |
00:21:13.440
Well, and strings and Java are immutable.
link |
00:21:16.440
And if there's there's optimizations for short runs and it's complicated, but but generally think about them as a separate allocation.
link |
00:21:24.440
And so when you append them together, you have to go allocate a third thing because somebody might have a point or two either of the other ones, right?
link |
00:21:31.440
And you can't go change them.
link |
00:21:32.440
So you have to go allocate a third thing because of the beauty of how the Swift value semantics system works out.
link |
00:21:37.440
If you have a string in Swift, you say, hey, put in X, right?
link |
00:21:40.440
And they say, append on YZW.
link |
00:21:43.440
It knows that there's only one reference to that and so can do an in place update.
link |
00:21:49.440
And so you're not allocating tons of stuff on the side, you're not you don't have all this problems when you pass it off.
link |
00:21:55.440
You can know you have the only reference.
link |
00:21:57.440
If you pass it off to multiple different people, but nobody changes it, they can all share the same thing.
link |
00:22:02.440
So you get a lot of the benefit of purely immutable design.
link |
00:22:05.440
And so you get a really nice sweet spot that I haven't seen in other languages.
link |
00:22:09.440
Yeah, that's interesting.
link |
00:22:10.440
I thought I thought there was going to be a philosophical like narrative here that you're going to have to pay a cost for it.
link |
00:22:19.440
Because it sounds like I think value semantics is beneficial for easing of debugging or minimizing the risk of errors, like bringing the errors closer to the source.
link |
00:22:35.440
Bringing the symptom of the air closer to the source of the air, however you say that.
link |
00:22:40.440
But you're saying there's not a performance cost either if you implement correctly.
link |
00:22:46.440
Well, so there's tradeoffs with everything.
link |
00:22:48.440
And so if you are doing very low level stuff, then sometimes you can notice cost.
link |
00:22:53.440
But then what you're doing is you're saying, what is the right default?
link |
00:22:56.440
So coming back to user interface, when you talk about programming languages, one of the major things that Swift does that makes people love it.
link |
00:23:04.440
That is not obvious when it comes to designing a language is this UI principle of progressive disclosure of complexity.
link |
00:23:13.440
So Swift, like many languages, is very powerful.
link |
00:23:16.440
The question is, when do you have to learn the power as a user?
link |
00:23:20.440
So Swift, like Python, allows you to start with like print hello world, right?
link |
00:23:24.440
Certain other languages start with like public static void main class, like all the ceremony, right?
link |
00:23:31.440
And so you go to teach a new person.
link |
00:23:34.440
Hey, welcome to this new thing.
link |
00:23:36.440
Let's talk about public access control classes.
link |
00:23:40.440
What's that?
link |
00:23:41.440
String system.out.println.
link |
00:23:43.440
Like packages.
link |
00:23:45.440
Right.
link |
00:23:46.440
And so instead, if you take this and you say, hey, we need packages, you know, modules.
link |
00:23:51.440
We need powerful things like classes.
link |
00:23:54.440
We need data structures.
link |
00:23:55.440
We need like all these things.
link |
00:23:57.440
The question is, how do you factor the complexity and how do you make it so that the normal case scenario is you're dealing with things that work the right way and the right way and give you good performance by default.
link |
00:24:09.440
But then as a power user, if you want to dive down to it, you have full cc performance, full control over low level pointers.
link |
00:24:16.440
You can call malloc if you want to call malloc.
link |
00:24:18.440
This is not recommended on the first page of every tutorial, but it's actually really important when you want to get work done.
link |
00:24:23.440
Right. And so being able to have that is really the design and program language design.
link |
00:24:28.440
And design is really, really hard.
link |
00:24:31.440
It's something that I think a lot of people kind of outside of UI, again, a lot of people just think is subjective.
link |
00:24:39.440
Like there's nothing, you know, it's just like curly braces or whatever.
link |
00:24:43.440
It's just like somebody's preference.
link |
00:24:45.440
But actually good design is something that you can feel.
link |
00:24:48.440
And how many people are involved with good design?
link |
00:24:51.440
So if we look at Swift, but look at historically, I mean, this might touch like, it's almost like a Steve Jobs question too.
link |
00:24:59.440
Like how much dictatorial decision making is required versus collaborative.
link |
00:25:07.440
And we'll talk about how all that can go wrong or right.
link |
00:25:11.440
Yeah, we'll Swift.
link |
00:25:12.440
So I can't speak to in general all designed everywhere.
link |
00:25:15.440
So the way it works with Swift is that there's a core team.
link |
00:25:19.440
And so core team is six or seven people ish something like that that is people that have been working with Swift since very early days.
link |
00:25:26.440
And so by early days is not that long ago.
link |
00:25:29.440
Okay. Yeah.
link |
00:25:30.440
So it's it became public in 2014.
link |
00:25:33.440
So it's been six years public now.
link |
00:25:35.440
But but so that's enough time that there's a story there.
link |
00:25:39.440
And there's mistakes have been made that then get fixed and you learn something.
link |
00:25:43.440
And then you, you know, and so what the core team does is it provides continuity.
link |
00:25:48.440
And so you want to have a, okay, well, there's a big hole that we want to fill.
link |
00:25:53.440
We know we want to fill it.
link |
00:25:55.440
So don't do other things that invade that space until we fill the hole, right?
link |
00:25:59.440
There, there's a boulder that's missing here.
link |
00:26:01.440
We want to, we will do that boulder, even though it's not today.
link |
00:26:04.440
Keep, keep out of that space.
link |
00:26:06.440
And the whole team remembers of the, remembers the myth of the boulder that's there.
link |
00:26:11.440
Yeah. Yeah.
link |
00:26:12.440
There's a general sense of what the future looks like in broad strokes and a shared understanding of that.
link |
00:26:16.440
Combined with the shared understanding of what has happened in the past that worked out well and didn't work out well.
link |
00:26:21.440
The next level out is you have the, what's called the swift evolution community.
link |
00:26:25.440
And you've got in that case, hundreds of people that really care passionately about the ways swift evolves.
link |
00:26:30.440
And that's like an amazing thing to again, the core team doesn't necessarily need to come up with all the good ideas.
link |
00:26:36.440
You got hundreds of people out there that care about something and they come up with really good ideas too.
link |
00:26:40.440
And that provides this like tumbling rock tumbler for ideas.
link |
00:26:44.440
And so the, the evolution process is, you know, a lot of people in a discourse forum,
link |
00:26:49.440
they're like hatching it out and trying to like talk about, okay, well, should we go left or right?
link |
00:26:53.440
Or if we did this, what would be good?
link |
00:26:55.440
And, you know, here you're talking about hundreds of people.
link |
00:26:57.440
So you're not going to get consensus necessarily.
link |
00:26:59.440
They're not obvious consensus.
link |
00:27:01.440
And so there's a proposal process that then allows the core team and the community to work this out.
link |
00:27:08.440
And what the core team does is it aims to get consensus out of the community and provide guardrails,
link |
00:27:14.440
but also provide long term, make sure we're going the right direction kind of things.
link |
00:27:20.440
So does that group represent like the, how much people will love the user interface?
link |
00:27:27.440
Like, do you think they're able to capture that?
link |
00:27:29.440
Well, I mean, it's something we talk about a lot.
link |
00:27:31.440
It's something we care about.
link |
00:27:32.440
How will we, how will we do that for debate?
link |
00:27:34.440
But I think that we've done pretty well so far.
link |
00:27:36.440
Is the beginner in mind?
link |
00:27:38.440
Because you said the progressive disclosure.
link |
00:27:40.440
Yeah.
link |
00:27:41.440
So we care a lot about, a lot about that, a lot about power, a lot about efficiency,
link |
00:27:46.440
a lot about there are many factors to good design.
link |
00:27:48.440
And you have to figure out a way to kind of work your way through that.
link |
00:27:53.440
So if you like think about like the language I love is Lisp,
link |
00:27:57.440
probably still because I use Emacs,
link |
00:27:59.440
but I haven't done anything any serious working lists,
link |
00:28:02.440
but it has a ridiculous amount of parentheses.
link |
00:28:05.440
I've also, you know, with Java and C++, the braces, you know,
link |
00:28:14.440
I like, I enjoyed the comfort of being between braces, you know,
link |
00:28:20.440
and then Python is really sorry to interrupt, just like,
link |
00:28:23.440
and last thing to me as a designer, if I was a language designer, God forbid,
link |
00:28:28.440
as I would be very surprised that Python with no braces would nevertheless somehow
link |
00:28:36.440
be comforting also.
link |
00:28:38.440
So like, I can see arguments for all of these.
link |
00:28:40.440
But look at this.
link |
00:28:41.440
This is evidence that it's not about braces versus tabs.
link |
00:28:44.440
Right.
link |
00:28:45.440
Exactly.
link |
00:28:46.440
You're good.
link |
00:28:47.440
It's a good point.
link |
00:28:48.440
Right.
link |
00:28:49.440
So like, you know, there's, there's evidence.
link |
00:28:50.440
But see, like it's one of the most argued about things.
link |
00:28:52.440
Oh yeah, of course.
link |
00:28:53.440
Just like tabs and spaces, which it doesn't, I mean,
link |
00:28:55.440
there's one obvious right answer, but it doesn't, it doesn't actually matter.
link |
00:28:59.440
What's that?
link |
00:29:00.440
Come on, friends.
link |
00:29:01.440
Like, come on.
link |
00:29:02.440
What are you trying to do to me here?
link |
00:29:03.440
People are going to, yeah, half the people are going to tune out.
link |
00:29:05.440
Yeah.
link |
00:29:06.440
So, so these two, you're able to identify things that don't really matter for the
link |
00:29:11.440
experience.
link |
00:29:12.440
Well, no, no, it's always a really hard, so the easy decisions are easy.
link |
00:29:16.440
Right.
link |
00:29:17.440
I mean, you can find those are not the interesting ones.
link |
00:29:19.440
The hard ones are the ones that are most interesting.
link |
00:29:21.440
Right.
link |
00:29:22.440
So there's a lot of places where, Hey, we want to do a thing.
link |
00:29:25.440
Everybody agrees we should do it.
link |
00:29:27.440
There's one proposal on the table, but it has all these bad things associated with
link |
00:29:31.440
it.
link |
00:29:32.440
Well, okay, what are we going to do about that?
link |
00:29:34.440
Do we just take it?
link |
00:29:35.440
Do we delay it?
link |
00:29:36.440
Do we say, Hey, well, maybe there's this other feature that if we do that first,
link |
00:29:39.440
this will work out better.
link |
00:29:41.440
How does this, if we do this, are we paying ourselves into a corner?
link |
00:29:45.440
Right.
link |
00:29:46.440
And so this is where, again, you're having that core team of people that has some
link |
00:29:50.440
opportunity and has perspective, has some of the historical understanding is really
link |
00:29:54.440
valuable because you get, it's not just like one brain, you get the power of multiple
link |
00:29:58.440
people coming together to make good decisions and then you get the best out of all these
link |
00:30:02.440
people.
link |
00:30:03.440
And you also can harness the community around it.
link |
00:30:06.440
What about like the decision of whether like in Python, having one type or having,
link |
00:30:12.440
you know, strict typing?
link |
00:30:14.440
Yeah, okay.
link |
00:30:15.440
Yeah, let's talk about this.
link |
00:30:16.440
So I like how you put that, by the way.
link |
00:30:19.440
So many people would say that Python doesn't have types.
link |
00:30:21.440
Doesn't have types.
link |
00:30:22.440
Yeah.
link |
00:30:23.440
But you're right.
link |
00:30:24.440
I've listened to you enough to where I'm a fan of yours and I've listened to way too
link |
00:30:29.440
many podcasts and videos of you talking about this.
link |
00:30:32.440
Oh yeah.
link |
00:30:33.440
So I would argue that Python has one type.
link |
00:30:35.440
And so, so like when you import Python and Swift, which by the way works really well,
link |
00:30:39.440
you have everything comes in as a Python object.
link |
00:30:42.440
Now here they're trade offs because, you know, it depends on where you're optimizing for
link |
00:30:47.440
and Python is a super successful language for a really good reason.
link |
00:30:50.440
Because it has one type, you get duck typing for free and things like this.
link |
00:30:55.440
But also you're pushing, you're making it very easy to pound out code on one hand, but
link |
00:31:00.440
you're also making it very easy to introduce complicated bugs.
link |
00:31:04.440
You have to debug and you pass the string into something that expects an integer and
link |
00:31:08.440
it doesn't immediately die.
link |
00:31:10.440
It goes all the way down the stack trace and you find yourself in the middle of some code
link |
00:31:13.440
that you really didn't want to know anything about and it blows up and you're just saying,
link |
00:31:16.440
well, what did I do wrong?
link |
00:31:17.440
Right.
link |
00:31:18.440
And so types are good and bad and they have trade offs are good for performance and certain
link |
00:31:22.440
other things, depending on where you're coming from, but it's all about trade offs.
link |
00:31:26.440
And so this is, this is what design is, right?
link |
00:31:28.440
Design is about weighing trade offs and trying to understand the ramifications of the things
link |
00:31:33.440
that you're weighing, like types or not, or one type or many types.
link |
00:31:38.440
But also within many types, how powerful do you make that type system is another very
link |
00:31:42.440
complicated question with lots of trade offs.
link |
00:31:46.440
It's very interesting, by the way, but, but that's like one, one dimension.
link |
00:31:51.440
And there's a bunch of other dimensions, JIT compiled versus static compiled garbage
link |
00:31:55.440
collected versus reference counted versus memory, manual memory management versus, you
link |
00:32:00.440
know, like in like all these different trade offs and how you balance them or what make
link |
00:32:04.440
the program language good.
link |
00:32:05.440
Good currency.
link |
00:32:06.440
Yep.
link |
00:32:07.440
So in all those things, I guess when you're designing the language, you also have to
link |
00:32:11.440
think of how that's going to get all compiled down to.
link |
00:32:14.440
If you care about performance.
link |
00:32:16.440
Yeah.
link |
00:32:17.440
Well, and go back to list, right?
link |
00:32:18.440
So list also, I would say JavaScript is another example of a very simple language, right?
link |
00:32:23.440
And so one of the, so I also love Lisp.
link |
00:32:26.440
I don't use it as much as maybe you do or you did.
link |
00:32:29.440
No, I think we're both everyone who loves Lisp.
link |
00:32:32.440
It's like, you love, it's like, I don't know, I love Frank Sinatra, but like how often do
link |
00:32:37.440
I seriously listen to Frank Sinatra?
link |
00:32:39.440
Sure.
link |
00:32:40.440
But, but, but you look at that or you look at JavaScript, which is another very different,
link |
00:32:44.440
but relatively simple language.
link |
00:32:46.440
And there's certain things that don't exist in the language, but there's, there is inherent
link |
00:32:50.440
complexity to the problems that we're trying to model.
link |
00:32:52.440
And so what happens to the complexity?
link |
00:32:54.440
In the case of both of them, for example, you say, well, what about large scale software
link |
00:32:59.440
development?
link |
00:33:00.440
Okay.
link |
00:33:01.440
Well, you need something like packages.
link |
00:33:02.440
Neither language has a like language affordance for packages.
link |
00:33:05.440
And so what you get is patterns.
link |
00:33:07.440
You get things like NPN.
link |
00:33:08.440
You get things like, you know, like these ecosystems that get built around.
link |
00:33:12.440
And I'm a believer that if you don't model at least the most important inherent complexity
link |
00:33:17.440
in the language, then what ends up happening is that complexity gets pushed elsewhere.
link |
00:33:22.440
And when it gets pushed elsewhere, sometimes that's great because often building things
link |
00:33:26.440
as libraries is very flexible and very powerful and allows you to evolve and things like that.
link |
00:33:30.440
But often it leads to a lot of unnecessary divergence in the force and fragmentation.
link |
00:33:35.440
And, and when that happens, you just get kind of a mess.
link |
00:33:39.440
And so the question is, how do you, how do you balance that?
link |
00:33:42.440
Don't put too much stuff in the language because that's really expensive and makes
link |
00:33:45.440
things complicated.
link |
00:33:46.440
But how do you model enough of the inherent complexity of the problem that you provide
link |
00:33:51.440
the framework and the structure for people to think about?
link |
00:33:54.440
Well, so, so the, the, the key thing to think about with, with programming languages and
link |
00:33:59.440
you think about what a programming language is there for is it's about making a human
link |
00:34:03.440
more productive.
link |
00:34:04.440
So like there's an old, I think it's Steve Jobs quote about it's a bicycle for the mind.
link |
00:34:09.440
Right.
link |
00:34:10.440
You can, you can, you can definitely walk, but you'll get there a lot faster if you can
link |
00:34:16.440
bicycle on your way.
link |
00:34:17.440
And a programming language is a bicycle for the mind.
link |
00:34:20.440
Yeah.
link |
00:34:21.440
Basically.
link |
00:34:22.440
Wow.
link |
00:34:23.440
That's a really interesting way to think about it.
link |
00:34:24.440
By, by raising the level of abstraction now, you can fit more things in your head.
link |
00:34:27.440
By being able to just directly leverage somebody's library, you can now get something done quickly.
link |
00:34:33.440
In the case of Swift, Swift UI is this new framework that Apple has released recently
link |
00:34:37.440
for doing UI programming.
link |
00:34:39.440
And it has this declarative programming model, which defines away entire classes of bugs.
link |
00:34:45.440
It's made, it builds on value semantics and many other nice Swift things.
link |
00:34:48.440
And what this does is allows you to just get way more done with way less code.
link |
00:34:53.440
And now your productivity as a developer is much higher.
link |
00:34:56.440
Right.
link |
00:34:57.440
And so that, that's really what programming languages should be about is it's not about
link |
00:35:00.440
tabs versus spaces or curly braces or whatever.
link |
00:35:02.440
It's about how productive do you make the person.
link |
00:35:05.440
And you can only see that when you have libraries that were built with the right intention that
link |
00:35:11.440
the language was designed for.
link |
00:35:13.440
And with Swift, I think we're still a little bit early.
link |
00:35:16.440
But Swift UI and many other things that are coming out now are really showing that.
link |
00:35:20.440
And I think that they're opening people's eyes.
link |
00:35:22.440
It's kind of interesting to think about like how that, you know, the knowledge of something,
link |
00:35:29.440
of how good the bicycle is, how people learn about that.
link |
00:35:33.440
You know, so I've used C++.
link |
00:35:35.440
Now this is not going to be a trash talking session about C++, but use C++ for a really
link |
00:35:41.440
long time.
link |
00:35:42.440
I think you're there if you want.
link |
00:35:43.440
I have the scars.
link |
00:35:45.440
I feel like I spent many years without realizing like there's languages that could, for my
link |
00:35:52.440
particular life style, brain style, thinking style, there's languages that could make me
link |
00:35:58.440
more productive in the debugging stage, in the just the development stage and thinking
link |
00:36:04.440
like the bicycle for the mind that I could fit more stuff into my.
link |
00:36:07.440
Python's a great example of that, right?
link |
00:36:09.440
I mean, a machine learning framework in Python is a great example of that.
link |
00:36:12.440
It's just very high abstraction level.
link |
00:36:14.440
And so you can be thinking about things on a like very high level algorithmic level.
link |
00:36:18.440
Instead of thinking about, okay, well, am I copying this tensor to a GPU or not?
link |
00:36:22.440
Right?
link |
00:36:23.440
It's not, it's not what you want to be thinking about.
link |
00:36:25.440
And as I was telling you, I mean, I guess the question I had is, you know, how does a
link |
00:36:30.440
person like me or in general people discover more productive, you know, languages?
link |
00:36:36.440
Like how I was, as I've been telling you offline, I've been looking for like a project
link |
00:36:41.440
to work on in Swift so I can really try it out.
link |
00:36:45.440
I mean, my intuition was like doing a Hello World is not going to get me there to get
link |
00:36:51.440
me to experience the power of language.
link |
00:36:53.440
You need a few weeks to change your metabolism.
link |
00:36:55.440
Exactly.
link |
00:36:56.440
I think that's beautifully put.
link |
00:36:59.440
That's one of the problems with people with diets.
link |
00:37:01.440
Like I'm actually currently to go in parallel, but in a small tangent is I've been recently
link |
00:37:07.440
eating only meat.
link |
00:37:09.440
Okay.
link |
00:37:10.440
And okay.
link |
00:37:11.440
So most people are like, I think that's horribly unhealthy or whatever.
link |
00:37:16.440
You have like a million, whatever the science is, it just doesn't sound right.
link |
00:37:22.440
Well, so back when I was in college, we did the Atkins diet.
link |
00:37:25.440
That was, that was a thing.
link |
00:37:26.440
And similar.
link |
00:37:27.440
And but if you, you have to always give these things a chance.
link |
00:37:30.440
I mean, I was dieting, I was not dieting, but it's just the things that you like.
link |
00:37:35.440
If I eat personally, if I eat meat, just everything, I could be super focused or more focused than
link |
00:37:41.440
usual.
link |
00:37:42.440
I just feel great.
link |
00:37:44.440
I've been running a lot of, you know, doing pushups and posts and so on.
link |
00:37:47.440
And you Python is similar in that sense for me.
link |
00:37:50.440
Where are you going with this?
link |
00:37:53.440
I mean, literally, I just felt ahead like a stupid smile on my face when I first started
link |
00:37:59.440
using Python.
link |
00:38:00.440
I could code up really quick things.
link |
00:38:03.440
Like I would see the world.
link |
00:38:05.440
I'll be empowered to write a script to, to, you know, to do some basic data processing,
link |
00:38:11.440
to rename files on my computer.
link |
00:38:13.440
Yeah.
link |
00:38:14.440
Right.
link |
00:38:15.440
And like Pearl didn't do that for me a little bit.
link |
00:38:19.440
And again, like none of these are about which, which is best or something like that.
link |
00:38:23.440
But there's definitely better and worse here.
link |
00:38:25.440
But it clicks.
link |
00:38:26.440
Well, yeah.
link |
00:38:27.440
And if you, if you look at Pearl, for example, you get bogged down and scalers versus arrays
link |
00:38:32.440
versus hashes versus type globs and like all that kind of stuff.
link |
00:38:35.440
And, and Python's like, yeah, let's not do this.
link |
00:38:38.440
Right.
link |
00:38:39.440
And some of it is debugging.
link |
00:38:40.440
Like everyone has different priorities.
link |
00:38:41.440
But for me, it's, can I create systems for myself that empower me to debug quickly?
link |
00:38:47.440
Like I've always been a big fan, even just crewing like asserts, like always stating
link |
00:38:55.440
things that should be true, which in Python I found in myself do more because of type,
link |
00:39:00.440
all these kinds of stuff.
link |
00:39:02.440
Well, you could think of types in a program language as being kind of assert.
link |
00:39:05.440
Yeah.
link |
00:39:06.440
They could check to compile time.
link |
00:39:08.440
Right.
link |
00:39:09.440
So how do you learn a new thing?
link |
00:39:11.440
Well, so this, or how do, how do people learn new things?
link |
00:39:13.440
Right.
link |
00:39:14.440
This, this is hard.
link |
00:39:15.440
People don't like to change.
link |
00:39:16.440
People generally don't like change around them either.
link |
00:39:19.440
And so we're all very slow to adapt and change.
link |
00:39:23.440
And usually there's a catalyst that's required to, to force yourself over the, over, over
link |
00:39:27.440
this.
link |
00:39:28.440
So for learning a programming language, it really comes down to finding an excuse, like
link |
00:39:33.440
build a thing that that's that the language is actually good for that the ecosystem is
link |
00:39:37.440
ready for.
link |
00:39:38.440
And so, and so if you were to write an iOS app, for example, that'd be the easy case.
link |
00:39:44.440
Obviously you would use Swift for that.
link |
00:39:46.440
Right.
link |
00:39:47.440
Android.
link |
00:39:48.440
So Swift runs on Android.
link |
00:39:50.440
Oh, does it?
link |
00:39:51.440
Oh, yeah.
link |
00:39:52.440
Yeah.
link |
00:39:53.440
Swift runs in lots of places.
link |
00:39:54.440
How does that work?
link |
00:39:55.440
Okay.
link |
00:39:56.440
So Swift, Swift is built on top of LLVM.
link |
00:39:58.440
Yeah.
link |
00:39:59.440
LLVM runs everywhere.
link |
00:40:00.440
LLVM, for example, bills the Android kernel.
link |
00:40:03.440
Oh, okay.
link |
00:40:04.440
So yeah.
link |
00:40:05.440
Okay.
link |
00:40:06.440
They realize this.
link |
00:40:07.440
Yeah.
link |
00:40:08.440
So Swift, Swift is very portable, runs on Windows.
link |
00:40:10.440
There's, it runs on lots of different things.
link |
00:40:12.440
And Swift, sorry to interrupt that.
link |
00:40:14.440
Swift UI.
link |
00:40:15.440
And then there's a thing called UIKit.
link |
00:40:17.440
So can I build an app with Swift?
link |
00:40:20.440
Well, so that's the thing is the ecosystem is what matters there.
link |
00:40:24.440
So Swift UI and UIKit are Apple technologies.
link |
00:40:27.440
Okay, got it.
link |
00:40:28.440
And so they happen to like Swift UI happens to be written in Swift, but it's an Apple
link |
00:40:31.440
proprietary framework that Apple loves and wants to keep on its platform, which makes
link |
00:40:36.440
total sense.
link |
00:40:37.440
You go to Android and you don't have that library.
link |
00:40:39.440
Yeah.
link |
00:40:40.440
So Android has a different ecosystem of things that hasn't been built out and doesn't
link |
00:40:44.440
work as well with Swift.
link |
00:40:45.440
And so you can totally use Swift to do like arithmetic and things like this, but building
link |
00:40:50.440
a UI with Swift on Android is not a great experience right now.
link |
00:40:54.440
So if I wanted to learn Swift, what's the, I mean, the one practical different version
link |
00:41:01.440
of that is Swift for TensorFlow, for example.
link |
00:41:05.440
And one of the inspiring things for me with both TensorFlow and PyTorch is how quickly
link |
00:41:11.440
the community can like switch from different libraries.
link |
00:41:14.440
Yep.
link |
00:41:15.440
Like you could see some of the communities switching to PyTorch now, but it's very easy
link |
00:41:21.440
to see.
link |
00:41:22.440
And then TensorFlow is really stepping up its game.
link |
00:41:24.440
And then there's no reason why.
link |
00:41:26.440
I think the way it works is basically it has to be one GitHub repo, like one paper steps
link |
00:41:30.440
up.
link |
00:41:31.440
It gets people excited.
link |
00:41:32.440
And they're like, ah, I have to learn this.
link |
00:41:35.440
Swift for what?
link |
00:41:37.440
Swift again.
link |
00:41:38.440
Yeah.
link |
00:41:39.440
And then they learn and they fall in love with them.
link |
00:41:41.440
I mean, that's what happened with PyTorch.
link |
00:41:42.440
There has to be a reason, a catalyst.
link |
00:41:44.440
Yeah.
link |
00:41:45.440
And so, and, and there, I mean, people don't like change, but it turns out that once you've
link |
00:41:50.440
worked with one or two programming languages, the basics are pretty similar.
link |
00:41:54.440
And so one of the fun things about learning programming languages, even, even maybe less,
link |
00:41:58.440
I don't know if you agree with this is that when you start doing that, you start learning
link |
00:42:01.440
new things.
link |
00:42:02.440
Because you have a new way to do things and you're forced to do them.
link |
00:42:06.440
And that forces you to explore and to put you in learning mode.
link |
00:42:09.440
And when you get in learning mode, your mind kind of opens a little bit and you can, you
link |
00:42:13.440
can see things in a new way, even when you go back to the old place.
link |
00:42:16.440
Right.
link |
00:42:17.440
Yeah.
link |
00:42:18.440
So it would list with functional stuff.
link |
00:42:20.440
But I wish there was a kind of window.
link |
00:42:23.440
Maybe you can tell me if there is, there you go.
link |
00:42:25.440
This is a question to ask what is the most beautiful feature in a programming language.
link |
00:42:30.440
Before I ask it, let me say like with Python, I remember I saw list comprehensions.
link |
00:42:36.440
Yeah.
link |
00:42:37.440
It was like when I like really took it in.
link |
00:42:40.440
Yeah.
link |
00:42:41.440
I don't know.
link |
00:42:42.440
I just loved it.
link |
00:42:43.440
It was like fun to do.
link |
00:42:45.440
Like it was fun to do that kind of something about it to be able to filter through a list
link |
00:42:52.440
and to create a new list on a single line was elegant.
link |
00:42:56.440
I could all get into my head.
link |
00:42:58.440
It just made me fall in love with the language.
link |
00:43:01.440
Yep.
link |
00:43:02.440
So is there, let me ask you a question.
link |
00:43:04.440
Is there, what do you use the most beautiful feature in a programming languages that you've
link |
00:43:10.440
ever encountered in Swift maybe and then outside of Swift?
link |
00:43:14.440
I think the thing that I like the most from a programming language.
link |
00:43:18.440
So, so I think the thing you have to think about with the programming language, again,
link |
00:43:22.440
what is the goal?
link |
00:43:23.440
You're trying to get people to get things done quickly.
link |
00:43:26.440
And so you need libraries, you need high quality libraries, and then you need a user base around
link |
00:43:32.440
them that can assemble them and do cool things with them.
link |
00:43:34.440
Right.
link |
00:43:35.440
And so to me, the question is what enables high quality libraries?
link |
00:43:38.440
Yeah.
link |
00:43:39.440
And there's a huge divide in the world between libraries who enable high quality libraries
link |
00:43:48.440
versus the ones that put special stuff in the language.
link |
00:43:52.440
So, programming languages that enable high quality libraries, got it.
link |
00:43:57.440
So, and what I mean by that is expressive libraries that then feel like a natural integrated
link |
00:44:03.440
part of the language itself.
link |
00:44:05.440
So, an example of this in Swift is that int and float and also ray and string, things
link |
00:44:11.440
like this, these are all part of the library.
link |
00:44:13.440
Like int is not hard coded into Swift.
link |
00:44:17.440
And so what that means is that because int is just a library thing defined in the standard
link |
00:44:22.440
library, along with strings and rays and all the other things that come with the standard
link |
00:44:25.440
library.
link |
00:44:26.440
Well, hopefully you do like int, but anything that any language features that you needed
link |
00:44:32.440
to define int, you can also use in your own types.
link |
00:44:35.440
So, if you wanted to find a quaternion or something like this, right?
link |
00:44:41.440
Well, it doesn't come in the standard library.
link |
00:44:43.440
There's a very special set of people that care a lot about this, but those people are
link |
00:44:48.440
also important.
link |
00:44:49.440
It's not about classism, right?
link |
00:44:51.440
It's not about the people who care about instant fluids are more important than the people
link |
00:44:54.440
care about quaternions.
link |
00:44:55.440
And so to me, the beautiful things about programming languages is when you allow those communities
link |
00:45:00.440
to build high quality libraries that feel native, that feel like they're built into the compiler
link |
00:45:05.440
without having to be.
link |
00:45:07.440
What does it mean for the int to be part of not hard coded in?
link |
00:45:13.440
So what is an int?
link |
00:45:18.440
Int is just an integer.
link |
00:45:20.440
In this case, it's like a 64 bit integer or something like this.
link |
00:45:24.440
But so the 64 bit is hard coded or no?
link |
00:45:28.440
No, none of that's hard coded.
link |
00:45:29.440
So int, if you go look at how it's implemented, it's just a struct in Swift.
link |
00:45:34.440
And so it's a struct, and then how do you add two structs?
link |
00:45:37.440
Well, you define plus.
link |
00:45:38.440
And so you can define plus on int.
link |
00:45:41.440
Well, you can define plus on your thing too.
link |
00:45:43.440
You can define int as an odd method or something like that on it.
link |
00:45:47.440
And so, yeah, you can add methods on things.
link |
00:45:50.440
Yeah.
link |
00:45:51.440
So you can define operators like how it behaves.
link |
00:45:55.440
That's used beautiful when there's something about the language which enables others to
link |
00:46:00.440
create libraries which are not hacky.
link |
00:46:05.440
Yeah, they feel native.
link |
00:46:07.440
And so one of the best examples of this is Lisp.
link |
00:46:10.440
Right?
link |
00:46:11.440
Because in Lisp, all the libraries are basically part of the language.
link |
00:46:15.440
You write term rewrite systems and things like this.
link |
00:46:18.440
Can you, as a counter example, provide what makes it difficult to write a library that's native?
link |
00:46:23.440
Is it the Python C?
link |
00:46:25.440
Well, so one example, I'll give you two examples, Java and C++ or Java and C.
link |
00:46:32.440
They both allow you to define your own types.
link |
00:46:35.440
But int is hard coded in the language.
link |
00:46:38.440
Okay.
link |
00:46:39.440
Well, why?
link |
00:46:40.440
Well, in Java, for example, coming back to this whole reference semantic value, semantic
link |
00:46:43.440
thing, int gets passed around by value.
link |
00:46:48.440
Yeah.
link |
00:46:49.440
But if you make like a pair or something like that, a complex number, right?
link |
00:46:54.440
It's a class in Java and now it gets passed around by reference, by pointer.
link |
00:46:59.440
And so now you lose value semantics, right?
link |
00:47:02.440
You lost math.
link |
00:47:03.440
Okay.
link |
00:47:04.440
Well, that's not great, right?
link |
00:47:06.440
If you can do something with int, why can't I do it with my type?
link |
00:47:09.440
Yeah.
link |
00:47:10.440
Right?
link |
00:47:11.440
So that's the negative side of the thing I find beautiful is when you can solve that,
link |
00:47:17.440
when you can have full expressivity where you as a user of the language have as much
link |
00:47:22.440
or almost as much power as the people who implemented all the standard built in stuff.
link |
00:47:27.440
Because what that enables is that enables truly beautiful libraries.
link |
00:47:31.440
You know, it's kind of weird because I've gotten used to that.
link |
00:47:36.440
That's one, I guess, other aspect of program language design.
link |
00:47:39.440
You have to think, you know, the old first principles thinking, like, why are we doing
link |
00:47:44.440
it this way?
link |
00:47:45.440
By the way, I mean, I remember because I was thinking about the Waller's operator and
link |
00:47:51.440
I'll ask you about it later, but it hit me that like the equal sign for assignment.
link |
00:47:57.440
Yeah.
link |
00:47:58.440
Like, why are we using the equal sign?
link |
00:48:00.440
It's wrong.
link |
00:48:01.440
And that's not the only solution, right?
link |
00:48:04.440
So if you look at Pascal, they use colon equals for assignment and equals for equality.
link |
00:48:11.440
And they use like less than greater than instead of the not equal.
link |
00:48:14.440
Yeah.
link |
00:48:15.440
Like there are other answers here.
link |
00:48:16.440
So, but like, and yeah, I'd like to ask you all, but how do you then decide to break convention
link |
00:48:24.440
to say, you know what?
link |
00:48:27.440
Everybody's doing it wrong.
link |
00:48:29.440
We're going to do it right.
link |
00:48:31.440
Yeah.
link |
00:48:32.440
So it's like an ROI, like return on investment trade off, right?
link |
00:48:35.440
So if you do something weird, let's just say like not like colon equal instead of equal
link |
00:48:40.440
for assignment, that would be weird with today's aesthetic, right?
link |
00:48:44.440
And so you'd say, cool, this is theoretically better, but is it better in which ways?
link |
00:48:49.440
Like, what do I get out of that?
link |
00:48:50.440
Do I define away class of bugs?
link |
00:48:52.440
Well, one of the class of bugs that C has is that you can use like, you know, if X equals
link |
00:48:57.440
without equals equals, if X equals Y, right?
link |
00:49:01.440
Well, turns out you can solve that problem in lots of ways.
link |
00:49:05.440
Clang, for example, GCC, all these compilers will detect that as a, as a likely bug, produce
link |
00:49:10.440
a warning.
link |
00:49:11.440
Do they?
link |
00:49:12.440
Yeah.
link |
00:49:13.440
GCC didn't.
link |
00:49:14.440
And it's like, one of the important things about programming language design is like,
link |
00:49:19.440
you're literally creating suffering in the world.
link |
00:49:23.440
Okay.
link |
00:49:24.440
Like, I feel like, I mean, one way to see it is the bicycle for the mind, but the other
link |
00:49:29.440
way is to like, minimizing suffering.
link |
00:49:31.440
Well, you have to decide if it's worth it, right?
link |
00:49:33.440
And so let's go back to that.
link |
00:49:35.440
Okay.
link |
00:49:36.440
But, but if you, if you look at this, and again, this is where there's a lot of detail
link |
00:49:39.440
that goes into each of these things, equal and C returns a value.
link |
00:49:45.440
Yeah.
link |
00:49:46.440
That's messed up.
link |
00:49:48.440
That allows you say X equals Y equals Z, like that works in C.
link |
00:49:52.440
Yeah.
link |
00:49:53.440
Is it messed up?
link |
00:49:54.440
You know, well, so that most people think it's messed up, I think it is very by messed
link |
00:49:59.440
up.
link |
00:50:00.440
What I mean is it is very rarely used for good.
link |
00:50:03.440
And it's often used for bugs.
link |
00:50:05.440
Yeah.
link |
00:50:06.440
Right.
link |
00:50:07.440
That's a good definition of messed up.
link |
00:50:09.440
Yeah.
link |
00:50:10.440
You could use, you know, it's a, in hindsight, this was not such a great idea.
link |
00:50:13.440
Right.
link |
00:50:14.440
Now, one of the things with Swift that is really powerful and one of the reasons it's actually
link |
00:50:17.440
good versus it being full of good ideas is that when, when we launched Swift one, we
link |
00:50:24.440
announced that it was public, people could use it, people could build apps, but it was
link |
00:50:28.440
going to change and break.
link |
00:50:30.440
Okay.
link |
00:50:31.440
When Swift two came out, we said, Hey, it's open source and there's this open process
link |
00:50:34.440
which people can help evolve and direct the language.
link |
00:50:38.000
So the community at large, like Swift users can now help shape the language as it is.
link |
00:50:43.440
And what happened is that part, as part of that process is a lot of really bad mistakes
link |
00:50:47.840
got taken out.
link |
00:50:49.680
So for example, Swift used to have the C style plus plus and minus minus operators.
link |
00:50:54.440
Like, what does it mean when you put it before versus after?
link |
00:50:58.440
Right.
link |
00:50:59.440
Well, that got cargo culted from C into Swift early on.
link |
00:51:02.440
What's cargo culted?
link |
00:51:03.440
Cargo culted means brought forward without really considering it.
link |
00:51:07.440
Okay.
link |
00:51:08.440
This is maybe not the most PC term.
link |
00:51:11.440
But I have to look it up in urban dictionary.
link |
00:51:13.440
Yeah.
link |
00:51:14.440
Yeah.
link |
00:51:15.440
So it got pulled into C without, or it got pulled into Swift without very good consideration.
link |
00:51:20.840
And we went through this process and one of the first things got ripped out was plus
link |
00:51:24.720
plus and minus minus because they lead to confusion.
link |
00:51:27.800
They have very little value over saying, you know, X plus equals one and X plus equals
link |
00:51:32.560
one is way more clear.
link |
00:51:34.360
And so when you're optimizing for teachability and clarity and bugs and this multidimensional
link |
00:51:38.960
space that you're looking at, things like that really matter.
link |
00:51:42.400
And so being first principles on where you're coming from and what you're trying to achieve
link |
00:51:46.400
and being anchored on the objective is really important.
link |
00:51:49.680
Well, let me ask you about the most sort of this podcast isn't about information, it's
link |
00:51:58.200
about drama.
link |
00:51:59.200
Okay.
link |
00:52:00.200
Let me talk to you about some drama.
link |
00:52:01.200
So you mentioned Pascal and colon equals, there's something that's called the walrus
link |
00:52:08.040
operator.
link |
00:52:09.040
Okay.
link |
00:52:10.040
And Python and Python 3.8 added the walrus operator.
link |
00:52:15.760
And the reason I think it's interesting is not just because of the feature, it has the
link |
00:52:21.720
same kind of expression feature that you can mention to see that it returns the value of
link |
00:52:26.000
the assignment.
link |
00:52:27.000
And then maybe you can comment on that in general, but on the other side of it, it's
link |
00:52:31.480
also the thing that toppled the dictator.
link |
00:52:38.040
It finally drove Guido to step down from BDFL, the toxicity of the community.
link |
00:52:42.920
So maybe what do you think about the walrus operator in Python?
link |
00:52:47.040
Is there an equivalent thing in Swift that really stress tested the community?
link |
00:52:54.520
And then on the flip side, what do you think about Guido stepping down over it?
link |
00:52:58.840
Yeah.
link |
00:52:59.840
Well, if I look past the details of the walrus operator, one of the things that makes it
link |
00:53:03.280
most polarizing is that it's syntactic sugar.
link |
00:53:05.840
Okay.
link |
00:53:06.840
What do you mean by syntactic sugar?
link |
00:53:09.160
It means you can take something that already exists in language and you can express it
link |
00:53:12.440
in a more concise way.
link |
00:53:14.480
So okay, I'm going to play Dolph's advocate.
link |
00:53:16.040
So this is great.
link |
00:53:18.880
Is that an objective or subjective statement?
link |
00:53:22.440
Can you argue that basically anything is syntactic sugar or not?
link |
00:53:27.600
No.
link |
00:53:28.600
Not everything is syntactic sugar.
link |
00:53:30.400
So for example, the type system, can you have classes versus, do you have types or not?
link |
00:53:40.480
So one type versus many types is not something that affects syntactic sugar.
link |
00:53:44.880
And so if you say, I want to have the ability to define types, I have to have all this language
link |
00:53:49.360
mechanics to define classes and oh, now I have to have inheritance and I have all this
link |
00:53:54.840
stuff, that's just making the language more complicated.
link |
00:53:57.200
That's not about sugaring it.
link |
00:54:00.960
Swift has the sugar.
link |
00:54:02.440
So Swift has this thing called iflet and it has various operators that are used to concisify
link |
00:54:09.000
specific use cases.
link |
00:54:10.560
So the problem with syntactic sugar, when you're talking about, hey, I have a thing
link |
00:54:14.840
that takes a lot to write and I have a new way to write it.
link |
00:54:17.720
You have this horrible trade off, which becomes almost completely subjective, which is how
link |
00:54:23.400
often does this happen and does it matter?
link |
00:54:26.400
And one of the things that is true about human psychology, particularly when you're talking
link |
00:54:29.160
about introducing a new thing, is that people overestimate the burden of learning something.
link |
00:54:36.240
And so it looks foreign when you haven't gotten used to it.
link |
00:54:38.520
But if it was there from the beginning, of course, it's just part of Python.
link |
00:54:42.400
Unquestionably, this is just the thing I know and it's not a new thing that you're worried
link |
00:54:47.160
about learning.
link |
00:54:48.160
It's just part of the deal.
link |
00:54:49.520
Now with Guido, I don't know Guido well.
link |
00:54:54.040
Yeah.
link |
00:54:55.040
Have you passed cost much?
link |
00:54:56.800
Yeah.
link |
00:54:57.800
I've met him a couple of times, but I don't know Guido well.
link |
00:55:00.200
But the sense that I got out of that whole dynamic was that he had put not just the decision
link |
00:55:06.860
maker weight on his shoulders, but it was so tied to his personal identity that he took
link |
00:55:13.400
it personally and he felt the need and he kind of put himself in the situation of being
link |
00:55:16.880
the person instead of building a base of support around him.
link |
00:55:21.040
I mean, this is probably not quite literally true.
link |
00:55:24.200
But by too much, so there's too much too much concentrated on him, right?
link |
00:55:28.960
And so and that can wear you down.
link |
00:55:31.000
Well, yeah, particularly because people then say Guido, you're a horrible person.
link |
00:55:35.400
I hate this thing, blah, blah, blah, blah, blah, blah, and sure, it's like, you know,
link |
00:55:38.800
maybe 1% of the community that's doing that.
link |
00:55:41.440
But Python's got a big community and 1% of millions of people is a lot of hate mail.
link |
00:55:46.840
And that just from human factor will just wear on you to clarify, it looked from just
link |
00:55:51.960
what I saw in the messaging for the let's not look at the million Python users, but
link |
00:55:55.960
at the Python core developers, it feels like the majority, the big majority on a vote were
link |
00:56:02.720
opposed to it.
link |
00:56:03.720
Okay.
link |
00:56:04.720
I'm not that close to it.
link |
00:56:05.720
So, so, so this, okay, so the situation is like literally, uh, yeah, I mean, the majority
link |
00:56:12.080
of the core developers again, we're opposed it.
link |
00:56:14.400
So I, and they weren't, they weren't even like against it.
link |
00:56:21.000
It was, uh, there was a few, well, they were against it, but the against it wasn't like,
link |
00:56:25.480
this is a bad idea.
link |
00:56:27.880
They were more like, we don't see why this is a good idea.
link |
00:56:31.520
And what that results in is there's a stalling feeling, like you, you just slow things down.
link |
00:56:38.080
Now, from my perspective, that you could argue this, and I think it's a very, it's very interesting
link |
00:56:44.520
if we look at politics today and the way Congress works, it's slowed down everything.
link |
00:56:49.200
It's a dampener.
link |
00:56:50.200
Yeah.
link |
00:56:51.200
It's a dampener, but like that's a dangerous thing too, because if it dampens things like,
link |
00:56:56.440
you know, well, what are you talking about?
link |
00:56:59.440
Like it's a low pass filter, but if you need billions of dollars injected into the economy
link |
00:57:03.600
or trillions of dollars, then suddenly stuff happens.
link |
00:57:06.400
Right.
link |
00:57:07.400
Yeah.
link |
00:57:08.400
For sure.
link |
00:57:09.400
So you're talking about...
link |
00:57:10.400
I'm not, I'm not, I'm not defending our political situation just to be clear, but you're talking
link |
00:57:13.840
about like a global pandemic.
link |
00:57:16.440
Well, I was hoping we could fix like the healthcare system and the educations and like, you know,
link |
00:57:22.680
uh...
link |
00:57:23.680
I'm not, I'm not a politics person.
link |
00:57:24.680
I don't, I don't, I don't know.
link |
00:57:25.680
Um, when it comes to languages, the community's kind of right in terms of it's a very high
link |
00:57:31.280
burden to add something to a language.
link |
00:57:33.320
So as soon as you add something, you have a community of people building on it and you
link |
00:57:35.920
can't remove it.
link |
00:57:37.480
Okay.
link |
00:57:38.480
And if there's a community of people that feel really uncomfortable with it, then taking
link |
00:57:42.080
it slow, I think is, is, is an important thing to do.
link |
00:57:45.720
And there's no rush, particularly if something that's 25 years old and is very established
link |
00:57:50.280
and, you know, it's not like coming, coming into its own.
link |
00:57:53.920
What about features?
link |
00:57:55.400
Well, well, so I think that the issue with, with Guido is that maybe this is a case where
link |
00:58:00.440
he realized that had outgrown him and it went from being the language.
link |
00:58:05.800
So Python, I mean, Guido is amazing, but, but Python isn't about Guido anymore.
link |
00:58:12.360
It's about the users.
link |
00:58:13.640
And to a certain extent, the users own it.
link |
00:58:15.640
And, you know, Guido spent years of his life, a significant fraction of his career on Python.
link |
00:58:23.160
And from his perspective, I imagine he's like, well, this is my thing.
link |
00:58:25.760
I should be able to do the thing I think is right.
link |
00:58:28.480
But you can also understand the users where they feel like, you know, this is my thing.
link |
00:58:33.080
I use this like, and, um, and I don't know, it's, it's, it's a hard, it's a hard thing.
link |
00:58:38.360
But what, if we could talk about leadership in this, because it's so interesting to me,
link |
00:58:42.480
I'm going to, I'm going to make, I'm going to work.
link |
00:58:44.680
Hopefully somebody makes it.
link |
00:58:45.680
If not, I'll make it a water supply shirt because I think it represents to me, maybe
link |
00:58:50.560
it's my Russian roots or something.
link |
00:58:53.920
You know, it's the burden of leadership.
link |
00:58:56.240
Like I feel like to push back.
link |
00:59:01.000
I feel like progress can only, like most difficult decisions, just like you said, there'll be
link |
00:59:06.560
a lot of divisiveness over, especially in the passionate community.
link |
00:59:12.280
It just feels like leaders need to take those risky decisions that, that if you like listen,
link |
00:59:20.600
that with some non zero probability, maybe even a high probability would be the wrong
link |
00:59:25.080
decision, but they have to use their gut and make that decision.
link |
00:59:28.800
Well, this, this is like one of the things where you see amazing founders, the founders
link |
00:59:34.840
understand exactly what's happened and why, how the company got there and are willing
link |
00:59:37.960
to say to, we have been doing thing X the last 20 years, but today we're going to do
link |
00:59:43.720
a thing Y and they make a major pivot for the whole company.
link |
00:59:47.440
The company lines up behind them, they move and it's the right thing.
link |
00:59:50.640
But then when the founder dies, the successor doesn't always feel that, that agency to be
link |
00:59:57.240
able to make those kinds of decisions.
link |
00:59:59.280
Even though they're the CEO, they could theoretically do whatever.
link |
01:00:02.440
There's two reasons for that in my opinion, or in many cases, it's always different.
link |
01:00:07.520
But one of which is they weren't there for all the decisions that were made.
link |
01:00:11.720
And so they don't know the principles in which those decisions were made.
link |
01:00:15.520
And once the principles change, you're, you should be obligated to change what you're
link |
01:00:20.280
doing and change direction.
link |
01:00:22.760
And so if you don't know how you got to where you are, it just seems like gospel.
link |
01:00:27.520
And you're not going to question it.
link |
01:00:29.880
You may not understand that it really is the right thing to do, so you just may not see
link |
01:00:33.320
it.
link |
01:00:34.320
That's so brilliant.
link |
01:00:35.320
I never thought of it that way.
link |
01:00:36.320
It's so much higher burden when as a leader you step into a thing that's already worked
link |
01:00:41.160
for a long time.
link |
01:00:42.160
Yeah.
link |
01:00:43.160
Yeah.
link |
01:00:44.160
And if you change it and it doesn't work out, now you're the person who screwed it up.
link |
01:00:46.520
People always second guess that.
link |
01:00:47.840
Yeah.
link |
01:00:48.840
And the second thing is that even if you decide to make a change, even if you're theoretically
link |
01:00:52.080
in charge, you're just a person that thinks they're in charge.
link |
01:00:57.240
Yeah.
link |
01:00:58.240
Meanwhile, you have to motivate the troops.
link |
01:00:59.240
You have to explain it to them in terms they'll understand.
link |
01:01:00.560
You have to get them to buy into it and believe in it because if they don't, then they're
link |
01:01:04.560
not going to be able to make the turn even if you tell them their bonuses are going to
link |
01:01:07.840
be curtailed.
link |
01:01:08.840
They're just not going to buy into it.
link |
01:01:10.880
And so there's only so much power you have as a leader and you have to understand what
link |
01:01:15.120
those limitations are.
link |
01:01:16.360
I used to be BDFL, you've been BDFL of some stuff.
link |
01:01:21.640
You're very heavy on the B, the benevolent, benevolent, dictated for life, I guess LVM.
link |
01:01:28.760
Yeah.
link |
01:01:29.760
So I still lead the LVM world.
link |
01:01:32.520
I mean, what's the role of, so then on Swift, you said that there's a group of people.
link |
01:01:38.400
Yeah.
link |
01:01:39.400
So if you contrast Python with Swift, right, one of the reasons, so everybody on the core
link |
01:01:44.440
team takes the role really seriously.
link |
01:01:46.360
And I think we all really care about where Swift goes.
link |
01:01:49.480
But you're almost delegating the final decision making to the wisdom of the group.
link |
01:01:55.040
And so it doesn't become personal.
link |
01:01:57.680
And also when you're talking with the community, so yeah, some people are very annoyed at certain
link |
01:02:02.400
decisions that get made.
link |
01:02:04.480
There's a certain faith in the process because it's a very transparent process.
link |
01:02:08.400
And when a decision gets made, a full rationale is provided, things like this.
link |
01:02:12.320
These are almost defense mechanisms to help both guide future discussions and provide
link |
01:02:16.840
case lock and like Supreme Court does about, this decision was made for this reason and
link |
01:02:21.040
here's the rationale and what we want to see more of or less of.
link |
01:02:24.720
But it's also a way to provide a defense mechanism so that when somebody's griping about it,
link |
01:02:29.040
they're not saying that person did the wrong thing.
link |
01:02:32.080
They're saying, well, this thing sucks and later they move on and they get over it.
link |
01:02:38.560
Yeah.
link |
01:02:39.560
The analogy is Supreme Court, I think, is really good.
link |
01:02:42.720
But then, okay, not to get person on the SWIFT team, but like, is there, is there, like,
link |
01:02:47.640
it just seems like it's impossible for their, for division not to emerge.
link |
01:02:52.440
Well, each of the humans on the SWIFT core team, for example, are different and the membership
link |
01:02:57.600
of the SWIFT core team changes slowly over time, which is I think a healthy thing.
link |
01:03:02.660
And so each of these different humans have different opinions.
link |
01:03:05.440
Trust me, it's not, it's not a singular consciousness by any stretch of the imagination.
link |
01:03:11.080
You've got three major organizations, including Apple, Google and SciFive all working together.
link |
01:03:16.600
And it's a small group of people, but you need high trust.
link |
01:03:20.240
You need, again, it comes back to the principles of what you're trying to achieve and understanding
link |
01:03:26.040
what you're optimizing for.
link |
01:03:27.800
And I think that starting with strong principles and working towards decisions is always a
link |
01:03:32.760
good way to both make wise decisions in general, but then be able to communicate them to people
link |
01:03:37.800
so that they can buy into them and that, that is hard.
link |
01:03:41.480
And so you mentioned LVM, LVM is going to be 20 years old this December.
link |
01:03:46.840
So it's, it's showing its own age.
link |
01:03:48.960
Do you have like, like a, like a, like a dragon cake plan or do you have a?
link |
01:03:53.400
No, I should definitely do that.
link |
01:03:54.600
Yeah.
link |
01:03:55.600
If we can have a pandemic cake, everybody gets a slice of cake and it gets, you know,
link |
01:04:01.520
sent through email.
link |
01:04:04.520
But the, but LVM has had tons of its own challenges over time too, right?
link |
01:04:09.240
And one of the challenges that the LVM community has in my opinion is that it has a whole bunch
link |
01:04:14.600
of people that have been working on LVM for 10 years, right?
link |
01:04:19.120
Cause this happens some somehow.
link |
01:04:21.200
And LVM has always been one way, but it needs to be a different way, right?
link |
01:04:25.120
And they've worked on it for like 10 years is a long time to work on something.
link |
01:04:28.880
And you know, you suddenly can't see the faults in the thing that you're working on.
link |
01:04:33.560
And LVM has lots of problems and we need to address them and we need to make it better.
link |
01:04:36.760
And if we don't make it better, then somebody else will come up with a better idea, right?
link |
01:04:40.360
And so it's just kind of of that age where the community is like in danger of getting
link |
01:04:45.240
too calcified and, um, and so I'm happy to see new projects joining and new things mixing
link |
01:04:51.200
it up.
link |
01:04:52.200
You know, Fortran is now a new, a new thing in the LVM community, which is an area some
link |
01:04:55.400
good.
link |
01:04:56.400
I've been trying to find, uh, on this little tangent, find people who program in Cobalt
link |
01:05:01.200
or Fortran, Fortran especially to talk to it.
link |
01:05:04.880
They're hard to find.
link |
01:05:06.360
Yeah.
link |
01:05:07.360
Look to the, uh, scientific community.
link |
01:05:09.880
They still use Fortran quite a bit.
link |
01:05:12.160
Interesting thing you kind of mentioned with LVM or just in general, that as something
link |
01:05:16.400
evolved, you're not able to see the faults.
link |
01:05:19.800
So do you, uh, fall in love with the thing over time or do you start hating everything
link |
01:05:24.480
about the thing over time?
link |
01:05:26.200
Well, so, so my, my, my personal folly is that, um, I see maybe not all, but many of the faults
link |
01:05:33.480
and they grate on me and I don't have time to go fix them.
link |
01:05:35.600
Yeah.
link |
01:05:36.600
And they get magnified over time.
link |
01:05:37.600
Well, and they may not get magnified, but they never get fixed and it's like sand underneath
link |
01:05:41.160
the, you know, it's just like grating against you and it's like sand underneath your fingernails
link |
01:05:45.400
or something.
link |
01:05:46.400
It's just like, you know, it's there.
link |
01:05:47.400
You can't get rid of it.
link |
01:05:48.640
Um, and so the, the problem is that if other people don't see it, right, nobody ever, like
link |
01:05:55.040
I can't go, I don't have time to go write the code and fix it anymore.
link |
01:05:58.520
But then, uh, people are resistant to change and so you say, Hey, we should go fix this
link |
01:06:02.680
thing.
link |
01:06:03.680
Like, oh yeah, that sounds risky.
link |
01:06:04.680
Yeah.
link |
01:06:05.680
Well, is it the right thing or not?
link |
01:06:06.680
Are the challenges, uh, the, the group dynamics or is it also just technical?
link |
01:06:11.520
I mean, some of these features like, yeah, I think, uh, as an observer is almost like
link |
01:06:16.560
a fan in, in the, uh, you know, as a spectator, the whole thing, I don't often think about,
link |
01:06:23.640
you know, some things might actually be technically difficult to implement.
link |
01:06:27.080
An example of this is we, we built this new compiler framework called MLAR.
link |
01:06:30.840
Yes.
link |
01:06:31.840
MLAR is this a whole new framework.
link |
01:06:34.280
It's not, many people think it's about machine learning.
link |
01:06:37.360
The ML stands for multi level because compiler people can't name things very well.
link |
01:06:41.360
I guess.
link |
01:06:42.360
Can we dig into what MLIR is?
link |
01:06:44.960
Yeah.
link |
01:06:45.960
So when you look at compilers, compilers have historically been solutions for a given space.
link |
01:06:51.840
So LLVM is a, it's really good for dealing with CPUs, let's just say at a high level.
link |
01:06:58.200
You look at, um, Java, Java has a JVM.
link |
01:07:01.720
The JVM is very good for garbage collected languages that need dynamic compilations.
link |
01:07:05.760
It's very optimized for specific space.
link |
01:07:08.520
And so hotspot is one of the compilers that gets used in that space and that compiler's
link |
01:07:11.560
really good at that kind of stuff.
link |
01:07:13.400
Um, usually when you build these domain specific compilers, you end up building whole thing
link |
01:07:18.000
from scratch for each domain.
link |
01:07:21.120
Uh, what's a domain?
link |
01:07:23.480
So what, what, what's the scope of a domain?
link |
01:07:26.760
Well, so here I would say like, if you look at Swift, there's several different parts
link |
01:07:30.480
to the Swift compiler, um, one of which is covered by, um, the LLVM part of it.
link |
01:07:35.880
There's also a high level piece that's specific to Swift and there's a huge amount of redundancy
link |
01:07:41.480
between those two different infrastructures and a lot of re, re implemented stuff that
link |
01:07:46.480
is similar but different.
link |
01:07:48.560
What is LLVM defined?
link |
01:07:50.080
LLVM is effectively an infrastructure.
link |
01:07:53.120
So you can mix and match it in different ways.
link |
01:07:55.200
It's built out of libraries.
link |
01:07:56.200
You can use it for different things, but it's really good at CPUs and GPUs, CPUs and like
link |
01:08:00.680
the tip of the iceberg on GPUs.
link |
01:08:02.600
It's not really great at GPUs.
link |
01:08:04.160
Okay.
link |
01:08:05.160
Um, but it turns out languages that then use it to talk to CPUs.
link |
01:08:11.200
And so it turns out there's a lot of hardware out there that is custom accelerators.
link |
01:08:14.920
So machine learning, for example, there are a lot of, uh, matrix multiply accelerators
link |
01:08:18.680
and things like this, there, there's a whole world of hardware synthesis.
link |
01:08:22.920
So we're, we're using MLIR to build circuits.
link |
01:08:26.360
Okay.
link |
01:08:27.360
And so you're compiling for a domain of transistors.
link |
01:08:30.960
And so what MLIR does is it provides a tremendous amount of compiler infrastructure that allows
link |
01:08:34.840
you to build these domain specific compilers in a much faster way and have the result be
link |
01:08:40.440
good.
link |
01:08:41.440
If we're, if we're thinking about the future, now we're talking about like ASICs, so anything.
link |
01:08:46.800
Yeah.
link |
01:08:47.800
And so if we project into the future, it's very possible that the number of these kinds
link |
01:08:53.680
of ASICs, very specific, uh, infrastructure thing, the architecture things, uh, like multiplies
link |
01:09:04.320
exponentially.
link |
01:09:05.320
I hope so.
link |
01:09:06.320
Yeah.
link |
01:09:07.320
So that's MLIR.
link |
01:09:08.320
So what MLIR, what MLIR does is it allows you to build these compilers very efficiently.
link |
01:09:12.720
Right.
link |
01:09:13.720
So one of the things that coming back to the LLVM thing, and then we'll go to hardware,
link |
01:09:18.160
is, um, LLVM is a, is a specific compiler for a specific domain.
link |
01:09:23.360
MLIR is now this very general, very flexible thing that can solve lots of different kinds
link |
01:09:28.200
of problems.
link |
01:09:29.360
So LLVM is a subset of what MLIR does.
link |
01:09:32.480
So MLIR is, I mean, it's an ambitious project then.
link |
01:09:35.200
Yeah.
link |
01:09:36.200
It's a very ambitious project.
link |
01:09:37.200
Yeah.
link |
01:09:38.200
And so to make it even more confusing, MLIR has joined the LLVM umbrella project.
link |
01:09:42.480
So it's part of the LLVM family.
link |
01:09:44.200
Right.
link |
01:09:45.200
Um, but where this comes full circle is now folks that work on the LLVM part, the classic
link |
01:09:50.000
part that's 20 years old, um, aren't aware of all the cool new things that have been
link |
01:09:54.280
done in the new, the new thing that, you know, MLIR was built by me and many other people
link |
01:09:59.520
that knew a lot about LLVM.
link |
01:10:01.920
And so we fixed a lot of the mistakes that lived in LLVM.
link |
01:10:04.320
I mean, we have this community dynamic where it's like, well, there's this new thing, but
link |
01:10:08.640
it's not familiar.
link |
01:10:09.640
Nobody knows it.
link |
01:10:10.640
It's like it's new and so let's not trust it.
link |
01:10:12.840
And so it's just really interesting to see the cultural social dynamic that comes out
link |
01:10:16.480
of that.
link |
01:10:17.480
And, and, you know, I think it's super healthy because we're seeing the ideas percolate and
link |
01:10:21.600
we're seeing the technology diffusion happen as people get more comfortable with it.
link |
01:10:25.240
They start to understand things in their own terms and this just gets to the, it takes
link |
01:10:29.200
a while for ideas to propagate, even though, um, they may be very different than what people
link |
01:10:34.320
are used to.
link |
01:10:35.320
So maybe let's talk about that a little bit, the world of basics and well, actually you're
link |
01:10:40.200
a, you're, you're, you have a new role at sci five.
link |
01:10:45.720
What's that place about?
link |
01:10:47.520
What is the vision for their vision for, I would say the future of computer.
link |
01:10:53.360
So I lead the engineering and product teams at sci five.
link |
01:10:56.040
Sci five is a company who's was founded with this architecture called risk five risk fives
link |
01:11:03.240
a new instruction set instruction sets are the things inside of your computer that tell
link |
01:11:06.520
you how to run things, um, x 86 from Intel and arm from the arm company and things like
link |
01:11:12.360
this or other instruction sets.
link |
01:11:13.880
I've talked to science.
link |
01:11:14.880
I've talked to Dave Patterson who's super excited about risk five.
link |
01:11:17.720
Dave, Dave is awesome.
link |
01:11:18.720
He's brilliant.
link |
01:11:19.720
Yeah.
link |
01:11:20.720
The risk five is distinguished by not being proprietary.
link |
01:11:24.720
And so x a six can only be made by Intel and AMD arm can only be made by arm.
link |
01:11:30.440
They sell licenses to build arm ships to other companies, things like this MIPS is another
link |
01:11:34.840
instruction set that is owned by the MIPS company now wave and then it gets licensed out things
link |
01:11:39.520
like that.
link |
01:11:40.520
Um, and so risk five is an open standard that anybody can build chips for.
link |
01:11:45.240
And so sci five was founded by three of the founders of risk five that designed and built
link |
01:11:50.320
it in Berkeley working with Dave.
link |
01:11:53.320
And so that was the, the genesis of the company sci five today has some of the world's best
link |
01:11:58.240
risk five cores and we're selling them and that's really great.
link |
01:12:01.440
They're going to tons of products.
link |
01:12:02.480
It's very exciting.
link |
01:12:03.480
Um, so they're taking this, uh, thing that's open source and just being, uh, trying to
link |
01:12:07.520
be or are the best in the world at building these things.
link |
01:12:10.760
Yeah.
link |
01:12:11.760
So here it's the specifications open source.
link |
01:12:13.360
It's like saying TCP IP is an open standard or C is an open standard, but then you have
link |
01:12:18.560
to build an implementation of the standard.
link |
01:12:20.880
And so sci five on the one hand pushes forward and defined and pushes forward the standard.
link |
01:12:26.360
On the other hand, we have implementations that are best in class for different points
link |
01:12:30.360
in the space, depending on if you want a really tiny CPU or if you want a really big beefy
link |
01:12:34.680
one that, that, uh, is faster, but it uses more area and things like this.
link |
01:12:39.000
What about the actual manufacturer chip?
link |
01:12:41.200
So like what?
link |
01:12:42.200
Yeah.
link |
01:12:43.200
So where does that all fit?
link |
01:12:44.200
I'm going to ask a bunch of dumb questions because this is how we learn, right?
link |
01:12:48.320
And so, uh, what the way this works is that there's generally a separation of the people
link |
01:12:54.200
who design the circuits and then people who manufacture them.
link |
01:12:57.000
And so that you'll hear about fabs like TSMC and Samsung and things like this that actually
link |
01:13:02.360
produce the chips, but they take a design coming in and that design specifies how, um,
link |
01:13:08.760
how the, you know, you turn, uh, code for the chip into, uh, little rectangles that
link |
01:13:16.360
then use photo lithography to make, uh, mask sets and then burn transistors onto a chip
link |
01:13:22.240
or onto onto silicon rather.
link |
01:13:24.880
So we're talking about mass manufacturing.
link |
01:13:27.520
So yeah, they're talking about making hundreds of millions of parts and things like that.
link |
01:13:31.480
And so the fab handles the volume production, things like that.
link |
01:13:34.840
But when you look at this problem, um, the interesting thing about the space when you
link |
01:13:39.240
look at it is that, um, these, the steps that you go from designing a chip and writing the
link |
01:13:44.920
quote unquote code for it and things like fair log and languages like that down to what
link |
01:13:49.960
you hand off to the fab is a really well studied, really old problem.
link |
01:13:55.080
Okay.
link |
01:13:56.080
Um, tons of people have worked on it, lots of smart people have built systems and tools.
link |
01:14:00.080
Um, these tools then have generally gone through acquisitions and so they've ended up at three
link |
01:14:04.920
different major companies that build and sell these tools.
link |
01:14:07.880
They're called EDA tools like for electronic design automation.
link |
01:14:11.120
Um, the problem with this is you have huge amounts of fragmentation.
link |
01:14:14.640
You have loose standards, um, and the tools don't really work together.
link |
01:14:20.120
So you have tons of duct tape and you have tons of, uh, lost productivity.
link |
01:14:23.800
Now these are, uh, these are tools for design.
link |
01:14:26.800
So the risk five is a instruction.
link |
01:14:30.360
Like what is risk five?
link |
01:14:32.200
Like how deep does it go?
link |
01:14:33.400
How, how, how much does it touch the hardware?
link |
01:14:36.040
How much does it define how much of the hardware is?
link |
01:14:38.360
Yeah.
link |
01:14:39.360
So risk, risk five is all about, um, given a CPU.
link |
01:14:42.000
So the, the, the processor and your computer, how does the, the compiler, like Swift compiler,
link |
01:14:47.440
the C compiler, things like this, how does it make it work?
link |
01:14:50.560
So it's, what is the assembly code?
link |
01:14:52.800
And so you write risk five assembly instead of XA six assembly, for example.
link |
01:14:57.160
But it's the set of instructions as opposed to instructions.
link |
01:14:59.720
Yeah.
link |
01:15:00.720
What, what do you say it tells you how the compiler works?
link |
01:15:02.880
Well, sorry, it's what the compiler talks to.
link |
01:15:05.360
Okay.
link |
01:15:06.360
Yeah.
link |
01:15:07.360
And then, uh, the tooling you mentioned that the disparate tools are for what, for,
link |
01:15:11.160
when you're building a specific chip.
link |
01:15:13.440
So risk five in hardware, in hardware.
link |
01:15:15.600
Yeah.
link |
01:15:16.600
So, so risk five, you can buy a risk five core from sci five and say, Hey, I want to
link |
01:15:20.440
have a certain number of run a certain number of gigahertz.
link |
01:15:23.360
I want it to be this big.
link |
01:15:24.640
I want it to be, have these features.
link |
01:15:26.800
I want to have, um, like I want floating point or not, for example.
link |
01:15:31.160
Um, and then what you get is you get a description of a CPU with those characteristics.
link |
01:15:36.680
Now if you want to make a chip and you want to build like an iPhone chip or something
link |
01:15:40.200
like that, right?
link |
01:15:41.200
You have to take both the CPU, but then you have to talk to memory.
link |
01:15:44.480
You have to have timers, IOs, a GPU, other components.
link |
01:15:48.640
And so you need to pull all those things together into what's called an ASIC, an application
link |
01:15:54.400
specific in a grade circuit, so custom chip.
link |
01:15:57.000
And then you take that design and then you have to transform it into something that the
link |
01:16:01.160
fabs like TSMC, for example, know how to turn, take to production.
link |
01:16:06.880
Got it.
link |
01:16:07.880
So, but yeah.
link |
01:16:08.880
And so that process.
link |
01:16:09.880
I will, I can't help but see it as, is a big compiler.
link |
01:16:15.760
Yeah.
link |
01:16:16.760
It's a whole bunch of compilers written without thinking about it through that lens.
link |
01:16:21.040
And so.
link |
01:16:22.040
Isn't the universe a compiler?
link |
01:16:24.040
Yeah.
link |
01:16:25.040
Compilers do two things.
link |
01:16:26.840
They represent things and transform them.
link |
01:16:28.600
Yeah.
link |
01:16:29.600
And so there's a lot of things that end up being compilers.
link |
01:16:31.960
But this is, this is a space where we're talking about design and usability and the way you
link |
01:16:36.400
think about things, the way things compose correctly, it matters a lot.
link |
01:16:41.000
And so sci fi is investing a lot into that space.
link |
01:16:43.760
And we think that there's a lot, a lot of benefit that can be made by allowing people
link |
01:16:47.440
to design chips faster, get them to market quicker and scale out because, you know, at
link |
01:16:53.920
the alleged more end of Moore's law, you've got this problem of you're not getting free
link |
01:17:00.120
performance just by waiting another year for a faster CPU.
link |
01:17:03.720
And so you have to find performance in other ways.
link |
01:17:06.640
And one of the ways to do that is with custom accelerators and other things in hardware.
link |
01:17:12.000
And so we'll talk a little bit about, a little more about ASICs.
link |
01:17:17.560
But do you see that a lot of people, a lot of companies will try to have like different
link |
01:17:25.640
sets of requirements that this whole process to go for.
link |
01:17:28.480
So like, like almost different car companies might use different and like different PC manufacturers,
link |
01:17:35.240
like, so is this, like, is risk five in this whole process, is it potentially the future
link |
01:17:42.480
of all computing devices?
link |
01:17:44.400
Yeah, I think that, so if you look at risk five and step back from the silicon side of
link |
01:17:48.840
things, risk five is an open standard.
link |
01:17:51.840
And one of the things that has happened over the course of decades, if you look over the
link |
01:17:56.040
long arc of computing, somehow became decades old, is that you have companies that come
link |
01:18:02.160
and go and you have instruction sets that come and go.
link |
01:18:04.960
Like one example of this out of many is Sun with Spark, Sun one way, Spark still lives
link |
01:18:11.960
on it if you just do, but we have HP had this instruction set called PA risk.
link |
01:18:18.260
So PA risk was its big server business and had tons of customers.
link |
01:18:22.960
They decided to move to this architecture called Itanium from Intel.
link |
01:18:27.000
Yeah.
link |
01:18:28.000
It didn't work out so well.
link |
01:18:29.480
Yeah.
link |
01:18:30.480
Right.
link |
01:18:31.480
And so you have this issue of you're making many billion dollar investments on instruction
link |
01:18:36.000
sets that are owned by a company.
link |
01:18:38.360
And even companies as big as Intel don't always execute as well as they could.
link |
01:18:42.520
They have their own issues.
link |
01:18:44.160
HP, for example, decided that it wasn't in their best interest to continue investing
link |
01:18:48.120
in the space because it was very expensive.
link |
01:18:49.760
So they make technology decisions or they make their own business decisions and this
link |
01:18:54.320
means that as a customer, what do you do?
link |
01:18:58.000
You've sunk all this time, all this engineering, all this software work, all these, you've
link |
01:19:01.440
built other products around them and now you're stuck.
link |
01:19:03.880
Right.
link |
01:19:04.880
What risk five does is provide you more optionality in the space because if you buy an implementation
link |
01:19:10.720
of risk five from sci five and you should, they're the best ones.
link |
01:19:16.440
But if something bad happens to sci five in 20 years, right, well, great.
link |
01:19:20.560
You can turn around and buy risk five core from somebody else.
link |
01:19:23.520
And there's an ecosystem of people that are all making different risk five cores with
link |
01:19:27.080
different tradeoffs, which means that if you have more than one requirement, if you have
link |
01:19:30.920
a family of products, you can probably find something in the risk five space that fits
link |
01:19:34.960
your needs.
link |
01:19:35.960
Whereas with, if you're talking about XA six, for example, it's Intel's only going to
link |
01:19:40.960
bother to make certain classes of devices.
link |
01:19:43.640
Right.
link |
01:19:44.640
I see.
link |
01:19:46.640
So maybe a weird question, but like if sci five is like infinitely successful in the
link |
01:19:54.960
next 20, 30 years, what does the world look like?
link |
01:19:58.160
So like, how does the world of computing change?
link |
01:20:01.960
So too much diversity and hardware instruction sets, I think is bad.
link |
01:20:06.640
Like we have a lot of people that are using lots of different instruction sets, particularly
link |
01:20:11.320
in the embedded, the like very tiny microcontroller space, the thing in your toaster that are
link |
01:20:18.440
just weird and different for historical reasons.
link |
01:20:21.160
And so the compilers and the tool chains and the languages on top of them aren't there.
link |
01:20:27.280
And so the developers for that software have to use really weird tools because the ecosystem
link |
01:20:32.560
that supports is not big enough.
link |
01:20:34.280
So I expect that will change, right?
link |
01:20:35.680
People will have better tools and better languages, better features everywhere that then can
link |
01:20:39.880
serve as many different points in the space.
link |
01:20:43.360
And I think risk five will progressively eat more of the ecosystem because it can scale
link |
01:20:49.240
up, it can scale down sideways left, right.
link |
01:20:51.720
It's very flexible and very well considered and well designed instruction set.
link |
01:20:56.360
I think when you look at sci five tackling silicon and how people build chips, which
link |
01:21:00.120
is a very different space, that's where you say, I think we'll see a lot more custom
link |
01:21:06.640
chips.
link |
01:21:07.640
And that means that you get much more battery life, you get better, better tuned solutions
link |
01:21:12.800
for your IOT thingy, you get, you get people to move faster.
link |
01:21:18.320
You get the ability to have faster time to market, for example.
link |
01:21:21.520
So how many custom, so first of all, on the IOT side of things, do you see the number
link |
01:21:26.120
of smart toasters increasing exponentially?
link |
01:21:30.320
So and if you do, like how much customization per toaster is there?
link |
01:21:39.000
Do all toasters in the world run the same silicon, like the same design?
link |
01:21:44.160
Or is it different companies have different design?
link |
01:21:46.200
Like how much customization is possible here?
link |
01:21:49.000
Well, a lot of it comes down to cost, right?
link |
01:21:52.520
And so the way that chips work is you end up paying by the, one of the factors is the
link |
01:21:57.560
size of the chip, and so what ends up happening just from an economic perspective is there's
link |
01:22:03.280
only so many chips that get made in a year of a given design.
link |
01:22:07.480
And so often what customers end up having to do is they end up having to pick up a chip
link |
01:22:11.560
that exists that was built for somebody else so that they can then ship their product.
link |
01:22:16.720
And the reason for that is they don't have the volume of the iPhone, they can't afford
link |
01:22:20.000
to build a custom chip.
link |
01:22:21.560
However, what that means is they're now buying an off the shelf chip that isn't really good,
link |
01:22:27.000
isn't a perfect fit for their needs.
link |
01:22:28.280
And so they're paying a lot of money for it because they're buying silicon that they're
link |
01:22:31.880
not using.
link |
01:22:32.880
Well, if you now reduce the cost of designing the chip, now you get a lot more chips.
link |
01:22:37.960
And the more you reduce it, the easier it is to design chips.
link |
01:22:42.560
The more the world keeps evolving and we get more AI accelerators, we get more other things,
link |
01:22:46.800
we get more standards to talk to, we get 6G, right?
link |
01:22:51.680
You get changes in the world that you want to be able to talk to these different things.
link |
01:22:54.880
There's more diversity in the cross product of features that people want.
link |
01:22:58.840
And that drives differentiated chips in another direction.
link |
01:23:03.440
And so nobody really knows what the future looks like, but I think that there's a lot
link |
01:23:07.960
of silicon in the future.
link |
01:23:09.120
Speaking of the future, you said Moore's Law allegedly is dead.
link |
01:23:13.880
So do you think, do you agree with Dave Patterson and many folks that Moore's Law is dead?
link |
01:23:22.320
Moore, do you agree with Jim Keller, who was standing at the helm of the pirate ship saying
link |
01:23:28.880
it's still alive?
link |
01:23:31.240
Yeah.
link |
01:23:32.240
Well, so I agree with what they're saying and different people are interpreting the
link |
01:23:37.880
animal's law in different ways.
link |
01:23:39.800
So Jim would say, there's another 1,000 X left in physics and we can continue to squeeze
link |
01:23:46.200
the stone and make it faster and smaller and smaller geometries and all that kind of stuff.
link |
01:23:52.600
He's right.
link |
01:23:53.600
So Jim is absolutely right that there's a ton of progress left and we're not at the
link |
01:23:58.720
limit of physics yet.
link |
01:24:01.840
That's not really what Moore's Law is though.
link |
01:24:05.040
If you look at what Moore's Law is, is that it's a very simple evaluation of, okay, well,
link |
01:24:11.560
if you look at the cost per, I think it was cost per area and the most economic point in
link |
01:24:16.360
that space.
link |
01:24:17.360
And if you go look at the now quite old paper that describes this, Moore's Law has a specific
link |
01:24:23.040
economic aspect to it.
link |
01:24:25.560
And I think this is something that Dave and others often point out.
link |
01:24:28.400
And so on a technicality, that's right.
link |
01:24:31.400
I look at it from, so I can acknowledge both of those viewpoints.
link |
01:24:35.080
They're both right.
link |
01:24:36.080
They're both right.
link |
01:24:37.080
I'll give you a third wrong viewpoint that may be right in its own way.
link |
01:24:40.400
Which is single threaded performance doesn't improve like it used to.
link |
01:24:45.960
And it used to be back when you got a, you know, a Pentium 66 or something and the year
link |
01:24:51.440
before you had a Pentium 33 and now it's twice as fast, right?
link |
01:24:56.880
Well it was twice as fast at doing exactly the same thing, okay?
link |
01:25:01.120
Like literally the same program ran twice as fast.
link |
01:25:03.880
You just wrote a check and waited a year, year and a half.
link |
01:25:07.240
So that's what a lot of people think about Moore's Law and I think that is dead.
link |
01:25:12.000
And so what we're seeing instead is we're pushing, we're pushing people to write software
link |
01:25:16.440
in different ways.
link |
01:25:17.440
And so we're pushing people to write CUDA so they can get GPU compute and the thousands
link |
01:25:22.480
of cores on GPU.
link |
01:25:23.480
We're talking about C programmers having to use P threads because they now have, you
link |
01:25:27.800
know, 100 threads or 50 cores in a machine or something like that.
link |
01:25:32.040
You're now talking about machine learning accelerators that are now domain specific.
link |
01:25:35.320
And when you look at these kinds of use cases, you can still get performance.
link |
01:25:40.720
And Jim will come up with cool things that utilize the silicon in new ways for sure.
link |
01:25:45.920
But you're also going to change the programming model.
link |
01:25:48.920
And now when you start talking about changing the programming model, that's when you come
link |
01:25:51.520
back to languages and things like this too because often what you see is like you take
link |
01:25:58.480
the C programming language, right?
link |
01:25:59.800
The C programming language is designed for CPUs.
link |
01:26:03.440
And so if you want to talk to GPU, now you're talking to its cousin, CUDA.
link |
01:26:07.640
Okay.
link |
01:26:08.640
CUDA is a different thing with a different set of tools, a different world, a different
link |
01:26:13.160
way of thinking.
link |
01:26:14.520
And we don't have one world that scales.
link |
01:26:16.600
And I think that we can get there.
link |
01:26:18.440
We can have one world that scales in a much better way.
link |
01:26:21.120
On a small tangent, then I think most programming languages are designed for CPUs, for a single
link |
01:26:26.400
core, even just in their spirit, even if they allow for parallelization.
link |
01:26:30.520
So what does it look like for programming language to have parallelization or massive parallelization
link |
01:26:38.600
as its first principle?
link |
01:26:41.400
So the canonical example of this is the hardware design world.
link |
01:26:46.480
So Verilog, VHDL, these kinds of languages, they're what's called a high level synthesis
link |
01:26:52.760
language.
link |
01:26:53.760
This is the thing people design chips in.
link |
01:26:56.960
When you're designing a chip, it's kind of like a brain where you have infinite parallelism.
link |
01:27:02.920
You're like laying down transistors.
link |
01:27:06.080
Transistors are always running.
link |
01:27:07.080
Okay.
link |
01:27:08.080
And so you're not saying, run this transistor, then this transistor, then this transistor.
link |
01:27:11.920
It's like your brain.
link |
01:27:13.320
Your neurons are always just doing something.
link |
01:27:15.320
They're not clocked.
link |
01:27:16.320
They're just doing their thing.
link |
01:27:20.320
And so when you design a chip, or when you design a CPU, when you design a GPU, when
link |
01:27:24.680
you design, when you're laying down the transistors, similarly, you're talking about, well, okay,
link |
01:27:29.120
well, how do these things communicate?
link |
01:27:31.400
And so these languages exist.
link |
01:27:33.040
Verilog is a kind of mixed example of that.
link |
01:27:36.040
Now, these languages are really great.
link |
01:27:37.560
Yeah, at a very low level.
link |
01:27:39.240
Yeah.
link |
01:27:40.240
They're very low level.
link |
01:27:41.240
And abstraction is necessary here.
link |
01:27:42.600
And there's different approaches with that, and it's itself a very complicated world.
link |
01:27:49.080
But it's implicitly parallel.
link |
01:27:50.800
And so having that as a, as the domain that you program towards makes it so that by default,
link |
01:27:57.880
you get parallel systems.
link |
01:27:59.040
If you look at CUDA, CUDA is a point halfway in the space where in CUDA, when you write
link |
01:28:04.240
a CUDA kernel for your GPU, it feels like you're writing a scalar program.
link |
01:28:08.200
So you're like, you have ifs, you have for loops, stuff like this, you're just writing
link |
01:28:11.040
normal, normal code.
link |
01:28:12.760
But what happens outside of that in your driver is that it actually is running you on like
link |
01:28:16.800
a thousand things at once, right?
link |
01:28:18.960
And so it's parallel, but it has pulled it out of the programming model.
link |
01:28:24.000
And so now you as a programmer are working in a simpler world, and it's solved that for
link |
01:28:29.640
you.
link |
01:28:30.640
All right.
link |
01:28:31.640
How do you take the language like Swift?
link |
01:28:35.520
You know, if we think about GPUs, but also ASICs, maybe if we can dance back and forth
link |
01:28:40.840
between hardware and software, is, you know, how do you design for these features to be
link |
01:28:46.840
able to program, make it a first class citizen to be able to do like Swift for TensorFlow,
link |
01:28:53.160
to be able to do machine learning on current hardware, but also future hardware like TPUs
link |
01:28:59.600
and all kinds of ASICs that I'm sure will be popping up more and more.
link |
01:29:02.760
Yeah.
link |
01:29:03.760
Well, so a lot of this comes down to this whole idea of having the nuts and bolts underneath
link |
01:29:06.720
the covers that work really well.
link |
01:29:08.720
So you need, if you're talking to TPUs, you need, you know, MLIR or XLA or one of these
link |
01:29:13.200
compilers that talks to TPUs to build on top of, okay?
link |
01:29:17.640
And if you're talking to circuits, you need to figure out how to lay down the transistors
link |
01:29:21.440
and how to organize it and how to set up clocking and like all the domain problems that you
link |
01:29:24.640
get with circuits.
link |
01:29:27.520
Then you have to decide how to explain it to a human.
link |
01:29:29.240
What is EY?
link |
01:29:30.720
Right.
link |
01:29:31.720
And if you do it right, that's a library problem, not a language problem.
link |
01:29:36.560
And that works if you have a library or a language which allows your library to write
link |
01:29:41.680
things that feel native in the language by implementing libraries, because then you can
link |
01:29:47.000
innovate in programming models without having to change your syntax again and have to invent
link |
01:29:52.600
new code formatting tools and like all the other things that languages come with.
link |
01:29:57.760
And this gets really interesting.
link |
01:30:00.040
And so if you look at this space, the interesting thing once you separate out syntax becomes
link |
01:30:06.160
what is that programming model?
link |
01:30:07.920
And so do you want the CUDA style, I write one program and it runs in many places?
link |
01:30:14.400
Do you want the implicitly parallel model?
link |
01:30:16.920
How do you reason about that?
link |
01:30:17.920
How do you give developers, you know, chip architects the ability to express their intent?
link |
01:30:24.200
And that comes into this whole design question of how do you detect bugs quickly so you don't
link |
01:30:29.400
have to tape out a chip to find out it's wrong ideally, right?
link |
01:30:32.720
How do you, and you know, this is a spectrum.
link |
01:30:35.640
How do you make it so that people feel productive so their turnaround time is very quick?
link |
01:30:40.580
All these things are really hard problems.
link |
01:30:42.680
And in this world, I think that not a lot of effort has been put into that design problem
link |
01:30:47.960
and thinking about the layering and other pieces.
link |
01:30:50.600
Well, you've, on the topic of concurrency, you've written the Swift concurrency manifest.
link |
01:30:55.560
I think it's kind of interesting.
link |
01:30:57.960
Anything that has the word manifesto in is very interesting.
link |
01:31:02.520
Can you summarize the key ideas of each of the five parts you've written about?
link |
01:31:07.480
So what is a manifesto?
link |
01:31:09.000
Yes.
link |
01:31:10.000
How about, we start there.
link |
01:31:11.960
So in the Swift community, we have this problem, which is on the one hand, you want to have
link |
01:31:17.480
relatively small proposals that you can kind of fit in your head.
link |
01:31:21.480
You can understand the details at a very fine grain level that move the world forward.
link |
01:31:26.080
But then you also have these big arcs, okay?
link |
01:31:29.160
And often when you're working on something that is a big arc, but you're tackling it
link |
01:31:33.080
in small pieces, you have this question of, how do I know I'm not doing a random walk?
link |
01:31:36.920
Where are we going?
link |
01:31:38.320
Like, how does this add up?
link |
01:31:39.920
Furthermore, when you start that first, the first small step, what terminology do you
link |
01:31:45.000
use?
link |
01:31:46.000
How do we think about it?
link |
01:31:47.000
What is better and worse in the space?
link |
01:31:48.000
What are the principles?
link |
01:31:49.000
What are we trying to achieve?
link |
01:31:50.000
And so what a manifesto in the Swift community does is it starts to say, hey, well, let's
link |
01:31:54.040
step back from the details of everything and let's paint a broad picture to talk about
link |
01:31:58.600
how what we're trying to achieve.
link |
01:32:01.400
Let's give an example design point.
link |
01:32:02.960
Let's try to paint the big picture so that then we can zero in on the individual steps
link |
01:32:07.320
and make sure that we're making good progress.
link |
01:32:09.800
And so the Swift concurrency manifesto is something I wrote three years ago.
link |
01:32:13.960
It's been a while, maybe more, trying to do that for Swift in concurrency.
link |
01:32:18.920
It starts with some fairly simple things, like making the observation that when you
link |
01:32:25.400
have multiple different computers or multiple different threads that are communicating, it's
link |
01:32:29.080
best for them to be asynchronous.
link |
01:32:32.120
And so you need things to be able to run separately and then communicate with each other.
link |
01:32:35.880
And this means asynchronous.
link |
01:32:37.560
And this means that you need a way to modeling asynchronous communication.
link |
01:32:41.880
Many languages have features like this.
link |
01:32:44.000
Async await is a popular one.
link |
01:32:45.520
And so that's what I think is very likely in Swift.
link |
01:32:49.520
But as you start building this tower of abstractions, it's not just about how do you write this?
link |
01:32:53.760
You then reach into the, how do you get memory safety?
link |
01:32:57.600
Because you want correctness.
link |
01:32:58.600
You want debugability and sanity for developers.
link |
01:33:01.800
And how do you get that memory safety into the language?
link |
01:33:06.720
So if you take a language like Go or C or any of these languages, you get what's called
link |
01:33:11.000
a race condition when two different threads or Go routines or whatever touch the same
link |
01:33:15.360
point in memory, right?
link |
01:33:17.520
This is a huge like maddening problem to debug because it's not reproducible generally.
link |
01:33:24.640
And so there's tools, there's a whole ecosystem of solutions that built up around this.
link |
01:33:28.520
But it's a huge problem when you're writing concurrent code.
link |
01:33:31.200
And so with Swift, this whole value semantics thing is really powerful there because it turns
link |
01:33:35.920
out that math and copies actually work even in concurrent worlds.
link |
01:33:40.840
And so you get a lot of safety just out of the box, but there are also some hard problems
link |
01:33:44.560
and it talks about some of that.
link |
01:33:47.160
When you start building up to the next level up and you start talking beyond memory safety,
link |
01:33:50.560
you have to talk about what is the programmer model?
link |
01:33:53.080
How does a human think about this?
link |
01:33:54.400
So a developer that's trying to build a program think about this and it proposes a really
link |
01:33:59.320
old model with a new spin called actors.
link |
01:34:02.440
Actors are about saying we have islands of single threadedness logically.
link |
01:34:08.240
So you write something that feels like it's one programming, one program running in a
link |
01:34:12.640
unit, and then it communicates asynchronously with other things.
link |
01:34:18.080
And so making that expressive and natural feel good, be the first thing you reach for
link |
01:34:21.800
and being safe by default is a big part of the design of that proposal.
link |
01:34:26.120
When you start going beyond that, now you start to say, cool, well, these things that
link |
01:34:29.920
communicate asynchronously, they don't have to share memory.
link |
01:34:32.360
Well, if they don't have to share memory and they're sending messages to each other, why
link |
01:34:36.200
do they have to be in the same process?
link |
01:34:39.360
These things should be able to be in different processes on your machine.
link |
01:34:42.800
And why just processes?
link |
01:34:44.200
Well, why not different machines?
link |
01:34:46.800
And so now you have a very nice gradual transition towards distributed programming.
link |
01:34:51.880
And of course, when you start talking about the big future, the manifesto doesn't go
link |
01:34:56.560
into it.
link |
01:34:57.560
But accelerators are things you talk to asynchronously by sending messages to them, and how do you
link |
01:35:05.080
program those?
link |
01:35:06.080
Well, that gets very interesting.
link |
01:35:08.280
That's not in the proposal.
link |
01:35:10.100
And how much do you want to make that explicit, like the control of that whole process explicit
link |
01:35:17.440
to the programmer?
link |
01:35:18.440
Yeah, good question.
link |
01:35:19.440
So when you're designing any of these kinds of features or language features or even libraries,
link |
01:35:25.400
you have this really hard trade off that you have to make, which is how much is it magic
link |
01:35:29.800
or how much is it in the human's control?
link |
01:35:32.200
How much can they predict and control it?
link |
01:35:34.880
What do you do when the default case is the wrong case?
link |
01:35:40.400
And so when you're designing a system, I won't name names, but there are systems where it's
link |
01:35:48.840
really easy to get started, and then you jump it.
link |
01:35:52.720
So let's pick like logo.
link |
01:35:54.480
So something like this.
link |
01:35:55.680
So it's really easy to get started.
link |
01:35:57.160
It's really designed for teaching kids.
link |
01:35:59.760
But as you get into it, you hit a ceiling, and then you can't go any higher.
link |
01:36:03.320
And then what do you do?
link |
01:36:04.320
You have to go switch to a different world and rewrite all your code.
link |
01:36:07.200
And this logo is a silly example here.
link |
01:36:09.160
This exists in many other languages.
link |
01:36:11.440
With Python, you would say like concurrency, right?
link |
01:36:15.320
So Python has the global interpreter lock.
link |
01:36:17.400
So threading is challenging in Python.
link |
01:36:19.520
And so if you start writing a large scale application in Python, and then suddenly you
link |
01:36:23.960
need concurrency, you're kind of stuck with a series of bad trade offs, right?
link |
01:36:30.600
There's other ways to go where you say foist all the complexity on the user all at once.
link |
01:36:37.200
And that's also bad in a different way.
link |
01:36:38.920
And so what I prefer is building a simple model that you can explain that then has an
link |
01:36:45.680
escape hatch.
link |
01:36:47.040
So you get in, you have guardrails, memory safety works like this in Swift where you
link |
01:36:52.480
can start with, by default, if you use all the standard things, it's memory safe, you're
link |
01:36:57.160
not going to shoot your foot off.
link |
01:36:58.720
But if you want to get a C level pointer to something, you can explicitly do that.
link |
01:37:04.440
But by default, it's, there's guardrails.
link |
01:37:07.840
There's guardrails.
link |
01:37:08.840
Okay.
link |
01:37:09.840
So, but like, you know, whose job is it to figure out which part of the code is paralyzable?
link |
01:37:17.520
So in the case of the proposal, it is the humans job.
link |
01:37:21.160
So they decide how to architect their application.
link |
01:37:24.440
And then the runtime in the compiler is very predictable.
link |
01:37:29.240
And so this, this is in contrast to like, there's a long body of work, including on
link |
01:37:33.320
Fortran for auto paralyzing compilers.
link |
01:37:37.160
And this is an example of a bad thing.
link |
01:37:39.760
And my, so as a compiler person, I can write on compiler people.
link |
01:37:43.840
Often compiler people will say, cool, since I can't change the code, I'm going to write
link |
01:37:47.800
my compiler that then takes this unmodified code and makes it go way faster on this machine.
link |
01:37:52.400
Okay.
link |
01:37:53.400
And so it does pattern matching.
link |
01:37:56.360
It does like really deep analysis.
link |
01:37:58.640
Compiler people are really smart.
link |
01:37:59.640
And so they like want to like do something really clever and tricky.
link |
01:38:02.600
And you get like 10 X speed up by taking like an array of structures and turn it into a
link |
01:38:06.680
structure of arrays or something because it's so much better for memory.
link |
01:38:09.400
Like there's bodies, like tons of tricks.
link |
01:38:12.000
Yeah.
link |
01:38:13.000
They love optimization.
link |
01:38:14.000
Yeah.
link |
01:38:15.000
You love optimization.
link |
01:38:16.000
Everyone loves optimization.
link |
01:38:17.000
Well, and it's just this promise of build with my compiler and your thing goes fast.
link |
01:38:20.280
Yeah.
link |
01:38:21.280
Right.
link |
01:38:22.280
But here's the problem.
link |
01:38:23.280
You write a program.
link |
01:38:24.760
You run it with my compiler.
link |
01:38:25.760
It goes fast.
link |
01:38:26.760
You're very happy.
link |
01:38:27.760
Wow.
link |
01:38:28.760
It's so much faster than the other compiler.
link |
01:38:29.760
Yeah.
link |
01:38:30.760
Then you go and you got a feature to your program where you refactor some code and suddenly
link |
01:38:32.960
you got a 10 X loss in performance.
link |
01:38:34.960
Yeah.
link |
01:38:35.960
Well, why?
link |
01:38:36.960
What just happened there?
link |
01:38:37.960
What just happened?
link |
01:38:38.960
There's you, the heuristic, the pattern matching the compiler, whatever analysis it was doing
link |
01:38:43.080
just got defeated because you didn't inline a function or something, right?
link |
01:38:48.280
As a user, you don't know.
link |
01:38:49.440
You don't want to know.
link |
01:38:50.440
That was the whole point.
link |
01:38:51.440
You don't want to know how the compiler works.
link |
01:38:52.840
You don't want to know how the memory hierarchy works.
link |
01:38:54.560
You don't want to know how it got parallelized across all these things.
link |
01:38:57.440
You wanted that abstract away from you, but then the magic is lost as soon as you did
link |
01:39:02.240
something and you fall off a performance cliff.
link |
01:39:05.080
And now you're in this funny position where what do I do?
link |
01:39:07.960
I don't change my code.
link |
01:39:08.960
I don't fix that bug.
link |
01:39:11.000
It costs 10 X performance.
link |
01:39:12.000
Now what do I do?
link |
01:39:13.440
Well, this is the problem with unpredictable performance, right?
link |
01:39:16.360
If you care about performance, predictability is a very important thing.
link |
01:39:19.640
And so what the proposal does is it provides architectural patterns for being able to lay
link |
01:39:25.840
out your code, gives you full control over that, makes it really simple so you can explain
link |
01:39:29.920
it, and then if you want to scale out in different ways, you have full control over
link |
01:39:35.560
that.
link |
01:39:36.560
So in your sense, the intuition is for a compiler is too hard to do automated parallelization.
link |
01:39:43.480
Because the compilers do stuff automatically that's incredibly impressive for other things.
link |
01:39:50.560
But for parallelization, we're not even close to there.
link |
01:39:54.560
Well, it depends on the programming model.
link |
01:39:56.240
So there's many different kinds of compilers.
link |
01:39:58.440
And so if you talk about like a C compiler, a Swift compiler or something like that where
link |
01:40:02.560
you're writing imperative code, parallelizing that and reasoning about all the pointers
link |
01:40:07.000
and stuff like that is a very difficult problem.
link |
01:40:10.200
Now, if you switch domains, so there's this cool thing called machine learning, right?
link |
01:40:15.640
So machine learning nerds among other endearing things like solving cat detectors and other
link |
01:40:21.160
things like that have done this amazing breakthrough of producing a programming model, operations
link |
01:40:28.120
that you can pose together that has raised levels of abstraction high enough that suddenly
link |
01:40:33.520
you can have auto parallelizing compilers.
link |
01:40:36.880
You can write a model using TensorFlow and have it run on 1,024 nodes of a TPU.
link |
01:40:42.680
Yeah, that's true.
link |
01:40:44.080
I didn't even think about like, you know, because there's so much flexibility in the
link |
01:40:48.280
design of architectures that ultimately boiled down to a graph that's parallelizable for
link |
01:40:52.800
you, paralyzed for you.
link |
01:40:54.520
And if you think about it, that's pretty cool, right?
link |
01:40:57.800
And you think about batching, for example, as a way of being able to exploit more parallelism.
link |
01:41:02.000
Yeah.
link |
01:41:03.000
Like that's a very simple thing that now is very powerful.
link |
01:41:05.480
It didn't come out of the programming language, nerds, right, those people, like that came
link |
01:41:09.320
out of people that are just looking to solve a problem and use a few GPUs and organically
link |
01:41:13.520
developed by the community of people focusing on machine learning as an incredibly powerful
link |
01:41:18.720
abstraction layer that enables the compiler people to go and exploit that.
link |
01:41:22.960
And now you can drive supercomputers from Python.
link |
01:41:26.960
That's pretty cool.
link |
01:41:27.960
That's amazing.
link |
01:41:28.960
So just to pause on that, because I'm not sufficiently low level, I forget to admire the beauty and
link |
01:41:34.360
power of that, but maybe just to linger on it, like what does it take to run in your
link |
01:41:40.960
network fast?
link |
01:41:41.960
Like how hard is that compilation?
link |
01:41:44.040
It's really hard.
link |
01:41:45.040
So we just skipped, you said like it's amazing that that's a thing, but yeah, how hard is
link |
01:41:50.560
that of a thing?
link |
01:41:51.560
It's hard.
link |
01:41:52.560
And I would say that not all of the systems are really great, including the ones I helped
link |
01:41:58.120
build.
link |
01:41:59.120
So there's a lot of work left to be done there.
link |
01:42:00.600
Is it the compiler nerds working on that, or is it a whole new group of people?
link |
01:42:04.520
Well, it's a full stack problem, including compiler people, including APIs, so like Keras
link |
01:42:09.880
and the module API and PyTorch and Jax, and there's a bunch of people pushing on all the
link |
01:42:16.320
different parts of these things, because when you look at it, it's both how do I express
link |
01:42:20.240
the computation?
link |
01:42:21.240
Do I stack up layers?
link |
01:42:22.240
Well, cool.
link |
01:42:23.240
Like setting up a linear sequence of layers is great for the simple case, but how do
link |
01:42:26.920
I do the hard case?
link |
01:42:27.920
How do I do reinforcement learning?
link |
01:42:29.440
Well, now I need to integrate my application logic in this, right?
link |
01:42:32.880
Then it's, you know, the next level down of how do you represent that for the runtime?
link |
01:42:36.760
How do you get hardware abstraction?
link |
01:42:39.200
And then you get to the next level down of saying like, forget about abstraction.
link |
01:42:41.960
How do I get the peak performance out of my TPU or my iPhone accelerator or whatever,
link |
01:42:47.400
right?
link |
01:42:48.400
And all these different things.
link |
01:42:49.400
And so this is a layered problem with a lot of really interesting design and work going
link |
01:42:53.800
on in the space and a lot of really smart people working on it.
link |
01:42:57.160
Application learning is a very well funded area of investment right now.
link |
01:43:01.000
And so there's a lot of progress being made.
link |
01:43:03.040
So how much innovation is there on the lower level, so closer to the ASIC, so redesigning
link |
01:43:09.080
the hardware or redesigning concurrently compilers with that hardware?
link |
01:43:13.240
Is that, like if you were to predict the biggest, you know, the equivalent of Moore's law improvements
link |
01:43:20.520
in the inference and the training of neural networks and just all of that, where is that
link |
01:43:26.080
going to come from, you think?
link |
01:43:27.080
So you get scalability.
link |
01:43:28.080
You have different things.
link |
01:43:29.080
And so you get, you know, Jim Keller shrinking process technology, you get three nanometer
link |
01:43:35.080
instead of five or seven or 10 or 28 or whatever.
link |
01:43:38.480
And so that that marches forward and that provides improvements.
link |
01:43:41.440
You get architectural level performance.
link |
01:43:44.120
And so the, you know, a TPU with a matrix multiply unit and a systolic array is much
link |
01:43:48.920
more efficient than having a scaler core doing multiplies and ads and things like that.
link |
01:43:54.480
You then get system level improvements.
link |
01:43:58.760
So how you talk to memory, how you talk across a cluster of machines, how you scale out, how
link |
01:44:03.400
you have fast interconnects between machines, you then get system level programming models.
link |
01:44:08.840
So now that you have all this hardware, how do you utilize it?
link |
01:44:11.400
You then have algorithmic breakthroughs where you say, hey, wow, cool.
link |
01:44:14.680
Instead of training in, you know, ResNet 50 and a week, I'm now training it in, you know,
link |
01:44:20.560
25 seconds.
link |
01:44:21.560
Yeah.
link |
01:44:22.560
And now opening up.
link |
01:44:23.560
And this is a combination of, you know, new, new optimizers and new, new, new just training
link |
01:44:29.000
regimens and different, different approaches to train and, and all of these things come
link |
01:44:33.680
together to push the world forward.
link |
01:44:36.240
That was a beautiful exposition.
link |
01:44:40.160
But if you were to force to bet all your money on one of these, would you?
link |
01:44:45.360
Why do we have to?
link |
01:44:46.360
That's that.
link |
01:44:47.360
Fortunately, we have people working on all of this.
link |
01:44:50.920
It's an exciting time, right?
link |
01:44:52.360
So I mean, you know, open the eye, did this little paper showing the algorithmic improvement
link |
01:44:57.520
you can get.
link |
01:44:58.520
It's been, you know, improving exponentially.
link |
01:45:00.920
I haven't quite seen the same kind of analysis on other layers of the stack, I'm sure it's
link |
01:45:07.200
also improving significantly.
link |
01:45:09.120
I just, it's a, it's a nice intuition builder.
link |
01:45:12.280
I mean, there's a reason why Moore's law, that's the beauty of Moore's law is somebody
link |
01:45:18.480
writes a paper that makes a ridiculous prediction.
link |
01:45:21.480
Yeah.
link |
01:45:23.000
And it, you know, becomes reality in a sense.
link |
01:45:27.640
There's something about these narratives when you, when Chris Lattener on a silly little
link |
01:45:33.000
podcast makes all bets, all his money on a particular thing, somehow it can have a ripple
link |
01:45:38.880
effect of actually becoming real.
link |
01:45:40.960
That's an interesting aspect of it because like, it might have been, you know, we focused
link |
01:45:46.680
with Moore's law, most of the computing industry really, really focused on the hardware.
link |
01:45:52.920
I mean, software innovation, I don't know how much software innovation there was in
link |
01:45:56.800
terms of efficiency.
link |
01:45:57.800
What Intel give a bill takes away, right?
link |
01:45:59.920
Yeah.
link |
01:46:00.920
I mean, compilers improved significantly also.
link |
01:46:03.440
All right.
link |
01:46:04.440
Well, not really.
link |
01:46:05.440
So actually, I mean, some, I'm joking about how software's gotten slower pretty much as
link |
01:46:09.880
fast as hardware got better, at least through the nineties.
link |
01:46:13.400
There's another joke, another law in compilers, which is called, I think it's called Probstein's
link |
01:46:17.880
law, which is compilers double the performance of any given code every 18 years.
link |
01:46:26.400
So they move slowly.
link |
01:46:27.400
Yeah.
link |
01:46:28.400
Well, so well, yeah, it's exponential also.
link |
01:46:30.960
Yeah.
link |
01:46:31.960
You're making progress, but there again, it's not about the power of compilers is not just
link |
01:46:37.160
about how do you make the same thing go faster.
link |
01:46:39.360
It's how do you unlock the new hardware?
link |
01:46:41.840
A new chip came out.
link |
01:46:43.840
How do you utilize it?
link |
01:46:44.840
You say, oh, the programming model, how do we make people more productive?
link |
01:46:47.200
How do we have better error messages?
link |
01:46:52.280
Even such mundane things like how do I generate a very specific error message about your code?
link |
01:46:58.200
It actually makes people happy because then they know how to fix it, right?
link |
01:47:02.000
And it comes back to how do you help people get their job done?
link |
01:47:04.680
Yeah.
link |
01:47:05.680
And yeah.
link |
01:47:06.680
It's a world of exponentially increasing smart toasters, how do you expand computing
link |
01:47:13.000
to all these kinds of devices?
link |
01:47:16.560
Do you see this world where just everything's a computing surface?
link |
01:47:20.560
You see that possibility?
link |
01:47:22.400
Just everything's a computer?
link |
01:47:23.640
Yeah.
link |
01:47:24.640
I don't see any reason that that couldn't be achieved.
link |
01:47:26.760
It turns out that sand goes into glass and glass is pretty useful too.
link |
01:47:33.040
And, you know, why not?
link |
01:47:35.400
Why not?
link |
01:47:36.400
Yeah.
link |
01:47:37.400
A very important question then if we're living in a simulation and the simulation is running
link |
01:47:46.040
a computer, like what's the architecture of that computer, do you think?
link |
01:47:52.000
So you're saying is it a quantum system?
link |
01:47:54.680
Is it a...
link |
01:47:55.680
Yeah, like this whole quantum discussion, is it needed?
link |
01:47:57.680
Or can we run it with a risk five architecture, a bunch of CPUs?
link |
01:48:05.320
I think it comes down to the right tool for the job.
link |
01:48:08.080
And so...
link |
01:48:09.080
And what's the compiler?
link |
01:48:10.080
Yeah.
link |
01:48:11.080
Exactly.
link |
01:48:12.080
That's my question.
link |
01:48:13.080
Did I get that job?
link |
01:48:14.080
Made the universe compiler.
link |
01:48:16.920
And so there, as far as we know, quantum systems are the bottom of the pile of turtles so far.
link |
01:48:25.720
And so we don't know efficient ways to implement quantum systems without using quantum computers.
link |
01:48:32.440
And that's totally outside of everything we've talked about.
link |
01:48:35.200
But who runs that quantum computer?
link |
01:48:37.160
Yeah.
link |
01:48:38.160
Right.
link |
01:48:39.160
So if we really are living in a simulation, then is it bigger quantum computers?
link |
01:48:44.440
Is it different ones?
link |
01:48:45.440
Like how does that work?
link |
01:48:46.440
How does that scale?
link |
01:48:47.440
Well, it's the same size.
link |
01:48:50.040
It's the same size.
link |
01:48:51.040
But then the thought of the simulation is that you don't have to run the whole thing
link |
01:48:54.200
that, you know, we humans are cognitively very limited.
link |
01:48:56.880
Do you do checkpoints?
link |
01:48:57.880
Checkpoints, yeah.
link |
01:49:00.880
And if we, the point at which we humans, so you basically do minimal amount of, what
link |
01:49:06.960
is it, the SWIFT does, unwrite, copy unwrite.
link |
01:49:12.280
So you only adjust the simulation.
link |
01:49:15.760
Parallel universe theories, right?
link |
01:49:17.640
And so every time a decision's made, somebody opens the shorting in your box, then there's
link |
01:49:23.040
a fork.
link |
01:49:24.040
And then this could happen.
link |
01:49:25.920
And then thank you for considering the possibility.
link |
01:49:30.440
But yeah.
link |
01:49:31.440
So it may not require, you know, the entirety of the universe to simulate it, but it's interesting
link |
01:49:36.440
to think about as we create this, this higher and higher fidelity systems.
link |
01:49:43.600
But I do want to ask on the, on the quantum computer side, because everything we've talked
link |
01:49:47.560
about with us, with your work with sci fi, with everything with compilers, none of that
link |
01:49:52.960
includes quantum computers, right?
link |
01:49:55.160
That's true.
link |
01:49:56.160
So if you ever thought about what a, you know, this whole serious engineering work of quantum
link |
01:50:05.760
computers looks like of compilers, of architectures, all of that kind of stuff.
link |
01:50:10.760
So I've looked at it a little bit.
link |
01:50:11.760
I'd know almost nothing about it, which means that at some point I will have to find an excuse
link |
01:50:16.720
to get involved.
link |
01:50:17.720
Because that's how it works.
link |
01:50:18.720
But do you think, do you think that's the thing to be, like, is, with your little tingly
link |
01:50:22.840
senses of the timing of when to be involved?
link |
01:50:26.360
Is it not yet?
link |
01:50:27.360
Well, so, so the thing I do really well is I jump into messy systems and figure out how
link |
01:50:32.120
to make them figure out what the truth in the situation is, try to figure out what, what
link |
01:50:37.760
the unifying theory is, how to like factor the complexity, how to find a beautiful answer
link |
01:50:42.080
to a problem that has been well studied and lots of people have bashed their heads against
link |
01:50:46.400
it.
link |
01:50:47.400
I don't know that quantum computers are mature enough and accessible enough to be figured
link |
01:50:52.760
out yet.
link |
01:50:53.760
Right.
link |
01:50:54.760
And the, I think the open question with quantum computers is, is there a useful problem that
link |
01:51:00.200
gets solved with a quantum computer that makes it worth the economic cost of like having
link |
01:51:05.880
one of these things and having, having legions of people that, that, that set it up.
link |
01:51:11.560
You go back to the fifties, right?
link |
01:51:12.840
And there's the projections of the world can, will only need seven, seven computers.
link |
01:51:17.040
Right.
link |
01:51:18.040
Yeah.
link |
01:51:19.040
Well, and part of that was that people hadn't figured out what's useful for.
link |
01:51:22.120
What are the algorithms we want to run?
link |
01:51:23.320
What are the problems that get solved?
link |
01:51:24.320
And this comes back to how do we make the world better either economically or making
link |
01:51:28.920
somebody's life better or like solving a problem that wasn't solved before, things like this.
link |
01:51:33.440
And I think that just we're a little bit too early in that development cycle because it's
link |
01:51:37.080
still like literally a science project, not a negative connotation, right?
link |
01:51:41.640
It's literally a science project and the progress there's amazing.
link |
01:51:45.560
And so I don't know if it's 10 years away, if it's two years away, exactly where that
link |
01:51:50.760
breakthrough happens, but you look at machine learning, it, we went through a few winners
link |
01:51:58.600
before the AlexNet transition and then suddenly it had its breakout moment and that was the
link |
01:52:03.720
catalyst that then drove the talent flocking into it.
link |
01:52:07.800
That's what drove the economic applications of it.
link |
01:52:10.320
That's what drove the, the technology to go faster because you know, have more minds
link |
01:52:14.640
thrown at the problem.
link |
01:52:16.000
This is what caused like a serious knee and deep learning and the algorithms that we're
link |
01:52:21.800
using.
link |
01:52:22.800
And, and so I think that's what quantum needs to go through.
link |
01:52:25.600
And so right now it's in that, that formidable finding itself, getting the, the, like literally
link |
01:52:30.520
the physics figured out.
link |
01:52:33.320
And then it has to figure out the application that makes it so that's useful.
link |
01:52:37.480
Yeah, but I'm not skeptical that I think that will happen.
link |
01:52:40.800
I think it's just, you know, 10 years away, something like that.
link |
01:52:43.480
I forgot to ask what programming language do you think the simulation is written in?
link |
01:52:49.160
Probably Lisp.
link |
01:52:50.160
So not Swift, like if you were to bet, you were to bet, I'll just leave it at that.
link |
01:52:58.120
So I mean, we've mentioned that you worked with all these companies with, with talked
link |
01:53:01.760
about all these projects.
link |
01:53:04.040
It's kind of, if we just step back and zoom out about the way you did that work and we
link |
01:53:10.320
look at COVID times, this pandemic we're living through, that may, if I look at the
link |
01:53:15.760
way Silicon Valley folks are talking about it, the way MIT is talking about it, this
link |
01:53:20.280
might last for a long time, not just the virus, but the, the remote nature.
link |
01:53:28.400
The economic impact.
link |
01:53:29.400
Yeah.
link |
01:53:30.400
Yeah.
link |
01:53:31.400
It's going to be a mess.
link |
01:53:32.400
Do you think, what's your prediction?
link |
01:53:34.480
I mean, from sci fi to Google to, to just all the places you worked in, just Silicon
link |
01:53:42.960
Valley, you're in the middle of it.
link |
01:53:44.360
What do you think is, how is this whole place going to change?
link |
01:53:46.560
Yeah.
link |
01:53:47.560
So, I mean, I, I really can only speak to the tech perspective.
link |
01:53:50.440
I am in that bubble.
link |
01:53:54.160
I think it's going to be really interesting because the, you know, the zoom culture of
link |
01:53:58.160
being remote and on video chat all the time has really interesting effects on people.
link |
01:54:02.120
So on the one hand, it's a great normalizer.
link |
01:54:05.120
It's a normalizer that I think will help communities of people that have traditionally
link |
01:54:10.600
been underrepresented because now you're taking, in some cases, a face off because you don't
link |
01:54:16.600
have to have a camera going, right?
link |
01:54:18.880
And so you can have conversations without physical appearance being part of the, part
link |
01:54:21.840
of the dynamic, which is pretty powerful.
link |
01:54:24.600
You're taking remote employees that have already been remote and you're saying you're
link |
01:54:28.080
now on the same level and footing as everybody else, nobody gets whiteboards.
link |
01:54:33.560
You're not going to be the one person that doesn't get to be participating in the whiteboard
link |
01:54:36.360
conversation and that's pretty powerful.
link |
01:54:39.440
You've got, you're forcing people to think asynchronously in some cases because it's
link |
01:54:45.320
hard to just, just get people physically together and the bumping into each other forces people
link |
01:54:50.000
to find new ways to solve those problems.
link |
01:54:52.880
And I think that that leads to more inclusive behavior, which is good.
link |
01:54:56.800
On the other hand, it's also, it just sucks, right?
link |
01:55:01.040
And so the, the nature, the, the actual communication or it just sucks being not in, with people
link |
01:55:08.600
like on a daily basis and collaborating with them.
link |
01:55:11.680
Yeah.
link |
01:55:12.680
All of that.
link |
01:55:13.680
I mean, everything, this whole situation is terrible.
link |
01:55:15.720
What I meant primarily was the, I think that the most humans like working physically with
link |
01:55:22.280
humans.
link |
01:55:23.280
And this is something that not everybody, but many people are programmed to do.
link |
01:55:27.400
And I think that we get something out of that that is very hard to express, at least for
link |
01:55:30.640
me.
link |
01:55:31.640
And so maybe this isn't true of everybody, but, and so the question to me is, you know,
link |
01:55:37.000
when you get through that time of adaptation, right, you get out of March and April and
link |
01:55:41.960
you get into December and you get into next March, if it's not changed, right?
link |
01:55:46.400
It's already terrifying.
link |
01:55:47.880
Well, you think about that and you think about what is the nature of work and how do, how
link |
01:55:52.000
do we adapt?
link |
01:55:53.000
Well, humans are very adaptable species, right?
link |
01:55:54.960
And we can, we can learn things.
link |
01:55:57.320
And when we're forced to, and there's a catalyst to make that happen.
link |
01:56:00.680
And so what is it that comes out of this and are we better or worse off, right?
link |
01:56:04.640
I think that, you know, you look at the Bay Area, housing prices are insane.
link |
01:56:08.600
Well, why?
link |
01:56:09.600
Well, there's a high incentive to be physically located because if you don't have proximity,
link |
01:56:15.280
you end up paying for it and commute, right?
link |
01:56:18.520
And there's, there has been huge social pressure in terms of like, you will be there for the
link |
01:56:23.680
meeting, right?
link |
01:56:24.680
Or whatever scenario it is.
link |
01:56:26.680
And I think that's going to be way better.
link |
01:56:28.200
I think it's going to be much more of the norm to have remote employees and I think
link |
01:56:31.880
this is going to be really great.
link |
01:56:33.400
Do you, do you have friends or do you hear of people moving?
link |
01:56:37.200
I know one family friend that moved, they moved back to Michigan and, you know, they
link |
01:56:43.720
were a family with three kids living in a small apartment and like, we're going insane,
link |
01:56:49.480
right?
link |
01:56:50.960
And they're in tech, husband works for Google.
link |
01:56:54.360
So first of all, friends of mine have, are in the process of or are, have already lost
link |
01:56:59.880
the business.
link |
01:57:00.880
The thing that represents their passion, their dream.
link |
01:57:03.440
It could be small entrepreneur projects, but it could be large businesses like people that
link |
01:57:07.240
run gyms, like restaurants, like tons of things.
link |
01:57:10.400
So, but also just people like look at themselves in the mirror and ask the question of like,
link |
01:57:16.280
what do I want to do in life?
link |
01:57:17.680
For some reason they don't, haven't done it until COVID like, they really ask that question
link |
01:57:21.960
and that results often in moving or leaving the company here with starting your business
link |
01:57:27.960
or transitioning to a different company.
link |
01:57:30.640
Do you think we're going to see that a lot?
link |
01:57:34.240
Like, well, I can't speak to that.
link |
01:57:36.760
I mean, we're definitely going to see it at a higher frequency than we did before.
link |
01:57:40.080
Just because I think what you're trying to say is there are decisions that you make yourself
link |
01:57:45.760
and big life decisions that you make yourself.
link |
01:57:47.920
And like, I'm going to like quit my job and start a new thing.
link |
01:57:50.560
There's also decisions that get made for you.
link |
01:57:53.000
Like I got fired from my job.
link |
01:57:54.720
What am I going to do?
link |
01:57:55.720
Right?
link |
01:57:56.720
And that's not a decision that you think about, but you're forced to act.
link |
01:58:01.040
And so I think that those you're forced to act kind of moments where like, you know,
link |
01:58:05.240
the global pandemic comes and wipes out the economy and now your business doesn't exist.
link |
01:58:10.400
I think that does lead to more reflection, right?
link |
01:58:12.560
Because you're less anchored on what you have.
link |
01:58:15.080
And it's not a, what do I have to lose versus what do I have to gain AB comparison?
link |
01:58:20.600
It's more of a fresh slate.
link |
01:58:22.080
Cool.
link |
01:58:23.080
I could do anything now.
link |
01:58:24.440
Do I want to do the same thing I was doing?
link |
01:58:26.520
Did that make me happy?
link |
01:58:28.440
Is this now time to go back to college and take a class and learn new skill?
link |
01:58:33.200
Is this, is this a time to spend time with family?
link |
01:58:36.320
If you can afford to do that, is this time to like, you know, literally move into the
link |
01:58:40.440
parents?
link |
01:58:41.440
Right?
link |
01:58:42.440
I mean, all these things that were not normative before suddenly become, I think, very, the
link |
01:58:47.760
value system has changed.
link |
01:58:48.760
And I think that's actually a good thing in the short term, at least, because it leads
link |
01:58:54.440
to, you know, there's kind of been an over optimization along one, one set of priorities
link |
01:59:00.680
for the world, and now maybe we'll get to a more balanced and more interesting world
link |
01:59:05.120
where people are doing different things.
link |
01:59:06.760
I think it could be good.
link |
01:59:07.760
I think there could be more innovation that comes out of it, for example.
link |
01:59:10.440
What do you think about all the social chaos when in the middle of like,
link |
01:59:13.840
It sucks.
link |
01:59:14.840
You think it's, let me ask you, I hope, do you think it's all going to be okay?
link |
01:59:20.600
Well, I think humanity will survive.
link |
01:59:23.720
The, from the next extension, look, we're not all going to kill, yeah, well.
link |
01:59:27.160
Yeah.
link |
01:59:28.160
I don't think the virus is going to kill all the humans.
link |
01:59:29.560
I don't think all the humans are going to kill all the humans.
link |
01:59:31.960
I think that's unlikely, but I look at it as progress requires a catalyst, right?
link |
01:59:42.520
So you need a reason for people to be willing to do things that are uncomfortable.
link |
01:59:47.720
I think that the U.S., at least, but I think the world in general is a pretty unoptimal
link |
01:59:54.840
place to live in for a lot of people.
link |
01:59:56.880
And I think that what we're seeing right now is we're seeing a lot of unhappiness.
link |
02:00:00.680
And because of all the pressure, because of all the badness in the world that's coming
link |
02:00:05.720
together, it's really kind of igniting some of that debate that should have happened a
link |
02:00:09.000
long time ago, right?
link |
02:00:10.000
I mean, I think that we'll see more progress.
link |
02:00:11.680
You're asking about, offline, you're asking about politics and wouldn't be great if politics
link |
02:00:15.160
moved faster because there's all these problems in the world and we can move it.
link |
02:00:18.280
Well, people are intentional or inherently conservative.
link |
02:00:22.400
And so if you're talking about conservative people, particularly if they have heavy burdens
link |
02:00:26.640
on their shoulders, because they represent literally thousands of people, it makes sense
link |
02:00:32.200
to be conservative.
link |
02:00:33.200
But on the other hand, when you need change, how do you get it?
link |
02:00:37.280
The global pandemic will probably lead to some change.
link |
02:00:40.840
And it's not a directed plan, but I think that it leads to people asking really interesting
link |
02:00:46.880
questions.
link |
02:00:47.880
And some of those questions should have been asked a long time ago.
link |
02:00:50.200
Well, let me know if you've observed this as well.
link |
02:00:53.520
Something that's bothered me in the machine learning community, I'm guessing it might
link |
02:00:56.960
be prevalent in other places, is something that feels like in 2020 increased level of
link |
02:01:03.920
toxicity.
link |
02:01:05.920
People are just quicker to pile on, they're just harsh on each other to mob, pick a person
link |
02:01:17.120
that screwed up and make it a big thing.
link |
02:01:26.480
Have you observed that in other places?
link |
02:01:29.000
Is there some way out of this?
link |
02:01:30.160
I think there's an inherent thing in humanity that's kind of an us versus them thing, which
link |
02:01:34.600
is that you want to succeed and how do you succeed?
link |
02:01:37.200
Well, it's relative to somebody else.
link |
02:01:39.720
And so what's happening, at least in some part, is that with the internet and with online
link |
02:01:45.880
communication, the world's getting smaller, and so we're having some of the social ties
link |
02:01:52.880
of my town versus your town's football team turn into much larger and yet shallower problems.
link |
02:02:03.320
And people don't have time, the incentives, clickbait and all these things really feed
link |
02:02:09.440
into this machine, and I don't know where that goes.
link |
02:02:12.080
Yeah, the reason I think about that, I mentioned to you this offline a little bit, but I have
link |
02:02:19.760
a few difficult conversations scheduled, some of them political related, some of them within
link |
02:02:26.280
the community, difficult personalities that went through some stuff, I mean, one of them
link |
02:02:31.200
I've talked before, we'll talk again is Yann LeCun, he got a little crap on Twitter for
link |
02:02:39.160
talking about a particular paper and the bias within a data set.
link |
02:02:43.000
And then there's been a huge in my view, and I'm only comfortable saying it irrational,
link |
02:02:51.280
over exaggerated pile on on his comments, because he made pretty basic comments about
link |
02:02:57.240
the fact that if there's bias in the data, there's going to be bias in the results.
link |
02:03:02.480
So we should not have bias in the data, but people piled on to him because he said he
link |
02:03:07.360
trivialized the problem of bias, like it's a lot more than just bias in the data.
link |
02:03:13.320
But like, yes, that's a very good point.
link |
02:03:16.640
But that's that's not what he was saying, it's not what he was saying.
link |
02:03:19.880
And the response, like the implied response that he's basically sexist and racist is something
link |
02:03:28.840
that completely drives away the possibility of nuanced discussion.
link |
02:03:33.160
One nice thing about like a pocket long form conversation is you can talk it out.
link |
02:03:40.360
You can lay your reasoning out.
link |
02:03:42.960
And even if you're wrong, you can still show that you're a good human being underneath
link |
02:03:47.800
it.
link |
02:03:48.800
You know, your point about you can't have a productive discussion.
link |
02:03:50.600
Well, how do you get to that point where people can turn, they can learn, they can listen,
link |
02:03:55.160
they can think, they can engage versus just being a shallow like, like, and then keep
link |
02:04:01.120
moving, right?
link |
02:04:02.120
And I don't think that that progress really comes from that, right?
link |
02:04:07.080
And I don't think that one should expect that.
link |
02:04:09.800
I think that you'd see that as reinforcing individual circles and the us versus them
link |
02:04:14.080
thing.
link |
02:04:15.080
And I think that's fairly divisive.
link |
02:04:17.080
Yeah, I think there's a big role in like the people that bother me most on Twitter when
link |
02:04:24.200
I observe things is not the people who get very emotional, angry, like over the top.
link |
02:04:30.280
It's the people who like prop them up.
link |
02:04:34.000
It's all the, it's, it's, I think what should be the, we should teach each other is to be
link |
02:04:40.600
sort of empathetic.
link |
02:04:42.600
The thing that it's really easy to forget, particularly on like Twitter or the internet
link |
02:04:46.560
or the email, is that sometimes people just have a bad day.
link |
02:04:49.760
Yeah.
link |
02:04:50.760
Right.
link |
02:04:51.760
You have a bad day or you're like, I've been in this situation where it's like between
link |
02:04:55.000
meetings, like fire off a quick response in emails because I want to like help get
link |
02:04:58.640
something unblocked phrase it really objectively wrong.
link |
02:05:03.600
I screwed up and suddenly this is now something that sticks with people and it's not because
link |
02:05:09.600
they're bad.
link |
02:05:10.600
It's not because you're bad, just psychology of like, you said a thing, it sticks with
link |
02:05:15.840
you.
link |
02:05:16.840
You didn't mean it that way, but it really impacted somebody because the way they interpreted
link |
02:05:20.600
it.
link |
02:05:21.600
And this is just an aspect of working together as humans and I have a lot of optimism in
link |
02:05:25.360
the long term, the very long term about what we as humanity can do, but I think that's
link |
02:05:29.600
going to be, it's just always a rough ride.
link |
02:05:31.280
And you came into this by saying like, what is COVID and all the social strife that's
link |
02:05:36.280
happening right now mean?
link |
02:05:38.400
And I think that it's really bad in the short term, but I think it will lead to progress.
link |
02:05:42.680
And for that, I'm very thankful.
link |
02:05:45.000
Yeah.
link |
02:05:46.000
It's painful in the short term though.
link |
02:05:47.840
Well, yeah.
link |
02:05:48.840
I mean, people are out of jobs, like some people can't eat, like it's horrible.
link |
02:05:54.960
But you know, it's progress.
link |
02:05:57.080
So we'll see what happens.
link |
02:05:58.520
I mean, the real question is when you look back 10 years, 20 years, 100 years from now,
link |
02:06:03.640
how do we evaluate the decisions that are being made right now?
link |
02:06:06.800
I think that's really the way you can frame that and look at it.
link |
02:06:10.480
And you say, you know, you integrate across all the short term horribleness that's happening
link |
02:06:15.360
and you look at what that means and is the, you know, improvement across the world or
link |
02:06:19.640
the regression across the world significant enough to make it a good or a bad thing.
link |
02:06:24.640
I think that's the question.
link |
02:06:26.280
Yeah.
link |
02:06:27.280
And for that, it's good to study history.
link |
02:06:29.480
I mean, one of the big problems for me right now is I'm reading the rise and fall of the
link |
02:06:33.920
third Reich.
link |
02:06:34.920
Okay.
link |
02:06:35.920
Light reading.
link |
02:06:36.920
So it's everything is just, I just see parallels and it means it's, it's, you have to be really
link |
02:06:42.920
careful not to overstep it, but just the thing that worries me the most is the pain that
link |
02:06:49.440
people feel when a few things combined, which is like economic depression, which is quite
link |
02:06:56.440
possible in this country.
link |
02:06:58.240
And then just being disrespected by, in some kind of way, which the German people were
link |
02:07:04.040
really disrespected by most of the world, like in a way that's over the top, that something
link |
02:07:10.880
can build up and then all you need is a charismatic leader just to go either positive or negative
link |
02:07:18.280
in both work as long as they're charismatic.
link |
02:07:21.360
And it's taking advantage of, again, that, that inflection point that the world's in
link |
02:07:26.320
and what they do with it could be good or bad.
link |
02:07:29.960
And so it's, it's a good way to think about times now, like on the individual level, what
link |
02:07:34.880
we decide to do is when, when history is written, you know, 30 years from now, what happened
link |
02:07:40.040
in 2020, probably history is going to remember 2020 either for good or bad.
link |
02:07:47.000
And it's like up to us to write it, so it's good.
link |
02:07:49.480
Well, one of the things I've observed that I find fascinating is most people act as
link |
02:07:54.240
though the world doesn't change.
link |
02:07:56.520
You make decision knowingly, right, you make a decision where you're predicting the future
link |
02:08:02.480
based on what you've seen in the recent past.
link |
02:08:04.880
And so if something's always been, it's rained every single day, then of course you expect
link |
02:08:08.080
it to rain today too, right?
link |
02:08:10.120
On the other hand, the world changes all the time, yeah, constantly, like for better and
link |
02:08:15.720
for worse.
link |
02:08:17.040
And so the question is, if you're interested in something that's not right, what is the
link |
02:08:21.200
inflection point that led to a change and you can look to history for this?
link |
02:08:24.480
Like what is, what is the catalyst that led to that, that explosion that led to that bill
link |
02:08:28.960
that led to the, like you, you can kind of work your way backwards from that.
link |
02:08:33.480
And maybe if you pull together the right people and you get the right ideas together, you
link |
02:08:37.080
can actually start driving that change and doing it in a way that's productive and hurts
link |
02:08:40.760
fewer people.
link |
02:08:41.760
Yeah, like a single person, a single event can turn all of this.
link |
02:08:44.960
Absolutely.
link |
02:08:45.960
Everything starts somewhere.
link |
02:08:46.960
And often it's a combination of multiple factors, but, but yeah, this is, these, these
link |
02:08:50.840
things can be engineered.
link |
02:08:52.720
That's actually the optimistic view that I'm a long term optimist on pretty much everything
link |
02:08:57.440
and human nature, you know, we can look to all the negative things that the humanity
link |
02:09:01.680
has all the pettiness and all the like self self servingness and the just the, the cruelty,
link |
02:09:08.960
right?
link |
02:09:09.960
The, the biases, the just humans can be very horrible.
link |
02:09:13.440
But on the other hand, we're capable of amazing things.
link |
02:09:17.400
And, and the progress across, you know, 100 year chunks is striking.
link |
02:09:23.680
And even across decades, it's, we've come a long ways and there's still a long ways
link |
02:09:27.440
to go, but that doesn't mean that we've stopped.
link |
02:09:29.440
Yeah.
link |
02:09:30.440
The kind of stuff we did in the last 100 years is, is unbelievable.
link |
02:09:35.000
It's kind of scary to think what's going to happen this 100 years.
link |
02:09:37.440
It's scary, like exciting, like scary in a sense that it's kind of sad that the kind
link |
02:09:42.400
of technology is going to come out in 10, 20, 30 years will probably too old to really
link |
02:09:47.000
appreciate because you don't grow up with it.
link |
02:09:49.200
It'll be like kids these days with their virtual reality and their tiktoks and stuff
link |
02:09:54.000
like this.
link |
02:09:55.000
Like, how does this thing and like, come on, give me my, you know, static photo, you know,
link |
02:10:00.960
my Commodore 64.
link |
02:10:02.480
Yeah.
link |
02:10:03.480
Exactly.
link |
02:10:04.480
Okay.
link |
02:10:05.480
Sorry, we kind of skipped over.
link |
02:10:06.480
It asks on, you know, the machine learning world has been kind of inspired, their imagination
link |
02:10:15.080
captivated with GPT three and these language models.
link |
02:10:18.720
I thought it'd be cool to get your opinion on it.
link |
02:10:22.360
What's your thoughts on this exciting world of, it connects to computation actually is
link |
02:10:30.240
of language models that are huge and take multiple, many, many computers, not just to
link |
02:10:36.880
train, but to also do inference on.
link |
02:10:39.600
Sure.
link |
02:10:40.600
Well, I mean, it depends on what you're speaking to there, but I mean, I think that there's
link |
02:10:44.400
been a pretty well understood maximum deep learning that if you make the model bigger
link |
02:10:49.520
and you shove more data into it, assuming you train it right and you have a good model
link |
02:10:53.040
architecture that you'll get a better model out.
link |
02:10:55.920
And so I'm wondering on GPT three was not that surprising.
link |
02:10:59.840
On the other hand, a tremendous amount of engineering went into making it possible.
link |
02:11:04.920
The implications of it are pretty huge.
link |
02:11:07.040
I think that when GPT two came out, there was a very provocative blog post from open
link |
02:11:10.960
AI talking about, you know, we're not going to release it because of the social damage
link |
02:11:14.640
it could cause if it's misused.
link |
02:11:18.600
I think that's still a concern.
link |
02:11:20.080
I think that we need to look at how technology is applied and, you know, well meaning tools
link |
02:11:25.120
can be applied in very horrible ways and they can have very profound impact on that.
link |
02:11:30.560
I think the GPT three is a huge technical achievement.
link |
02:11:34.240
And what will GPT four be will probably be bigger, more expensive to train.
link |
02:11:38.880
Really cool architectural tricks.
link |
02:11:42.360
What do you think?
link |
02:11:43.360
Is there, I don't know how much thought you've done on distributed computing.
link |
02:11:48.800
Is there, is there some technical challenges that are interesting that you're hopeful about
link |
02:11:54.000
exploring in terms of, you know, a system that like a piece of code that, you know,
link |
02:12:01.200
would GPT four, that might have, I don't know, hundreds of trillions of parameters which
link |
02:12:09.440
have to run on thousands of computers.
link |
02:12:11.600
Is there some, is there some hope that we can make that happen?
link |
02:12:15.120
Yeah.
link |
02:12:16.120
Well, I mean, today you can, you can write a check and get access to a thousand TPU cores
link |
02:12:21.720
and do really interesting large scale training and inference and things like that in Google
link |
02:12:26.040
cloud, for example, right.
link |
02:12:27.600
And so I don't think it's a question about scales, a question about utility.
link |
02:12:33.320
And when I look at the transformer series of architectures that the GPT series is based
link |
02:12:38.080
on, it's really interesting to look at that because they're actually very simple designs.
link |
02:12:43.000
They're not recurrent.
link |
02:12:44.760
The training regimens are pretty simple.
link |
02:12:47.520
And so they don't really reflect like human brains, right, but they're really good at
link |
02:12:52.840
learning language models and they're unrolled enough that you get, you can simulate some
link |
02:12:57.560
recurrence, right.
link |
02:12:59.280
And so the question I think about is, where does this take us?
link |
02:13:03.120
Like, so we can just keep scaling it, have more parameters, more data, more things, we'll
link |
02:13:07.720
get a better result for sure.
link |
02:13:09.600
But are there architectural techniques that can lead to progress at a faster pace, right?
link |
02:13:15.320
This is when, you know, how do you get, instead of just like making it a constant time bigger,
link |
02:13:20.720
how do you get like an algorithmic improvement out of this, right?
link |
02:13:24.240
And whether it be a new training regimen, if it becomes sparse networks, for example,
link |
02:13:30.520
human brains sparse, all these networks are dense.
link |
02:13:33.880
The connectivity patterns can be very different.
link |
02:13:36.080
I think this is where I get very interested and I'm way out of my league on the deep learning
link |
02:13:40.640
side of this.
link |
02:13:41.640
But I think that could lead to big breakthroughs.
link |
02:13:43.760
When we talk about large scale networks, one of the things that Jeff Dean likes to talk
link |
02:13:47.600
about and he's given a few talks on is this idea of having a sparsely gated mixture of
link |
02:13:53.640
experts kind of a model where you have, you know, different nets that are trained and
link |
02:13:59.600
are really good at certain kinds of tasks.
link |
02:14:02.240
And so you have this distributor across a cluster and so you have a lot of different
link |
02:14:05.960
computers that end up being kind of locally specialized in different demands.
link |
02:14:09.880
And then when a query comes in, you gait it and use learn techniques to route to different
link |
02:14:14.600
parts of the network.
link |
02:14:15.600
And then you utilize the compute resources of the entire cluster by having specialization
link |
02:14:19.880
within it.
link |
02:14:20.880
And I don't know where that goes or when it starts to work, but I think things like that
link |
02:14:26.640
could be really interesting as well.
link |
02:14:28.520
And on the data side too, if you can think of data selection as a kind of programming.
link |
02:14:35.320
Yeah.
link |
02:14:36.320
I mean, at the center, if you look at like Karpathi talked about software 2.0, I mean,
link |
02:14:41.960
in a sense, data is the programming.
link |
02:14:44.840
So let me try to summarize Andre's position really quick before I disagree with it.
link |
02:14:51.240
So Andre Karpathi is amazing.
link |
02:14:53.520
So this is nothing personal with him.
link |
02:14:55.760
He's an amazing engineer.
link |
02:14:56.960
And also a good blog post writer.
link |
02:14:59.240
Yeah.
link |
02:15:00.240
Well, he's a great communicator.
link |
02:15:01.240
I mean, he's just an amazing person.
link |
02:15:02.760
He's also really sweet.
link |
02:15:04.880
So his basic premise is that software is suboptimal.
link |
02:15:09.360
I think we can all agree to that.
link |
02:15:12.040
He also points out that deep learning and other learning based techniques are really
link |
02:15:16.040
great because you can solve problems in more structured ways with less like ad hoc code
link |
02:15:22.040
that people write out and don't write test cases for in some cases.
link |
02:15:25.200
And so they don't even know if it works in the first place.
link |
02:15:27.960
And so if you start replacing systems of imperative code with deep learning models,
link |
02:15:34.560
then you get a better result.
link |
02:15:38.640
And I think that he argues that software 2.0 is a pervasively learned set of models.
link |
02:15:44.280
And you get away from writing code.
link |
02:15:46.240
And he's given talks where he talks about swapping over more and more and more parts
link |
02:15:50.560
of the code to being learned and driven that way.
link |
02:15:54.800
I think that works.
link |
02:15:56.000
And if you're predisposed to liking machine learning, then I think that that's definitely
link |
02:16:01.320
a good thing.
link |
02:16:02.320
And it's also good for accessibility in many ways because certain people are not going
link |
02:16:05.840
to write C code or something.
link |
02:16:07.880
And so having a data driven approach to do this kind of stuff I think can be very valuable.
link |
02:16:12.840
On the other hand, there are huge trade offs.
link |
02:16:14.320
It's not clear to me that software 2.0 is the answer.
link |
02:16:19.400
And probably Andre wouldn't argue that it's the answer for every problem either.
link |
02:16:23.240
But I look at machine learning as not a replacement for software 1.0.
link |
02:16:27.920
I look at it as a new programming paradigm.
link |
02:16:31.400
And so programming paradigms, when you look across domains, is structured programming
link |
02:16:37.120
where you go tos to if, then, else.
link |
02:16:40.480
Or functional programming from Lisp.
link |
02:16:42.480
And you start talking about higher order functions and values and things like this.
link |
02:16:45.960
Or you talk about object oriented programming, or you're talking about encapsulation, subclassing
link |
02:16:49.920
inheritance.
link |
02:16:50.920
You start talking about generic programming where you start talking about code reuse through
link |
02:16:56.080
specialization and different type instantiations.
link |
02:17:00.240
And you start talking about differentiable programming, something that I am very excited
link |
02:17:04.200
about in the context of machine learning.
link |
02:17:06.120
Talking about taking functions and generating variants like the derivative of another function.
link |
02:17:11.400
That's a programming paradigm that's very useful for solving certain classes of problems.
link |
02:17:16.440
Machine learning is amazing at solving certain classes of problems.
link |
02:17:19.400
You're not going to write a cat detector or even a language translation system by writing
link |
02:17:25.280
C code.
link |
02:17:26.280
That's not a very productive way to do things anymore.
link |
02:17:29.040
And so machine learning is absolutely the right way to do that.
link |
02:17:32.040
In fact, I would say that learned models are really one of the best ways to work with the
link |
02:17:36.280
human world in general.
link |
02:17:38.320
And so anytime you're talking about sensory input of different modalities, anytime that
link |
02:17:41.840
you're talking about generating things in a way that makes sense to a human, I think
link |
02:17:46.040
that learned models are really, really useful.
link |
02:17:48.560
And that's because humans are very difficult to characterize.
link |
02:17:52.760
And so this is a very powerful paradigm for solving classes of problems.
link |
02:17:56.640
But on the other hand, imperative code is two.
link |
02:17:59.760
You're not going to write a bootloader for your computer with a deep learning model.
link |
02:18:04.240
Deep learning models are very hardware intensive.
link |
02:18:07.120
They're very energy intensive because you have a lot of parameters and you can provably
link |
02:18:13.320
implement any function with a learned model like this has been shown.
link |
02:18:17.760
But that doesn't make it efficient.
link |
02:18:20.240
And so if you're talking about caring about a few orders and magnitudes worth of energy
link |
02:18:23.520
usage, then it's useful to have other tools in the toolbox.
link |
02:18:26.720
There's also robustness too.
link |
02:18:28.400
I mean, yeah, exactly.
link |
02:18:29.920
All the problems of dealing with data and bias and data, all the problems of, you know,
link |
02:18:34.440
software 2.0.
link |
02:18:35.440
And one of the great things that Andre is arguing towards, which I completely agree
link |
02:18:40.600
with him, is that when you start implementing things with deep learning, you need to learn
link |
02:18:45.080
from software 1.0 in terms of testing, continuous integration, how you deploy, how do you validate
link |
02:18:51.160
all these things and building systems around that so that you're not just saying, like,
link |
02:18:55.200
ooh, it seems like it's good.
link |
02:18:56.560
Ship it.
link |
02:18:57.560
Right?
link |
02:18:58.560
Well, what happens when I regress something?
link |
02:18:59.880
What happens when I make a classification that's wrong and now I hurt somebody, right?
link |
02:19:05.280
All these things you have to reason about.
link |
02:19:07.280
Yeah.
link |
02:19:08.280
But at the same time, the bootloader that works for us humans looks awfully a lot like
link |
02:19:14.200
a neural network, right?
link |
02:19:15.680
Yeah.
link |
02:19:16.680
It's messy and you can cut out different parts of the brain.
link |
02:19:19.880
There's a lot of this neural plasticity work that shows that it's going to adjust.
link |
02:19:24.320
I mean, it's a really interesting question.
link |
02:19:27.080
How much of the world's programming could be replaced by software 2.0?
link |
02:19:31.880
Like with...
link |
02:19:32.880
Oh, well, I mean, it's provably true that you could replace all of it.
link |
02:19:37.000
Right.
link |
02:19:38.000
So then it's a question of the tradeoffs.
link |
02:19:39.000
Right.
link |
02:19:40.000
So anything that's a function, you can.
link |
02:19:41.000
So it's not a question about if I think it's an economic question.
link |
02:19:45.080
It's a, what kind of talent can you get?
link |
02:19:47.920
What kind of tradeoffs in terms of maintenance, right?
link |
02:19:50.560
Those kinds of questions, I think, what kind of data can you collect?
link |
02:19:53.280
I think one of the reasons that I'm most interested in machine learning as a programming
link |
02:19:58.080
paradigm is that one of the things that we've seen across computing in general is that being
link |
02:20:02.520
laser focused on one paradigm often puts you in a box.
link |
02:20:06.640
It's not super great.
link |
02:20:08.640
And so you look at object engineering programming.
link |
02:20:10.520
Like it was all the rage in the early 80s and like everything has to be objects and
link |
02:20:14.200
people forgot about functional programming even though came first.
link |
02:20:17.720
And then people rediscovered that, hey, if you mix functional and object oriented and
link |
02:20:22.240
structure, like you mix these things together, you can provide very interesting tools that
link |
02:20:25.800
are good at solving different problems.
link |
02:20:28.520
And so the question there is how do you get the best way to solve the problems?
link |
02:20:32.720
It's not about whose tribe should win.
link |
02:20:35.080
Right.
link |
02:20:36.080
It's not about, you know, that shouldn't be the question.
link |
02:20:38.760
The question is how do you make it so that people can solve those problems the fastest
link |
02:20:42.200
and they have the right tools in their box to build good libraries and they can solve
link |
02:20:46.000
these problems.
link |
02:20:47.280
And when you look at that, that's like, you know, you look at reinforcement learning as
link |
02:20:50.440
one really interesting subdomain of this.
link |
02:20:53.160
Reinforcement learning, often you have to have the integration of a learned model combined
link |
02:20:58.080
with your Atari or whatever the other scenario it is that you're working in.
link |
02:21:02.640
You have to combine that thing with the robot control for the arm, right?
link |
02:21:07.760
And so now it's not just about that one paradigm.
link |
02:21:12.040
It's about integrating that with all the other systems that you have, including often legacy
link |
02:21:16.540
systems and things like this, right?
link |
02:21:18.240
And so to me, I think that the interesting thing to say is, like, how do you get the
link |
02:21:22.160
best out of this domain and how do you enable people to achieve things that they otherwise
link |
02:21:26.200
couldn't do without excluding all the good things we already know how to do.
link |
02:21:31.440
Right.
link |
02:21:32.440
But, okay, this is a crazy question, but we talked a lot about GPT3, but do you think
link |
02:21:39.280
it's possible that these language models that, in essence, in the language domain, software
link |
02:21:47.960
2.0 could replace some aspect of compilation, for example, or do program synthesis, replace
link |
02:21:54.560
some aspect of programming?
link |
02:21:56.240
Yeah, absolutely.
link |
02:21:57.680
So I think that the learned models in general are extremely powerful, and I think that people
link |
02:22:02.200
underestimate them.
link |
02:22:05.360
Maybe you can suggest what I should do.
link |
02:22:07.280
If you have access to the GPT3 API, would I be able to generate Swift code, for example?
link |
02:22:14.320
Do you think that could do something interesting?
link |
02:22:16.160
So GPT3 is probably not trained on the right corpus, so it probably has the ability to generate
link |
02:22:23.040
some Swift.
link |
02:22:24.040
I bet it does.
link |
02:22:25.040
It's probably not going to generate a large enough body of Swift to be useful, but take
link |
02:22:29.480
it a next step further.
link |
02:22:31.160
If you had the goal of training something like GPT3 and you wanted to train it to generate
link |
02:22:35.840
source code, it could definitely do that.
link |
02:22:39.440
Now, the question is, how do you express the intent of what you want filled in?
link |
02:22:44.400
You can definitely write a scaffolding of code and say fill in the hole and put in some
link |
02:22:49.840
for loops or put in some classes or whatever, and the power of these models is impressive,
link |
02:22:53.760
but there's an unsolved question, at least unsolved to me, which is, how do I express
link |
02:22:57.960
the intent of what to fill in?
link |
02:23:01.800
What you'd really want to have, and I don't know that these models are up to the task,
link |
02:23:06.320
is you want to be able to say, here's a scaffolding, and here are the assertions at the end, and
link |
02:23:12.640
the assertions always pass, and so you want a generative model, on the one hand, yes.
link |
02:23:16.640
Oh, that's fascinating.
link |
02:23:17.640
Yeah.
link |
02:23:18.640
Right?
link |
02:23:19.640
But you also want some loopback, some reinforcement learning system or something where you're actually
link |
02:23:24.040
saying, I need to hill climb towards something that is more correct, and I don't know that
link |
02:23:29.000
we have that.
link |
02:23:30.000
Well, it would generate not only a bunch of the code, but the checks that do the testing.
link |
02:23:35.960
It would generate the tests.
link |
02:23:37.080
I think the humans would generate the tests, right?
link |
02:23:38.920
Oh, okay.
link |
02:23:39.920
The test would grow.
link |
02:23:40.920
But it would be fascinating if...
link |
02:23:41.920
Well, the tests are the requirements.
link |
02:23:42.920
Yes, but the...
link |
02:23:43.920
Okay, so...
link |
02:23:44.920
Because you have to express to the model what you want to...
link |
02:23:47.160
You don't just want gibberish code.
link |
02:23:49.120
Look at how compelling this code looks.
link |
02:23:51.400
You want a story about four horned unicorns or something.
link |
02:23:54.480
Well, okay, so exactly.
link |
02:23:56.080
That's human requirements, but then I thought it's a compelling idea that the GPT4 model
link |
02:24:03.560
could generate checks that are more high fidelity that check for correctness.
link |
02:24:13.000
Because the code it generates, say I ask it to generate a function that gives me the Fibonacci
link |
02:24:20.720
sequence.
link |
02:24:21.720
Sure.
link |
02:24:22.720
I don't...
link |
02:24:23.720
So, decompose the problem, right, so you have two things.
link |
02:24:27.080
You have...
link |
02:24:28.080
You need the ability to generate syntactically corrects with code that's interesting, right?
link |
02:24:33.040
I think GPT series of model architectures can do that.
link |
02:24:37.120
But then you need the ability to add the requirements.
link |
02:24:41.440
So generate Fibonacci.
link |
02:24:43.400
The human needs to express that goal.
link |
02:24:46.120
We don't have that language that I know of.
link |
02:24:49.000
No, I mean, it can generate stuff.
link |
02:24:50.560
Have you seen what GPT3 can generate?
link |
02:24:52.880
You can say, I mean, there's interface stuff, like it can generate HTML.
link |
02:24:58.440
It can generate basic for loops that give you like...
link |
02:25:02.080
Right, but pick HTML.
link |
02:25:03.080
How do I say I want Google.com?
link |
02:25:05.200
Well, no, you could say...
link |
02:25:07.840
Or not literally Google.com.
link |
02:25:09.360
How do I say I want a webpage that's got a shopping cart and this and that?
link |
02:25:13.200
Yeah, it does that.
link |
02:25:14.200
I mean, so, okay, so just...
link |
02:25:15.200
I don't know if you've seen these demonstrations, but you type in, I want a red button with
link |
02:25:20.880
the text that says, hello, and you typed that natural language and it generates the
link |
02:25:24.760
correct HTML.
link |
02:25:25.760
Okay.
link |
02:25:26.760
I've done this demo.
link |
02:25:27.760
It's kind of compelling.
link |
02:25:28.760
So you have to prompt it with similar kinds of mappings.
link |
02:25:33.360
Of course, it's probably handpicked, I have to experiment that probably...
link |
02:25:38.080
But the fact that you can do that once, even out of like 20, is quite impressive.
link |
02:25:43.120
Again, that's very basic, like the HTML is kind of messy and bad.
link |
02:25:48.520
But yes, the intent is... the idea is the intent is specified in natural language.
link |
02:25:52.720
Okay.
link |
02:25:53.720
Yeah, so I have not seen that.
link |
02:25:54.720
That's really cool.
link |
02:25:55.720
Yeah.
link |
02:25:56.720
Yeah, but the question is the correctness of that.
link |
02:25:59.960
Like visually you can check, oh, the button is red, but for more complicated functions
link |
02:26:10.040
where the intent is harder to check, this goes into like NP completeness kind of things.
link |
02:26:15.560
Like I want to know that this code is correct and generates a giant thing that does some
link |
02:26:22.240
kind of calculation.
link |
02:26:23.760
It seems to be working.
link |
02:26:25.960
It's interesting to think like should the system also try to generate checks for itself
link |
02:26:31.160
for correctness.
link |
02:26:32.160
Yeah.
link |
02:26:33.160
I don't know.
link |
02:26:34.160
And this is way beyond my experience.
link |
02:26:37.200
The thing that I think about is that there doesn't seem to be a lot of equational reasoning
link |
02:26:42.800
going on.
link |
02:26:43.800
Right.
link |
02:26:44.800
It's kind of filling in and kind of propagating patterns that have been seen before into the
link |
02:26:48.720
future and into the generated result.
link |
02:26:50.800
And so if you want to get correctness, you kind of need to improve in kind of things
link |
02:26:55.080
and like higher level logic.
link |
02:26:57.440
And I don't know that...
link |
02:26:58.680
You could talk to Yon about that and see what the bright minds are thinking about right
link |
02:27:04.200
now.
link |
02:27:05.200
But I don't think the GPT is in that vein.
link |
02:27:08.320
It's still really cool.
link |
02:27:09.320
Yeah.
link |
02:27:10.320
And surprisingly, who knows, maybe reasoning is...
link |
02:27:13.520
Is overrated.
link |
02:27:14.520
Yeah.
link |
02:27:15.520
Is overrated.
link |
02:27:16.520
Right.
link |
02:27:17.520
I mean, do we reason?
link |
02:27:18.520
Yeah.
link |
02:27:19.520
How do you tell?
link |
02:27:20.520
Right?
link |
02:27:21.520
Are we just pattern matching based on what we have and then reverse justifying to ourselves?
link |
02:27:22.520
Yeah.
link |
02:27:23.520
Exactly.
link |
02:27:24.520
The reverse...
link |
02:27:25.520
So like I think what the neural networks are missing and I think GPT form might have
link |
02:27:29.800
is to be able to tell stories to itself about what it did.
link |
02:27:33.520
Yeah.
link |
02:27:34.520
Well, that's what humans do.
link |
02:27:35.520
Right?
link |
02:27:36.520
I mean, you talk about like network explainability.
link |
02:27:37.520
Right?
link |
02:27:38.520
And we give...
link |
02:27:39.520
No, that's a hard time about this.
link |
02:27:40.520
But humans don't know why we make decisions.
link |
02:27:42.480
We have this thing called intuition and then we try to like say, this feels like the right
link |
02:27:45.880
thing, but why?
link |
02:27:46.880
Right?
link |
02:27:47.880
And, you know, you wrestle with that when you're making hard decisions and is that science?
link |
02:27:52.480
Not really.
link |
02:27:53.480
Let me ask you about a few high level questions, I guess, is you've done a million things in
link |
02:28:01.520
your life and been very successful.
link |
02:28:04.160
A bunch of young folks listen to this, ask for advice from successful people like you.
link |
02:28:11.800
If you were to give advice to somebody, you know, another graduate student or some high
link |
02:28:17.560
school student about pursuing a career in computing or just advice about life in general,
link |
02:28:25.600
is there some words of wisdom you can give them?
link |
02:28:28.920
So I think you come back to change and, you know, profound leaps happen because people
link |
02:28:34.640
are willing to believe that change is possible and that the world does change and are willing
link |
02:28:40.040
to do the hard thing that it takes to make change happen and whether it be implementing
link |
02:28:44.520
a new programming language or implementing a new system or implementing a new research
link |
02:28:48.360
paper, designing a new thing, moving the world forward in science and philosophy, whatever,
link |
02:28:53.560
it really comes down to somebody who's willing to put in the work, right?
link |
02:28:57.200
And you have, the work is hard for a whole bunch of different reasons, one of which is
link |
02:29:02.080
you, it's work, right?
link |
02:29:06.960
And so you have to have the space in your life in which you can do that work, which is
link |
02:29:10.080
why going to grad school can be a beautiful thing for certain people.
link |
02:29:14.840
But also there's a self doubt that happens, like you're two years into a project, is it
link |
02:29:18.440
going anywhere?
link |
02:29:19.440
Right?
link |
02:29:20.440
Well, what do you do?
link |
02:29:21.440
Do you just give up?
link |
02:29:22.520
Because it's hard?
link |
02:29:23.520
Well, no.
link |
02:29:24.520
I mean, some people like suffering.
link |
02:29:27.160
And so you plow through it.
link |
02:29:29.320
The secret to me is that you have to love what you're doing and follow that passion because
link |
02:29:35.520
if, when you get to the hard times, that's when, you know, if you love what you're doing,
link |
02:29:40.160
you're willing to kind of push through.
link |
02:29:41.960
And this is really hard because it's hard to know what you will love doing until you
link |
02:29:48.840
start doing a lot of things.
link |
02:29:50.320
And so that's why I think that particularly early in your career, it's good to experiment.
link |
02:29:55.000
Do a little bit of everything.
link |
02:29:56.000
Go take the survey class on, you know, the first half of every class in your upper division,
link |
02:30:02.760
you know, lessons and just get exposure to things because certain things will resonate
link |
02:30:06.720
with you and you'll find out, wow, I'm really good at this.
link |
02:30:08.880
I'm really smart at this.
link |
02:30:10.440
Well, it's just because it's, it works with the way your brain.
link |
02:30:12.960
And when something jumps out, I mean, that's one of the things that people often ask about
link |
02:30:17.640
is like, well, I think there's a bunch of cool stuff out there.
link |
02:30:21.440
Like how do I pick the thing?
link |
02:30:23.320
Like, how do you, how do you hook in your life?
link |
02:30:27.600
How did you just hook yourself in and stuck with it?
link |
02:30:30.080
Well, I got lucky.
link |
02:30:31.280
Right?
link |
02:30:32.280
I think that many people forget that a huge amount of it, or most of it is luck, right?
link |
02:30:38.880
So let's not forget that.
link |
02:30:42.040
So for me, I fell in love with computers early on because they spoke to me, I guess.
link |
02:30:48.080
What language did they speak to?
link |
02:30:50.960
Basic.
link |
02:30:51.960
Basic.
link |
02:30:52.960
Yeah.
link |
02:30:53.960
But the, but then it was just kind of following a set of logical progressions, but also deciding
link |
02:31:00.000
that something that was hard was worth doing and a lot of fun, right?
link |
02:31:04.200
And so I think that that is also something that's true for many other domains, which
link |
02:31:08.160
is if you find something that you love doing that's also hard, if you invest yourself in
link |
02:31:13.280
it and add value to the world, then it will mean something generally, right?
link |
02:31:17.320
And again, that can be a research paper, that can be a software system, that can be a new
link |
02:31:21.520
robot, that can be, there's many things that that can be, but a lot of it is like real
link |
02:31:26.600
value comes from doing things that are hard.
link |
02:31:29.560
And that doesn't mean you have to suffer.
link |
02:31:33.360
But it's hard.
link |
02:31:34.360
I mean, you don't often hear that message.
link |
02:31:36.080
We talked about it last time a little bit, but it's one of my, not enough people talk
link |
02:31:41.960
about this, it's beautiful to hear a successful person.
link |
02:31:47.160
Well, in self doubt and imposter syndrome, and these are all things that successful people
link |
02:31:52.280
suffer with as well, particularly when they put themselves in a point of being uncomfortable,
link |
02:31:56.760
which I like to do now and then just because it puts you in learning mode.
link |
02:32:01.480
Like if you want to, if you want to grow as a person, put yourself in a room with a bunch
link |
02:32:06.520
of people that know way more about whatever you're talking about than you do and ask dumb
link |
02:32:10.880
questions.
link |
02:32:11.880
And guess what, smart people love to teach often, not always, but often, and if you listen,
link |
02:32:17.520
if you're prepared to listen, if you're prepared to grow, if you're prepared to make connections,
link |
02:32:20.760
you can do some really interesting things.
link |
02:32:22.640
And I think that a lot of progress is made by people who kind of hop between domains
link |
02:32:26.960
now and then, because they bring a perspective into a field that nobody else has, if people
link |
02:32:35.160
have only been working in that field themselves.
link |
02:32:38.400
We mentioned that the universe is kind of like a compiler, you know, the entirety of
link |
02:32:43.800
it, the whole evolution is kind of a kind of a compilation.
link |
02:32:47.960
Maybe us human beings are kind of compilers.
link |
02:32:50.640
Let me ask the old sort of question that I didn't ask you last time, which is, what's
link |
02:32:56.080
the meaning of it all?
link |
02:32:57.840
Is there a meaning?
link |
02:32:58.840
Like if you asked a compiler why, what would a compiler say, and what's the meaning of
link |
02:33:04.080
life?
link |
02:33:05.080
What's the meaning of life?
link |
02:33:06.080
You know, I'm prepared for not to mean anything.
link |
02:33:09.960
Here we are, all biological things programmed to survive and propagate our DNA, and maybe
link |
02:33:19.440
the universe is just a computer and you just go until entropy takes over the universe and
link |
02:33:25.400
then you're done.
link |
02:33:27.400
I don't think that's a very productive way to live your life, if so.
link |
02:33:33.160
And so I prefer to bias towards the other way, which is saying the universe has a lot
link |
02:33:37.280
of value.
link |
02:33:38.280
And I take happiness out of other people, and a lot of times part of that's having kids,
link |
02:33:43.960
but also the relationships you build with other people.
link |
02:33:47.120
And so the way I try to live my life is what can I do that has value?
link |
02:33:51.360
How can I move the world forward?
link |
02:33:52.560
How can I take what I'm good at and bring it into the world?
link |
02:33:57.680
And I'm one of these people that likes to work really hard and be very focused on the
link |
02:34:02.320
things that I do.
link |
02:34:03.320
And so if I'm going to do that, how can it be in a domain that actually will matter?
link |
02:34:07.480
Because a lot of things that we do, we find ourselves in the cycle of like, okay, I'm
link |
02:34:12.000
doing a thing, I'm very familiar with it, I've done it for a long time, I've never done
link |
02:34:15.840
anything else, but I'm not really learning.
link |
02:34:18.920
I'm not really, I'm keeping things going, but there's a younger generation that can
link |
02:34:23.920
do the same thing, maybe even better than me, right?
link |
02:34:26.720
Maybe if I actually step out of this and jump into something I'm less comfortable with,
link |
02:34:31.440
it's scary.
link |
02:34:32.440
But on the other hand, it gives somebody else a new opportunity, it also then puts you
link |
02:34:36.440
back in learning mode, and that can be really interesting.
link |
02:34:39.200
And one of the things I've learned is that when you go through that, that first you're
link |
02:34:43.360
deep into imposter syndrome, but when you start working your way out, you start to realize,
link |
02:34:47.800
hey, well, there's actually a method to this.
link |
02:34:50.280
And now I'm able to add new things because I bring different perspective.
link |
02:34:54.800
And this is one of the good things about bringing different kinds of people together.
link |
02:35:00.280
Diversity of thought is really important.
link |
02:35:02.240
And if you can pull together people that are coming at things from different directions,
link |
02:35:06.560
you often get innovation.
link |
02:35:08.080
And I love to see that, that aha moment where you're like, we've like really cracked this.
link |
02:35:12.720
This is something nobody's ever done before.
link |
02:35:15.360
And then if you can do it in a context where it adds value, other people can build on it,
link |
02:35:18.960
it helps move the world, then that's what really excites me.
link |
02:35:22.800
So that kind of description of the magic of the human experience, do you think we'll ever
link |
02:35:27.060
create that in like an AGI system?
link |
02:35:29.920
You think we'll be able to create, give AI systems a sense of meaning where they operate
link |
02:35:38.800
in this kind of world, exactly in the way you've described, which is they interact with
link |
02:35:42.760
each other, they interact with us humans.
link |
02:35:44.760
Sure.
link |
02:35:45.760
Sure.
link |
02:35:46.760
Well, so I mean, why are you being so a specist?
link |
02:35:49.640
Right?
link |
02:35:50.640
All right.
link |
02:35:51.640
So AGI versus Bionets versus Bionets, you know, what are we but machines, right?
link |
02:36:00.360
We're just programmed to run our, we have our objective function that we're optimized
link |
02:36:04.520
for, right?
link |
02:36:06.600
And so we're doing our thing.
link |
02:36:07.680
We think we have purpose, but do we really?
link |
02:36:09.400
Yeah.
link |
02:36:10.400
Right?
link |
02:36:11.400
I'm not prepared to say that those new fangled AGI's have no soul just because we don't understand
link |
02:36:16.160
them, right?
link |
02:36:17.160
And I think that would be when they, when they exist, that would be very premature to look
link |
02:36:23.440
at a new thing through your own lens without fully understanding it.
link |
02:36:28.280
You might be just saying that because AI systems in the future will be listening to this and
link |
02:36:32.880
then.
link |
02:36:33.880
Oh yeah.
link |
02:36:34.880
Exactly.
link |
02:36:35.880
You don't want to say anything.
link |
02:36:36.880
Yeah.
link |
02:36:37.880
SkyNet kills everybody.
link |
02:36:38.880
Please spare me.
link |
02:36:39.880
So why is, why is a look ahead thinking?
link |
02:36:42.480
Yeah.
link |
02:36:43.480
But I mean, I think that people spend a lot of time worrying about this kind of stuff.
link |
02:36:46.560
And I think that what we should be worrying about is how do we make the world better?
link |
02:36:50.160
And the thing that I'm most scared about with AGI's is not that, that necessarily the sky
link |
02:36:57.120
net will start shooting everybody with lasers and stuff like that to, to use us for our
link |
02:37:01.840
calories.
link |
02:37:02.840
And the thing that I'm worried about is that humanity, I think, needs a challenge.
link |
02:37:08.400
And if we get into a mode of not having a personal challenge, not having a personal contribution,
link |
02:37:13.680
whether that be like, you know, your kids and seeing what they grow into and helping,
link |
02:37:18.000
helping guide them, whether it be your community that you're engaged in, you're driving forward,
link |
02:37:23.000
whether it be your work and the things that you're doing and the people you're working
link |
02:37:25.720
with and the products you're building and the contribution there, if people don't have
link |
02:37:29.680
a objective, I'm afraid what that means.
link |
02:37:33.720
And I think that this would lead to a rise of the worst part of people, right, instead
link |
02:37:40.280
of people striving together and trying to make the world better.
link |
02:37:45.200
It could degrade into a very unpleasant world, but, but I don't know.
link |
02:37:51.160
I mean, we hopefully have a long ways to go before we discover that.
link |
02:37:55.080
And unfortunately, we have pretty on the ground problems with the pandemic right now.
link |
02:37:58.720
And so I think we should be focused on that as well.
link |
02:38:01.400
Yeah.
link |
02:38:02.400
Ultimately, just as you said, you're optimistic.
link |
02:38:04.640
I think it helps for us to be optimistic because that's faking until you make it.
link |
02:38:09.960
Yeah.
link |
02:38:10.960
Well, and why not?
link |
02:38:11.960
I mean, what's the other side?
link |
02:38:12.960
Right.
link |
02:38:13.960
So, I mean, I'm not personally a very religious person, but I've heard people say like, oh,
link |
02:38:18.960
yeah, of course I believe in God.
link |
02:38:20.400
Of course I go to church because if God's real, you know, I want to be on the right
link |
02:38:25.320
side of that.
link |
02:38:26.320
And if it's not real, it doesn't matter.
link |
02:38:27.320
Yeah, it doesn't matter.
link |
02:38:28.320
So, you know, that's a fair way to do it.
link |
02:38:31.600
Yeah.
link |
02:38:32.600
I mean, the same thing with nuclear deterrence, all, you know, global warming, all these things,
link |
02:38:38.520
all these threats, natural engineer pandemics, all these threats we face, I think it's paralyzing
link |
02:38:47.680
to be terrified of all the possible ways we could destroy ourselves.
link |
02:38:52.520
I think it's much better or at least productive to be hopeful and to engineer defenses against
link |
02:39:00.120
these things to engineer a future where like, you know, see like a positive future and engineer
link |
02:39:07.040
that future.
link |
02:39:08.040
Yeah.
link |
02:39:09.040
Well, and I think that's another thing to think about as, you know, a human, particularly
link |
02:39:12.960
if you're young and trying to figure out what it is that you want to be when you grow
link |
02:39:15.720
up like I am.
link |
02:39:18.040
I'm always looking for that.
link |
02:39:20.640
The question then is, how do you want to spend your time?
link |
02:39:24.440
And right now, there seems to be a norm of being a consumption culture, like I'm going
link |
02:39:29.080
to watch the news and revel in how horrible everything is right now.
link |
02:39:33.480
I'm going to go find out about the latest atrocity and find out all the details of like the terrible
link |
02:39:38.600
thing that happened and be outraged by it.
link |
02:39:42.000
You can spend a lot of time watching TV and watching the news sitcom or whatever people
link |
02:39:46.800
watch these days.
link |
02:39:47.800
But that's a lot of hours, right, and those are hours that if you're turned into being
link |
02:39:54.040
productive, learning, growing, experiencing, you know, when the pandemic's over, going
link |
02:40:00.280
exploring, right, it leads to more growth.
link |
02:40:03.960
And I think it leads to more optimism and happiness because you're, you're, you're building,
link |
02:40:08.360
right?
link |
02:40:09.360
You're building yourself, you're building your capabilities, you're building your viewpoints,
link |
02:40:12.280
you're building your perspective, and I think that a lot of the, the consuming of other people's
link |
02:40:19.080
messages leads to kind of a negative viewpoint, which you need to be aware of what's happening
link |
02:40:24.240
because that's also important, but there's a balance that I think focusing on creation
link |
02:40:29.160
is, is a very valuable thing to do.
link |
02:40:31.840
Yeah.
link |
02:40:32.840
So what you're saying is people should focus on working on the sexiest field of all, which
link |
02:40:37.400
is compiler design.
link |
02:40:38.400
Exactly.
link |
02:40:39.400
Hey, you can go work on machine learning and be crowded out by the, the thousands of graduates
link |
02:40:43.560
popping out of school that all want to do the same thing, or you could work in the place
link |
02:40:47.040
that people will overpay you because there's not enough smart people working in it.
link |
02:40:51.520
And here at the end of Moore's Law, according to some people, actually the software is the
link |
02:40:56.280
hard part too.
link |
02:40:57.280
Yeah.
link |
02:40:58.280
I mean, optimization is truly, truly beautiful.
link |
02:41:02.320
And also on the YouTube side or education side, you know, it's, there's a, it'd be nice
link |
02:41:08.400
to have some material that shows the beauty of compilers.
link |
02:41:12.000
Yeah.
link |
02:41:13.000
Yeah.
link |
02:41:14.000
That's, that's something.
link |
02:41:15.000
So that's a call for, for people to create that kind of content as well.
link |
02:41:18.920
Chris, you're one of my favorite people to talk to.
link |
02:41:22.160
It's such a huge honor that you would waste your time talking to me.
link |
02:41:26.240
I've always appreciated it.
link |
02:41:27.840
Thank you so much.
link |
02:41:28.840
I mean, the truth of it is you spent a lot of time talking to me just on, you know, walks
link |
02:41:33.520
and other things like that.
link |
02:41:34.520
So it's great to catch up with.
link |
02:41:35.760
Thanks, man.
link |
02:41:37.440
Thanks for listening to this conversation with Chris Latner.
link |
02:41:40.160
And thank you to our sponsors, Blinkist, an app that summarizes key ideas from thousands
link |
02:41:45.680
of books, Neuro, which is a maker of functional gum and mints that supercharged my mind, Masterclass,
link |
02:41:52.640
which are online courses from world experts.
link |
02:41:55.560
And finally, Cash App, which is an app for sending money to friends.
link |
02:42:00.320
Please check out these sponsors in the description to get a discount and to support this podcast.
link |
02:42:06.160
If you enjoyed this thing, subscribe on YouTube, review it with 5 stars on Apple Podcast, follow
link |
02:42:10.840
on Spotify, support on Patreon, connect with me on Twitter, Alex Friedman.
link |
02:42:16.440
And now let me leave you with some words from Chris Latner.
link |
02:42:19.200
So much of language design is about tradeoffs and you can't see those tradeoffs unless you
link |
02:42:24.040
have a community of people that really represent those different points.
link |
02:42:28.720
Thank you for listening and hope to see you next time.