back to index

Chris Lattner: Compilers, LLVM, Swift, TPU, and ML Accelerators | Lex Fridman Podcast #21


small model | large model

link |
00:00:00.000
The following is a conversation with Chris Latner.
link |
00:00:02.680
Currently, he's a senior director
link |
00:00:04.560
at Google working on several projects, including CPU, GPU,
link |
00:00:08.400
TPU accelerators for TensorFlow, Swift for TensorFlow,
link |
00:00:12.040
and all kinds of machine learning compiler magic
link |
00:00:14.400
going on behind the scenes.
link |
00:00:16.360
He's one of the top experts in the world
link |
00:00:18.440
on compiler technologies, which means he deeply
link |
00:00:21.160
understands the intricacies of how hardware and software come
link |
00:00:25.560
together to create efficient code.
link |
00:00:27.920
He created the LLVM compiler infrastructure project
link |
00:00:31.400
and the Clang compiler.
link |
00:00:33.360
He led major engineering efforts at Apple,
link |
00:00:36.000
including the creation of the Swift programming language.
link |
00:00:39.000
He also briefly spent time at Tesla
link |
00:00:41.720
as vice president of Autopilot software
link |
00:00:44.280
during the transition from Autopilot hardware 1
link |
00:00:46.760
to hardware 2, when Tesla essentially
link |
00:00:49.600
started from scratch to build an in house software
link |
00:00:52.640
infrastructure for Autopilot.
link |
00:00:54.800
I could have easily talked to Chris for many more hours.
link |
00:00:58.040
Compiling code down across the levels of abstraction
link |
00:01:01.200
is one of the most fundamental and fascinating aspects
link |
00:01:04.160
of what computers do, and he is one of the world
link |
00:01:06.640
experts in this process.
link |
00:01:08.560
It's rigorous science, and it's messy, beautiful art.
link |
00:01:12.880
This conversation is part of the Artificial Intelligence
link |
00:01:15.920
podcast.
link |
00:01:16.760
If you enjoy it, subscribe on YouTube, iTunes,
link |
00:01:19.440
or simply connect with me on Twitter at Lex Friedman,
link |
00:01:22.760
spelled F R I D.
link |
00:01:24.680
And now, here's my conversation with Chris Ladner.
link |
00:01:29.360
What was the first program you've ever written?
link |
00:01:33.160
My first program.
link |
00:01:34.120
Back, and when was it?
link |
00:01:35.360
I think I started as a kid, and my parents
link |
00:01:39.080
got a basic programming book.
link |
00:01:41.560
And so when I started, it was typing out programs
link |
00:01:44.200
from a book, and seeing how they worked,
link |
00:01:46.880
and then typing them in wrong, and trying
link |
00:01:49.680
to figure out why they were not working right,
link |
00:01:51.680
that kind of stuff.
link |
00:01:52.960
So BASIC, what was the first language
link |
00:01:54.880
that you remember yourself maybe falling in love with,
link |
00:01:58.360
like really connecting with?
link |
00:01:59.720
I don't know.
link |
00:02:00.400
I mean, I feel like I've learned a lot along the way,
link |
00:02:02.680
and each of them have a different special thing
link |
00:02:05.800
about them.
link |
00:02:06.640
So I started in BASIC, and then went like GW BASIC,
link |
00:02:09.720
which was the thing back in the DOS days,
link |
00:02:11.440
and then upgraded to QBASIC, and eventually QuickBASIC,
link |
00:02:15.280
which are all slightly more fancy versions of Microsoft
link |
00:02:18.200
BASIC.
link |
00:02:19.440
Made the jump to Pascal, and started
link |
00:02:21.360
doing machine language programming and assembly
link |
00:02:23.920
in Pascal, which was really cool.
link |
00:02:25.280
Turbo Pascal was amazing for its day.
link |
00:02:28.080
Eventually got into C, C++, and then kind of did
link |
00:02:31.600
lots of other weird things.
link |
00:02:33.400
I feel like you took the dark path, which is the,
link |
00:02:37.080
you could have gone Lisp.
link |
00:02:39.480
Yeah.
link |
00:02:40.000
You could have gone higher level sort
link |
00:02:41.680
of functional philosophical hippie route.
link |
00:02:44.600
Instead, you went into like the dark arts of the C.
link |
00:02:48.080
It was straight into the machine.
link |
00:02:49.720
Straight to the machine.
link |
00:02:50.680
So I started with BASIC, Pascal, and then Assembly,
link |
00:02:53.880
and then wrote a lot of Assembly.
link |
00:02:55.320
And I eventually did Smalltalk and other things like that.
link |
00:03:00.080
But that was not the starting point.
link |
00:03:01.880
But so what is this journey to C?
link |
00:03:05.080
Is that in high school?
link |
00:03:06.320
Is that in college?
link |
00:03:07.560
That was in high school, yeah.
link |
00:03:09.320
And then that was really about trying
link |
00:03:13.720
to be able to do more powerful things than what Pascal could
link |
00:03:16.240
do, and also to learn a different world.
link |
00:03:18.960
So he was really confusing to me with pointers
link |
00:03:20.760
and the syntax and everything, and it took a while.
link |
00:03:23.000
But Pascal's much more principled in various ways.
link |
00:03:28.800
C is more, I mean, it has its historical roots,
link |
00:03:33.400
but it's not as easy to learn.
link |
00:03:35.520
With pointers, there's this memory management thing
link |
00:03:39.880
that you have to become conscious of.
link |
00:03:41.680
Is that the first time you start to understand
link |
00:03:43.880
that there's resources that you're supposed to manage?
link |
00:03:46.520
Well, so you have that in Pascal as well.
link |
00:03:48.480
But in Pascal, like the caret instead of the star,
link |
00:03:51.440
there's some small differences like that.
link |
00:03:53.160
But it's not about pointer arithmetic.
link |
00:03:55.680
And in C, you end up thinking about how things get
link |
00:03:58.760
laid out in memory a lot more.
link |
00:04:00.840
And so in Pascal, you have allocating and deallocating
link |
00:04:04.160
and owning the memory, but just the programs are simpler,
link |
00:04:07.560
and you don't have to.
link |
00:04:10.080
Well, for example, Pascal has a string type.
link |
00:04:12.640
And so you can think about a string
link |
00:04:14.040
instead of an array of characters
link |
00:04:15.880
which are consecutive in memory.
link |
00:04:17.720
So it's a little bit of a higher level abstraction.
link |
00:04:20.400
So let's get into it.
link |
00:04:22.800
Let's talk about LLVM, C lang, and compilers.
link |
00:04:25.560
Sure.
link |
00:04:26.560
So can you tell me first what LLVM and C lang are?
link |
00:04:32.160
And how is it that you find yourself
link |
00:04:33.960
the creator and lead developer, one
link |
00:04:35.720
of the most powerful compiler optimization systems
link |
00:04:39.400
in use today?
link |
00:04:40.080
Sure.
link |
00:04:40.580
So I guess they're different things.
link |
00:04:43.320
So let's start with what is a compiler?
link |
00:04:47.080
Is that a good place to start?
link |
00:04:48.840
What are the phases of a compiler?
link |
00:04:50.200
Where are the parts?
link |
00:04:50.920
Yeah, what is it?
link |
00:04:51.600
So what is even a compiler used for?
link |
00:04:53.400
So the way I look at this is you have a two sided problem of you
link |
00:04:57.880
have humans that need to write code.
link |
00:05:00.120
And then you have machines that need to run
link |
00:05:01.880
the program that the human wrote.
link |
00:05:03.400
And for lots of reasons, the humans
link |
00:05:05.280
don't want to be writing in binary
link |
00:05:07.040
and want to think about every piece of hardware.
link |
00:05:09.080
And so at the same time that you have lots of humans,
link |
00:05:12.100
you also have lots of kinds of hardware.
link |
00:05:14.800
And so compilers are the art of allowing
link |
00:05:17.400
humans to think at a level of abstraction
link |
00:05:19.240
that they want to think about.
link |
00:05:20.920
And then get that program, get the thing that they wrote,
link |
00:05:23.600
to run on a specific piece of hardware.
link |
00:05:26.080
And the interesting and exciting part of all this
link |
00:05:29.480
is that there's now lots of different kinds of hardware,
link |
00:05:32.080
chips like x86 and PowerPC and ARM and things like that.
link |
00:05:35.780
But also high performance accelerators
link |
00:05:37.320
for machine learning and other things like that
link |
00:05:38.900
are also just different kinds of hardware, GPUs.
link |
00:05:41.520
These are new kinds of hardware.
link |
00:05:42.940
And at the same time, on the programming side of it,
link |
00:05:45.640
you have basic, you have C, you have JavaScript,
link |
00:05:48.680
you have Python, you have Swift.
link |
00:05:50.560
You have lots of other languages
link |
00:05:52.840
that are all trying to talk to the human in a different way
link |
00:05:55.200
to make them more expressive and capable and powerful.
link |
00:05:58.320
And so compilers are the thing
link |
00:06:01.500
that goes from one to the other.
link |
00:06:03.460
End to end, from the very beginning to the very end.
link |
00:06:05.200
End to end.
link |
00:06:06.040
And so you go from what the human wrote
link |
00:06:08.120
and programming languages end up being about
link |
00:06:11.600
expressing intent, not just for the compiler
link |
00:06:14.560
and the hardware, but the programming language's job
link |
00:06:17.980
is really to capture an expression
link |
00:06:20.920
of what the programmer wanted
link |
00:06:22.680
that then can be maintained and adapted
link |
00:06:25.120
and evolved by other humans,
link |
00:06:27.120
as well as interpreted by the compiler.
link |
00:06:29.720
So when you look at this problem,
link |
00:06:31.560
you have, on the one hand, humans, which are complicated.
link |
00:06:34.200
And you have hardware, which is complicated.
link |
00:06:36.760
And so compilers typically work in multiple phases.
link |
00:06:39.900
And so the software engineering challenge
link |
00:06:42.760
that you have here is try to get maximum reuse
link |
00:06:45.000
out of the amount of code that you write,
link |
00:06:47.140
because these compilers are very complicated.
link |
00:06:49.800
And so the way it typically works out
link |
00:06:51.240
is that you have something called a front end or a parser
link |
00:06:54.480
that is language specific.
link |
00:06:56.640
And so you'll have a C parser, and that's what Clang is,
link |
00:07:00.400
or C++ or JavaScript or Python or whatever.
link |
00:07:03.480
That's the front end.
link |
00:07:05.000
Then you'll have a middle part,
link |
00:07:07.120
which is often the optimizer.
link |
00:07:09.020
And then you'll have a late part,
link |
00:07:11.120
which is hardware specific.
link |
00:07:13.320
And so compilers end up,
link |
00:07:15.020
there's many different layers often,
link |
00:07:16.680
but these three big groups are very common in compilers.
link |
00:07:20.860
And what LLVM is trying to do
link |
00:07:22.200
is trying to standardize that middle and last part.
link |
00:07:25.360
And so one of the cool things about LLVM
link |
00:07:27.880
is that there are a lot of different languages
link |
00:07:29.740
that compile through to it.
link |
00:07:31.080
And so things like Swift, but also Julia, Rust,
link |
00:07:35.600
Clang for C, C++, Subjective C,
link |
00:07:39.140
like these are all very different languages
link |
00:07:40.940
and they can all use the same optimization infrastructure,
link |
00:07:43.780
which gets better performance,
link |
00:07:45.340
and the same code generation infrastructure
link |
00:07:47.240
for hardware support.
link |
00:07:48.780
And so LLVM is really that layer that is common,
link |
00:07:52.240
that all these different specific compilers can use.
link |
00:07:55.580
And is it a standard, like a specification,
link |
00:07:59.300
or is it literally an implementation?
link |
00:08:01.140
It's an implementation.
link |
00:08:02.140
And so I think there's a couple of different ways
link |
00:08:05.900
of looking at it, right?
link |
00:08:06.740
Because it depends on which angle you're looking at it from.
link |
00:08:09.700
LLVM ends up being a bunch of code, okay?
link |
00:08:12.660
So it's a bunch of code that people reuse
link |
00:08:14.460
and they build compilers with.
link |
00:08:16.540
We call it a compiler infrastructure
link |
00:08:18.060
because it's kind of the underlying platform
link |
00:08:20.060
that you build a concrete compiler on top of.
link |
00:08:22.580
But it's also a community.
link |
00:08:23.740
And the LLVM community is hundreds of people
link |
00:08:26.820
that all collaborate.
link |
00:08:27.980
And one of the most fascinating things about LLVM
link |
00:08:30.620
over the course of time is that we've managed somehow
link |
00:08:34.260
to successfully get harsh competitors
link |
00:08:37.060
in the commercial space to collaborate
link |
00:08:39.060
on shared infrastructure.
link |
00:08:41.120
And so you have Google and Apple,
link |
00:08:43.900
you have AMD and Intel,
link |
00:08:45.860
you have Nvidia and AMD on the graphics side,
link |
00:08:48.860
you have Cray and everybody else doing these things.
link |
00:08:52.620
And all these companies are collaborating together
link |
00:08:55.420
to make that shared infrastructure really, really great.
link |
00:08:58.520
And they do this not out of the goodness of their heart,
link |
00:09:01.380
but they do it because it's in their commercial interest
link |
00:09:03.420
of having really great infrastructure
link |
00:09:05.140
that they can build on top of
link |
00:09:06.740
and facing the reality that it's so expensive
link |
00:09:09.080
that no one company, even the big companies,
link |
00:09:11.160
no one company really wants to implement it all themselves.
link |
00:09:14.580
Expensive or difficult?
link |
00:09:16.100
Both.
link |
00:09:16.940
That's a great point because it's also about the skill sets.
link |
00:09:20.540
And the skill sets are very hard to find.
link |
00:09:26.020
How big is the LLVM?
link |
00:09:27.980
It always seems like with open source projects,
link |
00:09:30.780
the kind, an LLVM is open source?
link |
00:09:33.500
Yes, it's open source.
link |
00:09:34.420
It's about, it's 19 years old now, so it's fairly old.
link |
00:09:38.660
It seems like the magic often happens
link |
00:09:40.940
within a very small circle of people.
link |
00:09:43.020
Yes.
link |
00:09:43.860
At least their early birth and whatever.
link |
00:09:46.060
Yes, so the LLVM came from a university project,
link |
00:09:49.660
and so I was at the University of Illinois.
link |
00:09:51.540
And there it was myself, my advisor,
link |
00:09:53.900
and then a team of two or three research students
link |
00:09:57.500
in the research group,
link |
00:09:58.380
and we built many of the core pieces initially.
link |
00:10:02.100
I then graduated and went to Apple,
link |
00:10:03.740
and at Apple brought it to the products,
link |
00:10:06.480
first in the OpenGL graphics stack,
link |
00:10:09.340
but eventually to the C compiler realm,
link |
00:10:11.580
and eventually built Clang,
link |
00:10:12.780
and eventually built Swift and these things.
link |
00:10:14.640
Along the way, building a team of people
link |
00:10:16.380
that are really amazing compiler engineers
link |
00:10:18.620
that helped build a lot of that.
link |
00:10:20.060
And so as it was gaining momentum
link |
00:10:21.860
and as Apple was using it, being open source and public
link |
00:10:24.780
and encouraging contribution,
link |
00:10:26.440
many others, for example, at Google,
link |
00:10:28.780
came in and started contributing.
link |
00:10:30.220
And in some cases, Google effectively owns Clang now
link |
00:10:33.740
because it cares so much about C++
link |
00:10:35.540
and the evolution of that ecosystem,
link |
00:10:37.340
and so it's investing a lot in the C++ world
link |
00:10:41.420
and the tooling and things like that.
link |
00:10:42.980
And so likewise, NVIDIA cares a lot about CUDA.
link |
00:10:47.860
And so CUDA uses Clang and uses LLVM
link |
00:10:50.780
for graphics and GPGPU.
link |
00:10:54.060
And so when you first started as a master's project,
link |
00:10:58.940
I guess, did you think it was gonna go as far as it went?
link |
00:11:02.980
Were you crazy ambitious about it?
link |
00:11:06.340
No.
link |
00:11:07.180
It seems like a really difficult undertaking, a brave one.
link |
00:11:09.840
Yeah, no, no, no, it was nothing like that.
link |
00:11:11.380
So my goal when I went to the University of Illinois
link |
00:11:13.740
was to get in and out with a non thesis masters in a year
link |
00:11:17.540
and get back to work.
link |
00:11:18.720
So I was not planning to stay for five years
link |
00:11:22.200
and build this massive infrastructure.
link |
00:11:24.460
I got nerd sniped into staying.
link |
00:11:27.380
And a lot of it was because LLVM was fun
link |
00:11:29.580
and I was building cool stuff
link |
00:11:30.900
and learning really interesting things
link |
00:11:33.420
and facing both software engineering challenges,
link |
00:11:36.900
but also learning how to work in a team
link |
00:11:38.540
and things like that.
link |
00:11:40.100
I had worked at many companies as interns before that,
link |
00:11:43.620
but it was really a different thing
link |
00:11:45.860
to have a team of people that are working together
link |
00:11:48.060
and try and collaborate in version control.
link |
00:11:50.460
And it was just a little bit different.
link |
00:11:52.420
Like I said, I just talked to Don Knuth
link |
00:11:54.060
and he believes that 2% of the world population
link |
00:11:56.860
have something weird with their brain,
link |
00:11:58.820
that they're geeks, they understand computers,
link |
00:12:01.100
they're connected with computers.
link |
00:12:02.580
He put it at exactly 2%.
link |
00:12:04.380
Okay, so.
link |
00:12:05.540
He's a specific guy.
link |
00:12:06.580
It's very specific.
link |
00:12:08.780
Well, he says, I can't prove it,
link |
00:12:10.180
but it's very empirically there.
link |
00:12:13.180
Is there something that attracts you
link |
00:12:14.500
to the idea of optimizing code?
link |
00:12:16.940
And he seems like that's one of the biggest,
link |
00:12:19.180
coolest things about LLVM.
link |
00:12:20.900
Yeah, that's one of the major things it does.
link |
00:12:22.500
So I got into that because of a person, actually.
link |
00:12:26.460
So when I was in my undergraduate,
link |
00:12:28.220
I had an advisor, or a professor named Steve Vegdahl.
link |
00:12:32.060
And he, I went to this little tiny private school.
link |
00:12:35.740
There were like seven or nine people
link |
00:12:38.300
in my computer science department,
link |
00:12:40.340
students in my class.
link |
00:12:43.100
So it was a very tiny, very small school.
link |
00:12:47.460
It was kind of a wart on the side of the math department
link |
00:12:49.940
kind of a thing at the time.
link |
00:12:51.260
I think it's evolved a lot in the many years since then.
link |
00:12:53.820
But Steve Vegdahl was a compiler guy.
link |
00:12:58.300
And he was super passionate.
link |
00:12:59.580
And his passion rubbed off on me.
link |
00:13:02.740
And one of the things I like about compilers
link |
00:13:04.460
is that they're large, complicated software pieces.
link |
00:13:09.100
And so one of the culminating classes
link |
00:13:12.940
that many computer science departments,
link |
00:13:14.540
at least at the time, did was to say
link |
00:13:16.700
that you would take algorithms and data structures
link |
00:13:18.380
and all these core classes.
link |
00:13:19.460
But then the compilers class was one of the last classes
link |
00:13:21.740
you take because it pulls everything together.
link |
00:13:24.380
And then you work on one piece of code
link |
00:13:26.980
over the entire semester.
link |
00:13:28.700
And so you keep building on your own work,
link |
00:13:32.180
which is really interesting.
link |
00:13:33.460
And it's also very challenging because in many classes,
link |
00:13:36.060
if you don't get a project done, you just forget about it
link |
00:13:38.380
and move on to the next one and get your B or whatever it is.
link |
00:13:41.300
But here you have to live with the decisions you make
link |
00:13:43.860
and continue to reinvest in it.
link |
00:13:45.220
And I really like that.
link |
00:13:48.500
And so I did an extra study project
link |
00:13:50.700
with him the following semester.
link |
00:13:52.420
And he was just really great.
link |
00:13:53.940
And he was also a great mentor in a lot of ways.
link |
00:13:56.860
And so from him and from his advice,
link |
00:13:59.500
he encouraged me to go to graduate school.
link |
00:14:01.380
I wasn't super excited about going to grad school.
link |
00:14:03.420
I wanted the master's degree, but I
link |
00:14:05.540
didn't want to be an academic.
link |
00:14:08.940
But like I said, I kind of got tricked into saying
link |
00:14:11.100
and was having a lot of fun.
link |
00:14:12.180
And I definitely do not regret it.
link |
00:14:14.540
What aspects of compilers were the things you connected with?
link |
00:14:17.940
So LLVM, there's also the other part
link |
00:14:22.100
that's really interesting if you're interested in languages
link |
00:14:24.940
is parsing and just analyzing the language,
link |
00:14:29.620
breaking it down, parsing, and so on.
link |
00:14:31.220
Was that interesting to you, or were you
link |
00:14:32.580
more interested in optimization?
link |
00:14:34.060
For me, it was more so I'm not really a math person.
link |
00:14:37.420
I could do math.
link |
00:14:38.180
I understand some bits of it when I get into it.
link |
00:14:41.540
But math is never the thing that attracted me.
link |
00:14:43.940
And so a lot of the parser part of the compiler
link |
00:14:46.100
has a lot of good formal theories
link |
00:14:47.820
that Don, for example, knows quite well.
link |
00:14:50.060
I'm still waiting for his book on that.
link |
00:14:54.740
But I just like building a thing and seeing what it could do
link |
00:14:57.900
and exploring and getting it to do more things
link |
00:15:00.740
and then setting new goals and reaching for them.
link |
00:15:04.020
And in the case of LLVM, when I started working on that,
link |
00:15:09.580
my research advisor that I was working for was a compiler guy.
link |
00:15:13.420
And so he and I specifically found each other
link |
00:15:15.620
because we were both interested in compilers.
link |
00:15:16.940
And so I started working with him and taking his class.
link |
00:15:19.500
And a lot of LLVM initially was, it's
link |
00:15:21.580
fun implementing all the standard algorithms and all
link |
00:15:24.380
the things that people had been talking about
link |
00:15:26.380
and were well known.
link |
00:15:27.220
And they were in the curricula for advanced studies
link |
00:15:30.620
and compilers.
link |
00:15:31.340
And so just being able to build that was really fun.
link |
00:15:34.580
And I was learning a lot by, instead of reading about it,
link |
00:15:37.660
just building.
link |
00:15:38.660
And so I enjoyed that.
link |
00:15:40.220
So you said compilers are these complicated systems.
link |
00:15:42.820
Can you even just with language try
link |
00:15:46.180
to describe how you turn a C++ program into code?
link |
00:15:52.220
Like, what are the hard parts?
link |
00:15:53.460
Why is it so hard?
link |
00:15:54.620
So I'll give you examples of the hard parts along the way.
link |
00:15:57.020
So C++ is a very complicated programming language.
link |
00:16:01.060
It's something like 1,400 pages in the spec.
link |
00:16:03.500
So C++ by itself is crazy complicated.
link |
00:16:06.060
Can we just pause?
link |
00:16:07.140
What makes the language complicated in terms
link |
00:16:09.140
of what's syntactically?
link |
00:16:12.340
So it's what they call syntax.
link |
00:16:14.300
So the actual how the characters are arranged, yes.
link |
00:16:16.700
It's also semantics, how it behaves.
link |
00:16:20.020
It's also, in the case of C++, there's
link |
00:16:21.900
a huge amount of history.
link |
00:16:23.380
C++ is built on top of C. You play that forward.
link |
00:16:26.700
And then a bunch of suboptimal, in some cases, decisions
link |
00:16:29.860
were made, and they compound.
link |
00:16:31.620
And then more and more and more things
link |
00:16:33.380
keep getting added to C++, and it will probably never stop.
link |
00:16:36.980
But the language is very complicated
link |
00:16:38.540
from that perspective.
link |
00:16:39.540
And so the interactions between subsystems
link |
00:16:41.200
is very complicated.
link |
00:16:42.420
There's just a lot there.
link |
00:16:43.580
And when you talk about the front end,
link |
00:16:45.660
one of the major challenges, which
link |
00:16:47.060
clang as a project, the C, C++ compiler that I built,
link |
00:16:51.140
I and many people built, one of the challenges we took on
link |
00:16:54.480
was we looked at GCC.
link |
00:16:57.780
GCC, at the time, was a really good industry standardized
link |
00:17:02.540
compiler that had really consolidated
link |
00:17:05.260
a lot of the other compilers in the world and was a standard.
link |
00:17:08.340
But it wasn't really great for research.
link |
00:17:10.620
The design was very difficult to work with.
link |
00:17:12.580
And it was full of global variables and other things
link |
00:17:16.620
that made it very difficult to reuse in ways
link |
00:17:18.540
that it wasn't originally designed for.
link |
00:17:20.420
And so with clang, one of the things that we wanted to do
link |
00:17:22.740
is push forward on better user interface,
link |
00:17:25.500
so make error messages that are just better than GCC's.
link |
00:17:28.060
And that's actually hard, because you
link |
00:17:29.580
have to do a lot of bookkeeping in an efficient way
link |
00:17:32.780
to be able to do that.
link |
00:17:33.700
We want to make compile time better.
link |
00:17:35.180
And so compile time is about making it efficient,
link |
00:17:37.500
which is also really hard when you're keeping
link |
00:17:38.900
track of extra information.
link |
00:17:40.540
We wanted to make new tools available,
link |
00:17:43.380
so refactoring tools and other analysis tools
link |
00:17:46.380
that GCC never supported, also leveraging the extra information
link |
00:17:50.540
we kept, but enabling those new classes of tools
link |
00:17:54.060
that then get built into IDEs.
link |
00:17:55.940
And so that's been one of the areas that clang has really
link |
00:17:59.380
helped push the world forward in,
link |
00:18:01.300
is in the tooling for C and C++ and things like that.
link |
00:18:05.060
But C++ and the front end piece is complicated.
link |
00:18:07.500
And you have to build syntax trees.
link |
00:18:09.000
And you have to check every rule in the spec.
link |
00:18:11.340
And you have to turn that back into an error message
link |
00:18:14.020
to the human that the human can understand
link |
00:18:16.020
when they do something wrong.
link |
00:18:17.820
But then you start doing what's called lowering,
link |
00:18:20.740
so going from C++ and the way that it represents
link |
00:18:23.060
code down to the machine.
link |
00:18:24.980
And when you do that, there's many different phases
link |
00:18:27.380
you go through.
link |
00:18:29.660
Often, there are, I think LLVM has something like 150
link |
00:18:33.020
different what are called passes in the compiler
link |
00:18:36.260
that the code passes through.
link |
00:18:38.780
And these get organized in very complicated ways,
link |
00:18:41.860
which affect the generated code and the performance
link |
00:18:44.360
and compile time and many other things.
link |
00:18:45.980
What are they passing through?
link |
00:18:47.300
So after you do the clang parsing, what's the graph?
link |
00:18:53.980
What does it look like?
link |
00:18:54.900
What's the data structure here?
link |
00:18:56.100
Yeah, so in the parser, it's usually a tree.
link |
00:18:59.060
And it's called an abstract syntax tree.
link |
00:19:01.100
And so the idea is you have a node for the plus
link |
00:19:04.580
that the human wrote in their code.
link |
00:19:06.820
Or the function call, you'll have a node for call
link |
00:19:09.020
with the function that they call and the arguments they pass,
link |
00:19:11.900
things like that.
link |
00:19:14.460
This then gets lowered into what's
link |
00:19:16.620
called an intermediate representation.
link |
00:19:18.620
And intermediate representations are like LLVM has one.
link |
00:19:22.100
And there, it's what's called a control flow graph.
link |
00:19:26.940
And so you represent each operation in the program
link |
00:19:31.220
as a very simple, like this is going to add two numbers.
link |
00:19:34.480
This is going to multiply two things.
link |
00:19:35.980
Maybe we'll do a call.
link |
00:19:37.460
But then they get put in what are called blocks.
link |
00:19:40.260
And so you get blocks of these straight line operations,
link |
00:19:43.580
where instead of being nested like in a tree,
link |
00:19:45.340
it's straight line operations.
link |
00:19:46.900
And so there's a sequence and an ordering to these operations.
link |
00:19:49.780
So within the block or outside the block?
link |
00:19:51.820
That's within the block.
link |
00:19:52.980
And so it's a straight line sequence of operations
link |
00:19:54.980
within the block.
link |
00:19:55.740
And then you have branches, like conditional branches,
link |
00:19:58.980
between blocks.
link |
00:20:00.140
And so when you write a loop, for example, in a syntax tree,
link |
00:20:04.860
you would have a for node, like for a for statement
link |
00:20:08.060
in a C like language, you'd have a for node.
link |
00:20:10.540
And you have a pointer to the expression
link |
00:20:12.200
for the initializer, a pointer to the expression
link |
00:20:14.080
for the increment, a pointer to the expression
link |
00:20:16.040
for the comparison, a pointer to the body.
link |
00:20:18.900
And these are all nested underneath it.
link |
00:20:21.060
In a control flow graph, you get a block
link |
00:20:22.900
for the code that runs before the loop, so the initializer
link |
00:20:26.820
code.
link |
00:20:27.620
And you have a block for the body of the loop.
link |
00:20:30.340
And so the body of the loop code goes in there,
link |
00:20:33.780
but also the increment and other things like that.
link |
00:20:35.660
And then you have a branch that goes back to the top
link |
00:20:37.860
and a comparison and a branch that goes out.
link |
00:20:39.900
And so it's more of an assembly level kind of representation.
link |
00:20:43.820
But the nice thing about this level of representation
link |
00:20:46.060
is it's much more language independent.
link |
00:20:48.700
And so there's lots of different kinds of languages
link |
00:20:51.900
with different kinds of, you know,
link |
00:20:54.540
JavaScript has a lot of different ideas of what
link |
00:20:56.840
is false, for example.
link |
00:20:58.180
And all that can stay in the front end.
link |
00:21:00.780
But then that middle part can be shared across all those.
link |
00:21:04.220
How close is that intermediate representation
link |
00:21:07.540
to neural networks, for example?
link |
00:21:10.620
Are they, because everything you describe
link |
00:21:13.540
is a kind of echoes of a neural network graph.
link |
00:21:16.100
Are they neighbors or what?
link |
00:21:18.940
They're quite different in details,
link |
00:21:20.980
but they're very similar in idea.
link |
00:21:22.520
So one of the things that neural networks do
link |
00:21:24.320
is they learn representations for data
link |
00:21:26.900
at different levels of abstraction.
link |
00:21:29.140
And then they transform those through layers, right?
link |
00:21:33.940
So the compiler does very similar things.
link |
00:21:35.660
But one of the things the compiler does
link |
00:21:37.320
is it has relatively few different representations.
link |
00:21:40.660
Where a neural network often, as you get deeper, for example,
link |
00:21:43.100
you get many different representations
link |
00:21:44.820
in each layer or set of ops.
link |
00:21:47.380
It's transforming between these different representations.
link |
00:21:50.260
In a compiler, often you get one representation
link |
00:21:53.100
and they do many transformations to it.
link |
00:21:55.240
And these transformations are often applied iteratively.
link |
00:21:59.540
And for programmers, there's familiar types of things.
link |
00:22:02.940
For example, trying to find expressions inside of a loop
link |
00:22:06.180
and pulling them out of a loop so they execute for times.
link |
00:22:08.540
Or find redundant computation.
link |
00:22:10.740
Or find constant folding or other simplifications,
link |
00:22:15.380
turning two times x into x shift left by one.
link |
00:22:19.060
And things like this are all the examples
link |
00:22:21.980
of the things that happen.
link |
00:22:23.340
But compilers end up getting a lot of theorem proving
link |
00:22:26.180
and other kinds of algorithms that
link |
00:22:27.760
try to find higher level properties of the program that
link |
00:22:30.100
then can be used by the optimizer.
link |
00:22:32.280
Cool.
link |
00:22:32.780
So what's the biggest bang for the buck with optimization?
link |
00:22:38.140
Today?
link |
00:22:38.640
Yeah.
link |
00:22:39.140
Well, no, not even today.
link |
00:22:40.900
At the very beginning, the 80s, I don't know.
link |
00:22:42.900
Yeah, so for the 80s, a lot of it
link |
00:22:44.300
was things like register allocation.
link |
00:22:46.420
So the idea of in a modern microprocessor,
link |
00:22:50.460
what you'll end up having is you'll
link |
00:22:51.880
end up having memory, which is relatively slow.
link |
00:22:54.340
And then you have registers that are relatively fast.
link |
00:22:57.060
But registers, you don't have very many of them.
link |
00:23:00.340
And so when you're writing a bunch of code,
link |
00:23:02.600
you're just saying, compute this,
link |
00:23:04.180
put in a temporary variable, compute this, compute this,
link |
00:23:05.940
compute this, put in a temporary variable.
link |
00:23:07.780
I have a loop.
link |
00:23:08.220
I have some other stuff going on.
link |
00:23:09.780
Well, now you're running on an x86,
link |
00:23:11.660
like a desktop PC or something.
link |
00:23:13.900
Well, it only has, in some cases, some modes,
link |
00:23:16.860
eight registers.
link |
00:23:18.700
And so now the compiler has to choose what values get
link |
00:23:21.620
put in what registers at what points in the program.
link |
00:23:24.820
And this is actually a really big deal.
link |
00:23:26.580
So if you think about, you have a loop, an inner loop
link |
00:23:29.500
that executes millions of times maybe.
link |
00:23:31.620
If you're doing loads and stores inside that loop,
link |
00:23:33.620
then it's going to be really slow.
link |
00:23:35.040
But if you can somehow fit all the values inside that loop
link |
00:23:37.740
in registers, now it's really fast.
link |
00:23:40.180
And so getting that right requires a lot of work,
link |
00:23:43.020
because there's many different ways to do that.
link |
00:23:44.940
And often what the compiler ends up doing
link |
00:23:46.980
is it ends up thinking about things
link |
00:23:48.840
in a different representation than what the human wrote.
link |
00:23:52.020
You wrote into x.
link |
00:23:53.340
Well, the compiler thinks about that as four different values,
link |
00:23:56.820
each which have different lifetimes across the function
link |
00:23:59.280
that it's in.
link |
00:24:00.420
And each of those could be put in a register or memory
link |
00:24:03.180
or different memory or maybe in some parts of the code
link |
00:24:06.140
recomputed instead of stored and reloaded.
link |
00:24:08.360
And there are many of these different kinds of techniques
link |
00:24:10.700
that can be used.
link |
00:24:11.460
So it's adding almost like a time dimension to it's
link |
00:24:15.780
trying to optimize across time.
link |
00:24:18.300
So it's considering when you're programming,
link |
00:24:20.340
you're not thinking in that way.
link |
00:24:21.860
Yeah, absolutely.
link |
00:24:23.220
And so the RISC era made things.
link |
00:24:27.100
So RISC chips, R I S C. The RISC chips,
link |
00:24:32.020
as opposed to CISC chips.
link |
00:24:33.740
The RISC chips made things more complicated for the compiler,
link |
00:24:36.700
because what they ended up doing is ending up
link |
00:24:40.660
adding pipelines to the processor, where
link |
00:24:42.500
the processor can do more than one thing at a time.
link |
00:24:45.020
But this means that the order of operations matters a lot.
link |
00:24:47.740
So one of the classical compiler techniques that you use
link |
00:24:50.260
is called scheduling.
link |
00:24:51.940
And so moving the instructions around
link |
00:24:54.220
so that the processor can keep its pipelines full instead
link |
00:24:57.740
of stalling and getting blocked.
link |
00:24:59.220
And so there's a lot of things like that that
link |
00:25:01.180
are kind of bread and butter compiler techniques
link |
00:25:03.620
that have been studied a lot over the course of decades now.
link |
00:25:06.220
But the engineering side of making them real
link |
00:25:08.540
is also still quite hard.
link |
00:25:10.580
And you talk about machine learning.
link |
00:25:12.460
This is a huge opportunity for machine learning,
link |
00:25:14.420
because many of these algorithms are full of these
link |
00:25:17.620
hokey, hand rolled heuristics, which
link |
00:25:19.300
work well on specific benchmarks that don't generalize,
link |
00:25:21.820
and full of magic numbers.
link |
00:25:23.940
And I hear there's some techniques that
link |
00:25:26.620
are good at handling that.
link |
00:25:28.060
So what would be the, if you were to apply machine learning
link |
00:25:32.220
to this, what's the thing you're trying to optimize?
link |
00:25:34.740
Is it ultimately the running time?
link |
00:25:39.100
You can pick your metric, and there's running time,
link |
00:25:41.180
there's memory use, there's lots of different things
link |
00:25:43.900
that you can optimize for.
link |
00:25:44.940
Code size is another one that some people care about
link |
00:25:47.220
in the embedded space.
link |
00:25:48.860
Is this like the thinking into the future,
link |
00:25:51.700
or has somebody actually been crazy enough
link |
00:25:54.500
to try to have machine learning based parameter
link |
00:25:58.060
tuning for the optimization of compilers?
link |
00:26:01.060
So this is something that is, I would say, research right now.
link |
00:26:04.860
There are a lot of research systems
link |
00:26:06.820
that have been applying search in various forms.
link |
00:26:09.100
And using reinforcement learning is one form,
link |
00:26:11.460
but also brute force search has been tried for quite a while.
link |
00:26:14.460
And usually, these are in small problem spaces.
link |
00:26:18.180
So find the optimal way to code generate a matrix
link |
00:26:21.900
multiply for a GPU, something like that,
link |
00:26:24.460
where you say, there, there's a lot of design space of,
link |
00:26:28.580
do you unroll loops a lot?
link |
00:26:29.900
Do you execute multiple things in parallel?
link |
00:26:32.660
And there's many different confounding factors here
link |
00:26:35.340
because graphics cards have different numbers of threads
link |
00:26:38.100
and registers and execution ports and memory bandwidth
link |
00:26:41.020
and many different constraints that interact
link |
00:26:42.740
in nonlinear ways.
link |
00:26:44.460
And so search is very powerful for that.
link |
00:26:46.500
And it gets used in certain ways,
link |
00:26:49.820
but it's not very structured.
link |
00:26:51.220
This is something that we need,
link |
00:26:52.620
we as an industry need to fix.
link |
00:26:54.500
So you said 80s, but like, so have there been like big jumps
link |
00:26:59.220
in improvement and optimization?
link |
00:27:01.260
Yeah.
link |
00:27:02.340
Yeah, since then, what's the coolest thing?
link |
00:27:05.300
It's largely been driven by hardware.
link |
00:27:07.100
So, well, it's hardware and software.
link |
00:27:09.860
So in the mid nineties, Java totally changed the world,
link |
00:27:13.700
right?
link |
00:27:14.540
And I'm still amazed by how much change was introduced
link |
00:27:17.540
by the way or in a good way.
link |
00:27:19.340
So like reflecting back, Java introduced things like,
link |
00:27:22.420
all at once introduced things like JIT compilation.
link |
00:27:25.860
None of these were novel, but it pulled it together
link |
00:27:27.780
and made it mainstream and made people invest in it.
link |
00:27:30.580
JIT compilation, garbage collection, portable code,
link |
00:27:33.620
safe code, like memory safe code,
link |
00:27:36.620
like a very dynamic dispatch execution model.
link |
00:27:41.380
Like many of these things,
link |
00:27:42.620
which had been done in research systems
link |
00:27:44.060
and had been done in small ways in various places,
link |
00:27:46.900
really came to the forefront,
link |
00:27:47.980
really changed how things worked
link |
00:27:49.740
and therefore changed the way people thought
link |
00:27:51.980
about the problem.
link |
00:27:53.060
JavaScript was another major world change
link |
00:27:56.300
based on the way it works.
link |
00:27:59.300
But also on the hardware side of things,
link |
00:28:01.300
multi core and vector instructions really change
link |
00:28:06.660
the problem space and are very,
link |
00:28:09.460
they don't remove any of the problems
link |
00:28:10.820
that compilers faced in the past,
link |
00:28:12.380
but they add new kinds of problems
link |
00:28:14.540
of how do you find enough work
link |
00:28:16.380
to keep a four wide vector busy, right?
link |
00:28:20.020
Or if you're doing a matrix multiplication,
link |
00:28:22.660
how do you do different columns out of that matrix
link |
00:28:25.860
at the same time?
link |
00:28:26.700
And how do you maximally utilize the arithmetic compute
link |
00:28:30.140
that one core has?
link |
00:28:31.460
And then how do you take it to multiple cores?
link |
00:28:33.500
How did the whole virtual machine thing change
link |
00:28:35.780
the compilation pipeline?
link |
00:28:38.020
Yeah, so what the Java virtual machine does
link |
00:28:40.460
is it splits, just like I was talking about before,
link |
00:28:44.180
where you have a front end that parses the code,
link |
00:28:46.300
and then you have an intermediate representation
link |
00:28:48.020
that gets transformed.
link |
00:28:49.460
What Java did was they said,
link |
00:28:51.020
we will parse the code and then compile to
link |
00:28:53.100
what's known as Java byte code.
link |
00:28:55.500
And that byte code is now a portable code representation
link |
00:28:58.580
that is industry standard and locked down and can't change.
link |
00:29:02.420
And then the back part of the compiler
link |
00:29:05.100
that does optimization and code generation
link |
00:29:07.300
can now be built by different vendors.
link |
00:29:09.460
Okay.
link |
00:29:10.300
And Java byte code can be shipped around across the wire.
link |
00:29:13.020
It's memory safe and relatively trusted.
link |
00:29:16.860
And because of that, it can run in the browser.
link |
00:29:18.660
And that's why it runs in the browser, right?
link |
00:29:20.540
And so that way you can be in,
link |
00:29:22.980
again, back in the day, you would write a Java applet
link |
00:29:25.020
and as a web developer, you'd build this mini app
link |
00:29:29.300
that would run on a webpage.
link |
00:29:30.860
Well, a user of that is running a web browser
link |
00:29:33.620
on their computer.
link |
00:29:34.460
You download that Java byte code, which can be trusted,
link |
00:29:37.860
and then you do all the compiler stuff on your machine
link |
00:29:41.060
so that you know that you trust that.
link |
00:29:42.460
Now, is that a good idea or a bad idea?
link |
00:29:44.060
It's a great idea.
link |
00:29:44.900
I mean, it's a great idea for certain problems.
link |
00:29:46.240
And I'm very much a believer that technology is itself
link |
00:29:49.540
neither good nor bad.
link |
00:29:50.520
It's how you apply it.
link |
00:29:52.940
You know, this would be a very, very bad thing
link |
00:29:54.660
for very low levels of the software stack.
link |
00:29:56.980
But in terms of solving some of these software portability
link |
00:30:00.300
and transparency, or portability problems,
link |
00:30:02.820
I think it's been really good.
link |
00:30:04.240
Now, Java ultimately didn't win out on the desktop.
link |
00:30:06.600
And like, there are good reasons for that.
link |
00:30:09.420
But it's been very successful on servers and in many places,
link |
00:30:13.220
it's been a very successful thing over decades.
link |
00:30:16.300
So what has been LLVMs and C langs improvements
link |
00:30:21.300
and optimization that throughout its history,
link |
00:30:28.640
what are some moments we had set back
link |
00:30:31.080
and really proud of what's been accomplished?
link |
00:30:33.280
Yeah, I think that the interesting thing about LLVM
link |
00:30:36.160
is not the innovations and compiler research.
link |
00:30:40.120
It has very good implementations
link |
00:30:41.900
of various important algorithms, no doubt.
link |
00:30:44.880
And a lot of really smart people have worked on it.
link |
00:30:48.280
But I think that the thing that's most profound about LLVM
link |
00:30:50.560
is that through standardization, it made things possible
link |
00:30:53.840
that otherwise wouldn't have happened, okay?
link |
00:30:56.200
And so interesting things that have happened with LLVM,
link |
00:30:59.120
for example, Sony has picked up LLVM
link |
00:31:01.260
and used it to do all the graphics compilation
link |
00:31:03.920
in their movie production pipeline.
link |
00:31:06.080
And so now they're able to have better special effects
link |
00:31:07.920
because of LLVM.
link |
00:31:09.660
That's kind of cool.
link |
00:31:11.180
That's not what it was designed for, right?
link |
00:31:13.000
But that's the sign of good infrastructure
link |
00:31:15.480
when it can be used in ways it was never designed for
link |
00:31:18.800
because it has good layering and software engineering
link |
00:31:20.960
and it's composable and things like that.
link |
00:31:23.440
Which is where, as you said, it differs from GCC.
link |
00:31:26.120
Yes, GCC is also great in various ways,
link |
00:31:28.240
but it's not as good as infrastructure technology.
link |
00:31:31.800
It's really a C compiler, or it's a Fortran compiler.
link |
00:31:36.160
It's not infrastructure in the same way.
link |
00:31:38.920
Now you can tell I don't know what I'm talking about
link |
00:31:41.560
because I keep saying C lang.
link |
00:31:44.500
You can always tell when a person has clues,
link |
00:31:48.080
by the way, to pronounce something.
link |
00:31:49.400
I don't think, have I ever used C lang?
link |
00:31:52.580
Entirely possible, have you?
link |
00:31:54.120
Well, so you've used code, it's generated probably.
link |
00:31:58.200
So C lang and LLVM are used to compile
link |
00:32:01.760
all the apps on the iPhone effectively and the OSs.
link |
00:32:05.240
It compiles Google's production server applications.
link |
00:32:10.560
It's used to build GameCube games and PlayStation 4
link |
00:32:14.840
and things like that.
link |
00:32:16.680
So as a user, I have, but just everything I've done
link |
00:32:20.120
that I experienced with Linux has been,
link |
00:32:22.120
I believe, always GCC.
link |
00:32:23.560
Yeah, I think Linux still defaults to GCC.
link |
00:32:26.520
And is there a reason for that?
link |
00:32:27.800
Or is it because, I mean, is there a reason for that?
link |
00:32:29.440
It's a combination of technical and social reasons.
link |
00:32:32.040
Many Linux developers do use C lang,
link |
00:32:35.960
but the distributions, for lots of reasons,
link |
00:32:40.560
use GCC historically, and they've not switched, yeah.
link |
00:32:44.240
Because it's just anecdotally online,
link |
00:32:46.640
it seems that LLVM has either reached the level of GCC
link |
00:32:50.640
or superseded on different features or whatever.
link |
00:32:53.520
The way I would say it is that they're so close,
link |
00:32:55.200
it doesn't matter.
link |
00:32:56.040
Yeah, exactly.
link |
00:32:56.860
Like, they're slightly better in some ways,
link |
00:32:58.160
slightly worse than otherwise,
link |
00:32:59.160
but it doesn't actually really matter anymore, that level.
link |
00:33:03.280
So in terms of optimization breakthroughs,
link |
00:33:06.280
it's just been solid incremental work.
link |
00:33:09.160
Yeah, yeah, which describes a lot of compilers.
link |
00:33:12.520
The hard thing about compilers, in my experience,
link |
00:33:15.000
is the engineering, the software engineering,
link |
00:33:17.440
making it so that you can have hundreds of people
link |
00:33:20.160
collaborating on really detailed, low level work
link |
00:33:23.600
and scaling that.
link |
00:33:25.400
And that's really hard.
link |
00:33:27.880
And that's one of the things I think LLVM has done well.
link |
00:33:32.160
And that kind of goes back to the original design goals
link |
00:33:34.200
with it to be modular and things like that.
link |
00:33:37.200
And incidentally, I don't want to take all the credit
link |
00:33:38.880
for this, right?
link |
00:33:39.720
I mean, some of the best parts about LLVM
link |
00:33:41.760
is that it was designed to be modular.
link |
00:33:43.600
And when I started, I would write, for example,
link |
00:33:45.600
a register allocator, and then somebody much smarter than me
link |
00:33:48.500
would come in and pull it out and replace it
link |
00:33:50.720
with something else that they would come up with.
link |
00:33:52.680
And because it's modular, they were able to do that.
link |
00:33:55.200
And that's one of the challenges with GCC, for example,
link |
00:33:58.280
is replacing subsystems is incredibly difficult.
link |
00:34:01.280
It can be done, but it wasn't designed for that.
link |
00:34:04.680
And that's one of the reasons that LLVM's been
link |
00:34:06.080
very successful in the research world as well.
link |
00:34:08.760
But in a community sense, Guido van Rossum, right,
link |
00:34:12.960
from Python, just retired from, what is it?
link |
00:34:18.480
Benevolent Dictator for Life, right?
link |
00:34:20.500
So in managing this community of brilliant compiler folks,
link |
00:34:24.720
is there, did it, for a time at least,
link |
00:34:28.660
fall on you to approve things?
link |
00:34:31.480
Oh yeah, so I mean, I still have something like
link |
00:34:34.240
an order of magnitude more patches in LLVM
link |
00:34:37.980
than anybody else, and many of those I wrote myself.
link |
00:34:42.760
But you still write, I mean, you're still close to the,
link |
00:34:47.880
to the, I don't know what the expression is,
link |
00:34:49.480
to the metal, you still write code.
link |
00:34:51.000
Yeah, I still write code.
link |
00:34:52.220
Not as much as I was able to in grad school,
link |
00:34:54.240
but that's an important part of my identity.
link |
00:34:56.760
But the way that LLVM has worked over time
link |
00:34:58.880
is that when I was a grad student, I could do all the work
link |
00:35:01.360
and steer everything and review every patch
link |
00:35:04.120
and make sure everything was done
link |
00:35:05.800
exactly the way my opinionated sense
link |
00:35:09.040
felt like it should be done, and that was fine.
link |
00:35:11.760
But as things scale, you can't do that, right?
link |
00:35:14.300
And so what ends up happening is LLVM
link |
00:35:17.100
has a hierarchical system of what's called code owners.
link |
00:35:20.520
These code owners are given the responsibility
link |
00:35:22.880
not to do all the work,
link |
00:35:24.880
not necessarily to review all the patches,
link |
00:35:26.640
but to make sure that the patches do get reviewed
link |
00:35:28.800
and make sure that the right thing's happening
link |
00:35:30.320
architecturally in their area.
link |
00:35:32.160
And so what you'll see is you'll see that, for example,
link |
00:35:36.720
hardware manufacturers end up owning
link |
00:35:38.560
the hardware specific parts of their hardware.
link |
00:35:43.600
That's very common.
link |
00:35:45.520
Leaders in the community that have done really good work
link |
00:35:47.720
naturally become the de facto owner of something.
link |
00:35:50.880
And then usually somebody else is like,
link |
00:35:53.400
how about we make them the official code owner?
link |
00:35:55.520
And then we'll have somebody to make sure
link |
00:35:58.600
that all the patches get reviewed in a timely manner.
link |
00:36:00.320
And then everybody's like, yes, that's obvious.
link |
00:36:02.080
And then it happens, right?
link |
00:36:03.240
And usually this is a very organic thing, which is great.
link |
00:36:06.080
And so I'm nominally the top of that stack still,
link |
00:36:08.740
but I don't spend a lot of time reviewing patches.
link |
00:36:11.560
What I do is I help negotiate a lot of the technical
link |
00:36:16.520
disagreements that end up happening
link |
00:36:18.040
and making sure that the community as a whole
link |
00:36:19.660
makes progress and is moving in the right direction
link |
00:36:22.040
and doing that.
link |
00:36:23.920
So we also started a nonprofit six years ago,
link |
00:36:28.240
seven years ago, time's gone away.
link |
00:36:30.840
And the LLVM Foundation nonprofit helps oversee
link |
00:36:34.600
all the business sides of things and make sure
link |
00:36:36.440
that the events that the LLVM community has
link |
00:36:38.800
are funded and set up and run correctly
link |
00:36:41.600
and stuff like that.
link |
00:36:42.800
But the foundation is very much stays out
link |
00:36:45.160
of the technical side of where the project is going.
link |
00:36:49.060
Right, so it sounds like a lot of it is just organic.
link |
00:36:53.160
Yeah, well, LLVM is almost 20 years old,
link |
00:36:55.680
which is hard to believe.
link |
00:36:56.600
Somebody pointed out to me recently that LLVM
link |
00:36:59.720
is now older than GCC was when LLVM started, right?
link |
00:37:04.600
So time has a way of getting away from you.
link |
00:37:06.860
But the good thing about that is it has a really robust,
link |
00:37:10.400
really amazing community of people that are
link |
00:37:13.520
in their professional lives, spread across lots
link |
00:37:15.460
of different companies, but it's a community
link |
00:37:17.720
of people that are interested in similar kinds of problems
link |
00:37:21.120
and have been working together effectively for years
link |
00:37:23.680
and have a lot of trust and respect for each other.
link |
00:37:26.460
And even if they don't always agree that we're able
link |
00:37:29.240
to find a path forward.
link |
00:37:31.200
So then in a slightly different flavor of effort,
link |
00:37:34.480
you started at Apple in 2005 with the task
link |
00:37:38.120
of making, I guess, LLVM production ready.
link |
00:37:41.800
And then eventually 2013 through 2017,
link |
00:37:44.640
leading the entire developer tools department.
link |
00:37:48.360
We're talking about LLVM, Xcode, Objective C to Swift.
link |
00:37:53.920
So in a quick overview of your time there,
link |
00:37:58.580
what were the challenges?
link |
00:37:59.600
First of all, leading such a huge group of developers,
link |
00:38:03.240
what was the big motivator, dream, mission
link |
00:38:06.540
behind creating Swift, the early birth of it
link |
00:38:11.400
from Objective C and so on, and Xcode,
link |
00:38:13.400
what are some challenges?
link |
00:38:14.240
So these are different questions.
link |
00:38:15.900
Yeah, I know, but I wanna talk about the other stuff too.
link |
00:38:19.720
I'll stay on the technical side,
link |
00:38:21.240
then we can talk about the big team pieces, if that's okay.
link |
00:38:24.480
So it's to really oversimplify many years of hard work.
link |
00:38:29.060
LLVM started, joined Apple, became a thing,
link |
00:38:32.440
became successful and became deployed.
link |
00:38:34.600
But then there's a question about
link |
00:38:35.960
how do we actually parse the source code?
link |
00:38:38.880
So LLVM is that back part,
link |
00:38:40.320
the optimizer and the code generator.
link |
00:38:42.320
And LLVM was really good for Apple
link |
00:38:44.060
as it went through a couple of harder transitions.
link |
00:38:46.060
I joined right at the time of the Intel transition,
link |
00:38:47.960
for example, and 64 bit transitions,
link |
00:38:51.820
and then the transition to ARM with the iPhone.
link |
00:38:53.500
And so LLVM was very useful
link |
00:38:54.720
for some of these kinds of things.
link |
00:38:57.000
But at the same time, there's a lot of questions
link |
00:38:58.480
around developer experience.
link |
00:39:00.120
And so if you're a programmer pounding out
link |
00:39:01.960
at the time Objective C code,
link |
00:39:04.480
the error message you get, the compile time,
link |
00:39:06.520
the turnaround cycle, the tooling and the IDE,
link |
00:39:09.760
were not great, were not as good as they could be.
link |
00:39:13.000
And so, as I occasionally do, I'm like,
link |
00:39:18.080
well, okay, how hard is it to write a C compiler?
link |
00:39:20.720
And so I'm not gonna commit to anybody,
link |
00:39:22.560
I'm not gonna tell anybody, I'm just gonna just do it
link |
00:39:25.320
nights and weekends and start working on it.
link |
00:39:27.480
And then I built up in C,
link |
00:39:29.740
there's this thing called the preprocessor,
link |
00:39:31.160
which people don't like,
link |
00:39:33.040
but it's actually really hard and complicated
link |
00:39:35.480
and includes a bunch of really weird things
link |
00:39:37.700
like trigraphs and other stuff like that
link |
00:39:39.280
that are really nasty,
link |
00:39:40.960
and it's the crux of a bunch of the performance issues
link |
00:39:44.080
in the compiler.
link |
00:39:45.640
Started working on the parser
link |
00:39:46.640
and kind of got to the point where I'm like,
link |
00:39:47.800
ah, you know what, we could actually do this.
link |
00:39:49.880
Everybody's saying that this is impossible to do,
link |
00:39:51.460
but it's actually just hard, it's not impossible.
link |
00:39:53.960
And eventually told my manager about it,
link |
00:39:57.560
and he's like, oh, wow, this is great,
link |
00:39:59.220
we do need to solve this problem.
link |
00:40:00.360
Oh, this is great, we can get you one other person
link |
00:40:02.560
to work with you on this, you know?
link |
00:40:04.440
And slowly a team is formed and it starts taking off.
link |
00:40:08.360
And C++, for example, huge, complicated language.
link |
00:40:12.040
People always assume that it's impossible to implement
link |
00:40:14.360
and it's very nearly impossible,
link |
00:40:16.260
but it's just really, really hard.
link |
00:40:18.720
And the way to get there is to build it
link |
00:40:20.840
one piece at a time incrementally.
link |
00:40:22.480
And that was only possible because we were lucky
link |
00:40:26.440
to hire some really exceptional engineers
link |
00:40:28.160
that knew various parts of it very well
link |
00:40:30.380
and could do great things.
link |
00:40:32.680
Swift was kind of a similar thing.
link |
00:40:34.440
So Swift came from, we were just finishing off
link |
00:40:39.160
the first version of C++ support in Clang.
link |
00:40:42.600
And C++ is a very formidable and very important language,
link |
00:40:47.260
but it's also ugly in lots of ways.
link |
00:40:49.280
And you can't influence C++ without thinking
link |
00:40:52.320
there has to be a better thing, right?
link |
00:40:54.380
And so I started working on Swift, again,
link |
00:40:56.120
with no hope or ambition that would go anywhere,
link |
00:40:58.560
just let's see what could be done,
link |
00:41:00.800
let's play around with this thing.
link |
00:41:02.620
It was me in my spare time, not telling anybody about it,
link |
00:41:06.700
kind of a thing, and it made some good progress.
link |
00:41:09.420
I'm like, actually, it would make sense to do this.
link |
00:41:11.260
At the same time, I started talking with the senior VP
link |
00:41:14.800
of software at the time, a guy named Bertrand Serlet.
link |
00:41:17.720
And Bertrand was very encouraging.
link |
00:41:19.280
He was like, well, let's have fun, let's talk about this.
link |
00:41:22.080
And he was a little bit of a language guy,
link |
00:41:23.440
and so he helped guide some of the early work
link |
00:41:26.160
and encouraged me and got things off the ground.
link |
00:41:30.420
And eventually told my manager and told other people,
link |
00:41:34.280
and it started making progress.
link |
00:41:38.800
The complicating thing with Swift
link |
00:41:40.960
was that the idea of doing a new language
link |
00:41:43.880
was not obvious to anybody, including myself.
link |
00:41:47.840
And the tone at the time was that the iPhone
link |
00:41:50.240
was successful because of Objective C.
link |
00:41:53.440
Oh, interesting.
link |
00:41:54.440
Not despite of or just because of.
link |
00:41:57.160
And you have to understand that at the time,
link |
00:42:01.160
Apple was hiring software people that loved Objective C.
link |
00:42:05.400
And it wasn't that they came despite Objective C.
link |
00:42:07.960
They loved Objective C, and that's why they got hired.
link |
00:42:10.240
And so you had a software team that the leadership,
link |
00:42:13.080
in many cases, went all the way back to Next,
link |
00:42:15.200
where Objective C really became real.
link |
00:42:19.400
And so they, quote unquote, grew up writing Objective C.
link |
00:42:23.240
And many of the individual engineers
link |
00:42:25.720
all were hired because they loved Objective C.
link |
00:42:28.360
And so this notion of, OK, let's do new language
link |
00:42:30.560
was kind of heretical in many ways.
link |
00:42:34.120
Meanwhile, my sense was that the outside community wasn't really
link |
00:42:36.960
in love with Objective C. Some people were,
link |
00:42:38.560
and some of the most outspoken people were.
link |
00:42:40.360
But other people were hitting challenges
link |
00:42:42.620
because it has very sharp corners
link |
00:42:44.760
and it's difficult to learn.
link |
00:42:46.840
And so one of the challenges of making Swift happen that
link |
00:42:50.160
was totally non technical is the social part of what do we do?
link |
00:42:57.720
If we do a new language, which at Apple, many things
link |
00:43:00.320
happen that don't ship.
link |
00:43:02.240
So if we ship it, what is the metrics of success?
link |
00:43:05.560
Why would we do this?
link |
00:43:06.400
Why wouldn't we make Objective C better?
link |
00:43:08.060
If Objective C has problems, let's file off
link |
00:43:10.160
those rough corners and edges.
link |
00:43:12.160
And one of the major things that became the reason to do this
link |
00:43:15.640
was this notion of safety, memory safety.
link |
00:43:18.960
And the way Objective C works is that a lot of the object system
link |
00:43:23.240
and everything else is built on top of pointers in C.
link |
00:43:27.560
Objective C is an extension on top of C.
link |
00:43:29.960
And so pointers are unsafe.
link |
00:43:32.680
And if you get rid of the pointers,
link |
00:43:34.640
it's not Objective C anymore.
link |
00:43:36.480
And so fundamentally, that was an issue
link |
00:43:39.080
that you could not fix safety or memory safety
link |
00:43:42.200
without fundamentally changing the language.
link |
00:43:45.640
And so once we got through that part of the mental process
link |
00:43:49.920
and the thought process, it became a design process
link |
00:43:53.200
of saying, OK, well, if we're going to do something new,
link |
00:43:55.400
what is good?
link |
00:43:56.280
How do we think about this?
link |
00:43:57.400
And what do we like?
link |
00:43:58.200
And what are we looking for?
link |
00:44:00.040
And that was a very different phase of it.
link |
00:44:02.440
So what are some design choices early on in Swift?
link |
00:44:05.960
Like we're talking about braces, are you
link |
00:44:10.120
making a typed language or not, all those kinds of things.
link |
00:44:13.240
Yeah, so some of those were obvious given the context.
link |
00:44:16.040
So a typed language, for example,
link |
00:44:17.800
Objective C is a typed language.
link |
00:44:19.200
And going with an untyped language
link |
00:44:22.480
wasn't really seriously considered.
link |
00:44:24.320
We wanted the performance, and we
link |
00:44:26.000
wanted refactoring tools and other things
link |
00:44:27.680
like that that go with typed languages.
link |
00:44:29.600
Quick, dumb question.
link |
00:44:31.440
Was it obvious, I think this would be a dumb question,
link |
00:44:34.600
but was it obvious that the language
link |
00:44:36.360
has to be a compiled language?
link |
00:44:40.120
Yes, that's not a dumb question.
link |
00:44:42.080
Earlier, I think late 90s, Apple had seriously
link |
00:44:44.520
considered moving its development experience to Java.
link |
00:44:49.000
But Swift started in 2010, which was several years
link |
00:44:53.160
after the iPhone.
link |
00:44:53.880
It was when the iPhone was definitely
link |
00:44:55.380
on an upward trajectory.
link |
00:44:56.640
And the iPhone was still extremely,
link |
00:44:58.760
and is still a bit memory constrained.
link |
00:45:01.800
And so being able to compile the code
link |
00:45:04.440
and then ship it and then having standalone code that
link |
00:45:08.160
is not JIT compiled is a very big deal
link |
00:45:11.320
and is very much part of the Apple value system.
link |
00:45:15.200
Now, JavaScript's also a thing.
link |
00:45:17.480
I mean, it's not that this is exclusive,
link |
00:45:19.360
and technologies are good depending
link |
00:45:21.640
on how they're applied.
link |
00:45:23.880
But in the design of Swift, saying,
link |
00:45:26.600
how can we make Objective C better?
link |
00:45:28.320
Objective C is statically compiled,
link |
00:45:29.760
and that was the contiguous, natural thing to do.
link |
00:45:32.520
Just skip ahead a little bit, and we'll go right back.
link |
00:45:35.360
Just as a question, as you think about today in 2019
link |
00:45:40.040
in your work at Google, TensorFlow and so on,
link |
00:45:42.400
is, again, compilations, static compilation still
link |
00:45:48.600
the right thing?
link |
00:45:49.460
Yeah, so the funny thing after working
link |
00:45:52.000
on compilers for a really long time is that,
link |
00:45:55.880
and this is one of the things that LLVM has helped with,
link |
00:45:59.040
is that I don't look at compilations
link |
00:46:01.440
being static or dynamic or interpreted or not.
link |
00:46:05.240
This is a spectrum.
link |
00:46:07.680
And one of the cool things about Swift
link |
00:46:09.140
is that Swift is not just statically compiled.
link |
00:46:12.160
It's actually dynamically compiled as well,
link |
00:46:14.080
and it can also be interpreted.
link |
00:46:15.320
Though, nobody's actually done that.
link |
00:46:17.440
And so what ends up happening when
link |
00:46:20.400
you use Swift in a workbook, for example in Colab or in Jupyter,
link |
00:46:24.080
is it's actually dynamically compiling the statements
link |
00:46:26.360
as you execute them.
link |
00:46:28.160
And so this gets back to the software engineering problems,
link |
00:46:32.840
where if you layer the stack properly,
link |
00:46:34.960
you can actually completely change
link |
00:46:37.320
how and when things get compiled because you
link |
00:46:39.360
have the right abstractions there.
link |
00:46:41.120
And so the way that a Colab workbook works with Swift
link |
00:46:44.800
is that when you start typing into it,
link |
00:46:47.720
it creates a process, a Unix process.
link |
00:46:50.280
And then each line of code you type in,
link |
00:46:52.160
it compiles it through the Swift compiler, the front end part,
link |
00:46:56.120
and then sends it through the optimizer,
link |
00:46:58.360
JIT compiles machine code, and then
link |
00:47:01.120
injects it into that process.
link |
00:47:03.800
And so as you're typing new stuff,
link |
00:47:05.400
it's like squirting in new code and overwriting and replacing
link |
00:47:09.360
and updating code in place.
link |
00:47:11.200
And the fact that it can do this is not an accident.
link |
00:47:13.680
Swift was designed for this.
link |
00:47:15.560
But it's an important part of how the language was set up
link |
00:47:18.120
and how it's layered, and this is a nonobvious piece.
link |
00:47:21.320
And one of the things with Swift that
link |
00:47:23.160
was, for me, a very strong design point
link |
00:47:25.880
is to make it so that you can learn it very quickly.
link |
00:47:29.640
And so from a language design perspective,
link |
00:47:31.880
the thing that I always come back to
link |
00:47:33.340
is this UI principle of progressive disclosure
link |
00:47:36.440
of complexity.
link |
00:47:37.960
And so in Swift, you can start by saying print, quote,
link |
00:47:41.680
hello world, quote.
link |
00:47:44.040
And there's no slash n, just like Python, one line of code,
link |
00:47:47.160
no main, no header files, no public static class void,
link |
00:47:51.520
blah, blah, blah, string like Java has, one line of code.
link |
00:47:55.640
And you can teach that, and it works great.
link |
00:47:58.400
Then you can say, well, let's introduce variables.
link |
00:48:00.400
And so you can declare a variable with var.
link |
00:48:02.400
So var x equals 4.
link |
00:48:03.780
What is a variable?
link |
00:48:04.700
You can use x, x plus 1.
link |
00:48:06.280
This is what it means.
link |
00:48:07.600
Then you can say, well, how about control flow?
link |
00:48:09.520
Well, this is what an if statement is.
link |
00:48:10.860
This is what a for statement is.
link |
00:48:12.280
This is what a while statement is.
link |
00:48:15.280
Then you can say, let's introduce functions.
link |
00:48:17.280
And many languages like Python have
link |
00:48:20.020
had this kind of notion of let's introduce small things,
link |
00:48:22.820
and then you can add complexity.
link |
00:48:24.400
Then you can introduce classes.
link |
00:48:25.760
And then you can add generics, in the case of Swift.
link |
00:48:28.040
And then you can build in modules
link |
00:48:29.520
and build out in terms of the things that you're expressing.
link |
00:48:32.200
But this is not very typical for compiled languages.
link |
00:48:35.800
And so this was a very strong design point,
link |
00:48:38.000
and one of the reasons that Swift, in general,
link |
00:48:40.960
is designed with this factoring of complexity in mind
link |
00:48:43.480
so that the language can express powerful things.
link |
00:48:46.440
You can write firmware in Swift if you want to.
link |
00:48:49.280
But it has a very high level feel,
link |
00:48:51.900
which is really this perfect blend, because often you
link |
00:48:55.200
have very advanced library writers that
link |
00:48:57.520
want to be able to use the nitty gritty details.
link |
00:49:00.520
But then other people just want to use the libraries
link |
00:49:02.960
and work at a higher abstraction level.
link |
00:49:04.880
It's kind of cool that I saw that you can just
link |
00:49:07.240
interoperability.
link |
00:49:09.240
I don't think I pronounced that word enough.
link |
00:49:11.320
But you can just drag in Python.
link |
00:49:14.960
It's just strange.
link |
00:49:16.000
You can import, like I saw this in the demo.
link |
00:49:19.640
How do you make that happen?
link |
00:49:21.280
What's up with that?
link |
00:49:23.120
Is that as easy as it looks, or is it?
link |
00:49:25.560
Yes, as easy as it looks.
link |
00:49:27.000
That's not a stage magic hack or anything like that.
link |
00:49:29.600
I don't mean from the user perspective.
link |
00:49:31.400
I mean from the implementation perspective to make it happen.
link |
00:49:34.120
So it's easy once all the pieces are in place.
link |
00:49:37.000
The way it works, so if you think about a dynamically typed
link |
00:49:39.280
language like Python, you can think about it
link |
00:49:41.480
in two different ways.
link |
00:49:42.360
You can say it has no types, which
link |
00:49:45.800
is what most people would say.
link |
00:49:47.480
Or you can say it has one type.
link |
00:49:50.400
And you can say it has one type, and it's the Python object.
link |
00:49:53.320
And the Python object gets passed around.
link |
00:49:55.000
And because there's only one type, it's implicit.
link |
00:49:58.200
And so what happens with Swift and Python talking
link |
00:50:00.880
to each other, Swift has lots of types.
link |
00:50:02.760
It has arrays, and it has strings, and all classes,
link |
00:50:05.840
and that kind of stuff.
link |
00:50:07.000
But it now has a Python object type.
link |
00:50:11.120
So there is one Python object type.
link |
00:50:12.720
And so when you say import NumPy, what you get
link |
00:50:16.440
is a Python object, which is the NumPy module.
link |
00:50:19.840
And then you say np.array.
link |
00:50:21.960
It says, OK, hey, Python object, I have no idea what you are.
link |
00:50:24.960
Give me your array member.
link |
00:50:27.280
OK, cool.
link |
00:50:27.960
And it just uses dynamic stuff, talks to the Python interpreter,
link |
00:50:31.160
and says, hey, Python, what's the.array member
link |
00:50:33.680
in that Python object?
link |
00:50:35.720
It gives you back another Python object.
link |
00:50:37.400
And now you say parentheses for the call and the arguments
link |
00:50:40.040
you're going to pass.
link |
00:50:40.920
And so then it says, hey, a Python object
link |
00:50:43.520
that is the result of np.array, call with these arguments.
link |
00:50:47.840
Again, calling into the Python interpreter to do that work.
link |
00:50:50.320
And so right now, this is all really simple.
link |
00:50:53.680
And if you dive into the code, what you'll see
link |
00:50:55.960
is that the Python module in Swift
link |
00:50:58.440
is something like 1,200 lines of code or something.
link |
00:51:01.360
It's written in pure Swift.
link |
00:51:02.400
It's super simple.
link |
00:51:03.560
And it's built on top of the C interoperability
link |
00:51:06.560
because it just talks to the Python interpreter.
link |
00:51:09.520
But making that possible required
link |
00:51:11.080
us to add two major language features to Swift
link |
00:51:13.480
to be able to express these dynamic calls
link |
00:51:15.400
and the dynamic member lookups.
link |
00:51:17.240
And so what we've done over the last year
link |
00:51:19.480
is we've proposed, implement, standardized, and contributed
link |
00:51:23.960
new language features to the Swift language
link |
00:51:26.160
in order to make it so it is really trivial.
link |
00:51:29.560
And this is one of the things about Swift
link |
00:51:31.320
that is critical to the Swift for TensorFlow work, which
link |
00:51:35.000
is that we can actually add new language features.
link |
00:51:37.200
And the bar for adding those is high,
link |
00:51:39.160
but it's what makes it possible.
link |
00:51:42.280
So you're now at Google doing incredible work
link |
00:51:45.240
on several things, including TensorFlow.
link |
00:51:47.680
So TensorFlow 2.0 or whatever leading up to 2.0 has,
link |
00:51:53.080
by default, in 2.0, has eager execution.
link |
00:51:56.840
And yet, in order to make code optimized for GPU or TPU
link |
00:52:00.520
or some of these systems, computation
link |
00:52:04.120
needs to be converted to a graph.
link |
00:52:06.000
So what's that process like?
link |
00:52:07.440
What are the challenges there?
link |
00:52:08.960
Yeah, so I am tangentially involved in this.
link |
00:52:11.720
But the way that it works with Autograph
link |
00:52:15.280
is that you mark your function with a decorator.
link |
00:52:21.600
And when Python calls it, that decorator is invoked.
link |
00:52:24.280
And then it says, before I call this function,
link |
00:52:28.240
you can transform it.
link |
00:52:29.480
And so the way Autograph works is, as far as I understand,
link |
00:52:32.400
is it actually uses the Python parser
link |
00:52:34.440
to go parse that, turn it into a syntax tree,
link |
00:52:37.160
and now apply compiler techniques to, again,
link |
00:52:39.400
transform this down into TensorFlow graphs.
link |
00:52:42.320
And so you can think of it as saying, hey,
link |
00:52:44.920
I have an if statement.
link |
00:52:45.880
I'm going to create an if node in the graph,
link |
00:52:48.360
like you say tf.cond.
link |
00:52:51.080
You have a multiply.
link |
00:52:53.040
Well, I'll turn that into a multiply node in the graph.
link |
00:52:55.320
And it becomes this tree transformation.
link |
00:52:57.760
So where does the Swift for TensorFlow
link |
00:53:00.480
come in, which is parallels?
link |
00:53:04.960
For one, Swift is an interface.
link |
00:53:06.960
Like, Python is an interface to TensorFlow.
link |
00:53:09.200
But it seems like there's a lot more going on in just
link |
00:53:11.760
a different language interface.
link |
00:53:13.120
There's optimization methodology.
link |
00:53:15.960
So the TensorFlow world has a couple
link |
00:53:17.920
of different what I'd call front end technologies.
link |
00:53:21.240
And so Swift and Python and Go and Rust and Julia
link |
00:53:25.240
and all these things share the TensorFlow graphs
link |
00:53:29.320
and all the runtime and everything that's later.
link |
00:53:32.760
And so Swift for TensorFlow is merely another front end
link |
00:53:36.640
for TensorFlow, just like any of these other systems are.
link |
00:53:40.640
There's a major difference between, I would say,
link |
00:53:43.080
three camps of technologies here.
link |
00:53:44.600
There's Python, which is a special case,
link |
00:53:46.880
because the vast majority of the community effort
link |
00:53:49.160
is going to the Python interface.
link |
00:53:51.120
And Python has its own approaches
link |
00:53:52.920
for automatic differentiation.
link |
00:53:54.480
It has its own APIs and all this kind of stuff.
link |
00:53:58.160
There's Swift, which I'll talk about in a second.
link |
00:54:00.320
And then there's kind of everything else.
link |
00:54:02.040
And so the everything else are effectively language bindings.
link |
00:54:05.400
So they call into the TensorFlow runtime,
link |
00:54:07.960
but they usually don't have automatic differentiation
link |
00:54:10.920
or they usually don't provide anything other than APIs
link |
00:54:14.560
that call the C APIs in TensorFlow.
link |
00:54:16.440
And so they're kind of wrappers for that.
link |
00:54:18.360
Swift is really kind of special.
link |
00:54:19.840
And it's a very different approach.
link |
00:54:22.760
Swift for TensorFlow, that is, is a very different approach.
link |
00:54:25.360
Because there we're saying, let's
link |
00:54:26.880
look at all the problems that need
link |
00:54:28.400
to be solved in the full stack of the TensorFlow compilation
link |
00:54:34.080
process, if you think about it that way.
link |
00:54:35.680
Because TensorFlow is fundamentally a compiler.
link |
00:54:38.200
It takes models, and then it makes them go fast on hardware.
link |
00:54:42.760
That's what a compiler does.
link |
00:54:43.880
And it has a front end, it has an optimizer,
link |
00:54:47.560
and it has many back ends.
link |
00:54:49.320
And so if you think about it the right way,
link |
00:54:51.680
or if you look at it in a particular way,
link |
00:54:54.800
it is a compiler.
link |
00:54:59.280
And so Swift is merely another front end.
link |
00:55:02.120
But it's saying, and the design principle is saying,
link |
00:55:05.560
let's look at all the problems that we face as machine
link |
00:55:08.240
learning practitioners and what is the best possible way we
link |
00:55:11.320
can do that, given the fact that we can change literally
link |
00:55:13.840
anything in this entire stack.
link |
00:55:15.920
And Python, for example, where the vast majority
link |
00:55:18.440
of the engineering and effort has gone into,
link |
00:55:22.600
is constrained by being the best possible thing you
link |
00:55:25.000
can do with a Python library.
link |
00:55:27.320
There are no Python language features
link |
00:55:29.320
that are added because of machine learning
link |
00:55:31.040
that I'm aware of.
link |
00:55:32.600
They added a matrix multiplication operator
link |
00:55:34.640
with that, but that's as close as you get.
link |
00:55:38.320
And so with Swift, it's hard, but you
link |
00:55:41.460
can add language features to the language.
link |
00:55:43.800
And there's a community process for that.
link |
00:55:46.040
And so we look at these things and say, well,
link |
00:55:48.200
what is the right division of labor
link |
00:55:49.720
between the human programmer and the compiler?
link |
00:55:52.000
And Swift has a number of things that shift that balance.
link |
00:55:55.280
So because it has a type system, for example,
link |
00:56:00.560
that makes certain things possible for analysis
link |
00:56:02.680
of the code, and the compiler can automatically
link |
00:56:05.560
build graphs for you without you thinking about them.
link |
00:56:08.880
That's a big deal for a programmer.
link |
00:56:10.520
You just get free performance.
link |
00:56:11.680
You get clustering and fusion and optimization,
link |
00:56:14.400
things like that, without you as a programmer
link |
00:56:17.040
having to manually do it because the compiler can do it for you.
link |
00:56:20.080
Automatic differentiation is another big deal.
link |
00:56:22.240
And I think one of the key contributions of the Swift
link |
00:56:25.960
TensorFlow project is that there's
link |
00:56:29.640
this entire body of work on automatic differentiation
link |
00:56:32.120
that dates back to the Fortran days.
link |
00:56:34.120
People doing a tremendous amount of numerical computing
link |
00:56:36.400
in Fortran used to write these what they call source
link |
00:56:39.360
to source translators, where you take a bunch of code,
link |
00:56:43.280
shove it into a mini compiler, and it would push out
link |
00:56:46.640
more Fortran code.
link |
00:56:48.080
But it would generate the backwards passes
link |
00:56:50.240
for your functions for you, the derivatives.
link |
00:56:53.000
And so in that work in the 70s, a tremendous number
link |
00:56:57.840
of optimizations, a tremendous number of techniques
link |
00:57:01.160
for fixing numerical instability,
link |
00:57:02.920
and other kinds of problems were developed.
link |
00:57:05.080
But they're very difficult to port into a world
link |
00:57:07.600
where, in eager execution, you get an op by op at a time.
link |
00:57:11.280
You need to be able to look at an entire function
link |
00:57:13.280
and be able to reason about what's going on.
link |
00:57:15.720
And so when you have a language integrated automatic
link |
00:57:18.720
differentiation, which is one of the things
link |
00:57:20.520
that the Swift project is focusing on,
link |
00:57:22.760
you can open all these techniques
link |
00:57:24.680
and reuse them in familiar ways.
link |
00:57:28.640
But the language integration piece
link |
00:57:30.120
has a bunch of design room in it, and it's also complicated.
link |
00:57:33.240
The other piece of the puzzle here that's kind of interesting
link |
00:57:35.680
is TPUs at Google.
link |
00:57:37.560
So we're in a new world with deep learning.
link |
00:57:40.200
It constantly is changing, and I imagine,
link |
00:57:42.960
without disclosing anything, I imagine
link |
00:57:46.360
you're still innovating on the TPU front, too.
link |
00:57:48.400
Indeed.
link |
00:57:49.040
So how much interplay is there between software and hardware
link |
00:57:53.560
in trying to figure out how to together move
link |
00:57:55.240
towards an optimized solution?
link |
00:57:56.680
There's an incredible amount.
link |
00:57:57.760
So we're on our third generation of TPUs,
link |
00:57:59.480
which are now 100 petaflops in a very large liquid cooled box,
link |
00:58:04.640
virtual box with no cover.
link |
00:58:07.720
And as you might imagine, we're not out of ideas yet.
link |
00:58:11.240
The great thing about TPUs is that they're
link |
00:58:14.360
a perfect example of hardware software co design.
link |
00:58:17.520
And so it's about saying, what hardware
link |
00:58:19.800
do we build to solve certain classes of machine learning
link |
00:58:23.240
problems?
link |
00:58:23.840
Well, the algorithms are changing.
link |
00:58:26.480
The hardware takes some cases years to produce.
link |
00:58:30.360
And so you have to make bets and decide
link |
00:58:32.760
what is going to happen and what is the best way to spend
link |
00:58:36.520
the transistors to get the maximum performance per watt
link |
00:58:39.920
or area per cost or whatever it is that you're optimizing for.
link |
00:58:44.000
And so one of the amazing things about TPUs
link |
00:58:46.560
is this numeric format called bfloat16.
link |
00:58:49.960
bfloat16 is a compressed 16 bit floating point format,
link |
00:58:54.120
but it puts the bits in different places.
link |
00:58:55.960
And in numeric terms, it has a smaller mantissa
link |
00:58:58.960
and a larger exponent.
link |
00:59:00.400
That means that it's less precise,
link |
00:59:02.960
but it can represent larger ranges of values,
link |
00:59:05.680
which in the machine learning context
link |
00:59:07.280
is really important and useful because sometimes you
link |
00:59:09.960
have very small gradients you want to accumulate
link |
00:59:13.920
and very, very small numbers that
link |
00:59:17.480
are important to move things as you're learning.
link |
00:59:20.520
But sometimes you have very large magnitude numbers as well.
link |
00:59:23.160
And bfloat16 is not as precise.
link |
00:59:26.880
The mantissa is small.
link |
00:59:28.040
But it turns out the machine learning algorithms actually
link |
00:59:30.360
want to generalize.
link |
00:59:31.520
And so there's theories that this actually
link |
00:59:34.320
increases the ability for the network
link |
00:59:36.440
to generalize across data sets.
link |
00:59:37.960
And regardless of whether it's good or bad,
link |
00:59:41.160
it's much cheaper at the hardware level to implement
link |
00:59:43.680
because the area and time of a multiplier
link |
00:59:48.080
is n squared in the number of bits in the mantissa,
link |
00:59:50.840
but it's linear with size of the exponent.
link |
00:59:53.320
And you're connected to both efforts
link |
00:59:55.400
here both on the hardware and the software side?
link |
00:59:57.160
Yeah, and so that was a breakthrough
link |
00:59:58.880
coming from the research side and people
link |
01:00:01.440
working on optimizing network transport of weights
link |
01:00:06.000
across the network originally and trying
link |
01:00:08.240
to find ways to compress that.
link |
01:00:10.160
But then it got burned into silicon.
link |
01:00:12.120
And it's a key part of what makes TPU performance
link |
01:00:14.560
so amazing and great.
link |
01:00:17.880
Now, TPUs have many different aspects that are important.
link |
01:00:20.680
But the co design between the low level compiler bits
link |
01:00:25.080
and the software bits and the algorithms
link |
01:00:27.360
is all super important.
link |
01:00:28.680
And it's this amazing trifecta that only Google can do.
link |
01:00:32.880
Yeah, that's super exciting.
link |
01:00:34.240
So can you tell me about MLIR project, previously
link |
01:00:39.800
the secretive one?
link |
01:00:41.400
Yeah, so MLIR is a project that we
link |
01:00:43.040
announced at a compiler conference three weeks ago
link |
01:00:47.000
or something at the Compilers for Machine Learning
link |
01:00:49.280
conference.
link |
01:00:50.920
Basically, again, if you look at TensorFlow as a compiler stack,
link |
01:00:53.760
it has a number of compiler algorithms within it.
link |
01:00:56.120
It also has a number of compilers
link |
01:00:57.660
that get embedded into it.
link |
01:00:59.000
And they're made by different vendors.
link |
01:01:00.480
For example, Google has XLA, which
link |
01:01:02.840
is a great compiler system.
link |
01:01:04.680
NVIDIA has TensorRT.
link |
01:01:06.480
Intel has NGRAPH.
link |
01:01:08.640
There's a number of these different compiler systems.
link |
01:01:10.840
And they're very hardware specific.
link |
01:01:13.840
And they're trying to solve different parts of the problems.
link |
01:01:16.480
But they're all kind of similar in a sense of they
link |
01:01:19.400
want to integrate with TensorFlow.
link |
01:01:20.880
Now, TensorFlow has an optimizer.
link |
01:01:22.960
And it has these different code generation technologies
link |
01:01:25.540
built in.
link |
01:01:26.440
The idea of MLIR is to build a common infrastructure
link |
01:01:28.720
to support all these different subsystems.
link |
01:01:31.160
And initially, it's to be able to make it
link |
01:01:33.500
so that they all plug in together
link |
01:01:34.880
and they can share a lot more code and can be reusable.
link |
01:01:37.880
But over time, we hope that the industry
link |
01:01:39.680
will start collaborating and sharing code.
link |
01:01:42.480
And instead of reinventing the same things over and over again,
link |
01:01:45.320
that we can actually foster some of that working together
link |
01:01:49.280
to solve common problem energy that
link |
01:01:51.560
has been useful in the compiler field before.
link |
01:01:54.480
Beyond that, MLIR is some people have joked
link |
01:01:57.360
that it's kind of LLVM too.
link |
01:01:59.320
It learns a lot about what LLVM has been good
link |
01:02:01.840
and what LLVM has done wrong.
link |
01:02:04.360
And it's a chance to fix that.
link |
01:02:06.880
And also, there are challenges in the LLVM ecosystem as well,
link |
01:02:09.840
where LLVM is very good at the thing it was designed to do.
link |
01:02:12.760
But 20 years later, the world has changed.
link |
01:02:15.560
And people are trying to solve higher level problems.
link |
01:02:17.980
And we need some new technology.
link |
01:02:20.360
And what's the future of open source in this context?
link |
01:02:24.720
Very soon.
link |
01:02:25.760
So it is not yet open source.
link |
01:02:27.480
But it will be hopefully in the next couple months.
link |
01:02:29.320
So you still believe in the value of open source
link |
01:02:31.040
in these kinds of contexts?
link |
01:02:31.640
Oh, yeah.
link |
01:02:31.880
Absolutely.
link |
01:02:32.440
And I think that the TensorFlow community at large
link |
01:02:36.160
fully believes in open source.
link |
01:02:37.720
So I mean, there is a difference between Apple,
link |
01:02:40.120
where you were previously, and Google now,
link |
01:02:42.480
in spirit and culture.
link |
01:02:43.520
And I would say the open source in TensorFlow
link |
01:02:45.480
was a seminal moment in the history of software,
link |
01:02:48.400
because here's this large company releasing
link |
01:02:51.680
a very large code base that's open sourcing.
link |
01:02:56.200
What are your thoughts on that?
link |
01:02:58.520
Happy or not, were you to see that kind
link |
01:03:00.840
of degree of open sourcing?
link |
01:03:02.920
So between the two, I prefer the Google approach,
link |
01:03:05.360
if that's what you're saying.
link |
01:03:07.800
The Apple approach makes sense, given the historical context
link |
01:03:12.400
that Apple came from.
link |
01:03:13.400
But that's been 35 years ago.
link |
01:03:15.760
And I think that Apple is definitely adapting.
link |
01:03:18.200
And the way I look at it is that there's
link |
01:03:20.280
different kinds of concerns in the space.
link |
01:03:23.160
It is very rational for a business
link |
01:03:24.880
to care about making money.
link |
01:03:28.720
That fundamentally is what a business is about.
link |
01:03:31.640
But I think it's also incredibly realistic to say,
link |
01:03:34.880
it's not your string library that's
link |
01:03:36.360
the thing that's going to make you money.
link |
01:03:38.080
It's going to be the amazing UI product differentiating
link |
01:03:41.480
features and other things like that that you built on top
link |
01:03:43.840
of your string library.
link |
01:03:45.280
And so keeping your string library
link |
01:03:48.280
proprietary and secret and things
link |
01:03:50.360
like that is maybe not the important thing anymore.
link |
01:03:54.760
Where before, platforms were different.
link |
01:03:57.720
And even 15 years ago, things were a little bit different.
link |
01:04:01.520
But the world is changing.
link |
01:04:02.920
So Google strikes a very good balance,
link |
01:04:04.840
I think.
link |
01:04:05.340
And I think that TensorFlow being open source really
link |
01:04:09.040
changed the entire machine learning field
link |
01:04:12.000
and caused a revolution in its own right.
link |
01:04:14.080
And so I think it's amazingly forward looking
link |
01:04:17.560
because I could have imagined, and I wasn't at Google
link |
01:04:20.880
at the time, but I could imagine a different context
link |
01:04:23.160
and different world where a company says,
link |
01:04:25.520
machine learning is critical to what we're doing.
link |
01:04:27.640
We're not going to give it to other people.
link |
01:04:29.640
And so that decision is a profoundly brilliant insight
link |
01:04:35.560
that I think has really led to the world being
link |
01:04:37.480
better and better for Google as well.
link |
01:04:40.120
And has all kinds of ripple effects.
link |
01:04:42.200
I think it is really, I mean, you
link |
01:04:45.160
can't understate Google deciding how profound that
link |
01:04:48.800
is for software.
link |
01:04:49.840
It's awesome.
link |
01:04:50.880
Well, and again, I can understand the concern
link |
01:04:54.900
about if we release our machine learning software,
link |
01:04:58.440
our competitors could go faster.
link |
01:05:00.000
But on the other hand, I think that open sourcing TensorFlow
link |
01:05:02.500
has been fantastic for Google.
link |
01:05:03.960
And I'm sure that decision was very nonobvious at the time,
link |
01:05:09.120
but I think it's worked out very well.
link |
01:05:11.480
So let's try this real quick.
link |
01:05:13.240
You were at Tesla for five months
link |
01:05:15.640
as the VP of autopilot software.
link |
01:05:17.640
You led the team during the transition from H hardware
link |
01:05:20.520
one to hardware two.
link |
01:05:22.360
I have a couple of questions.
link |
01:05:23.520
So one, first of all, to me, that's
link |
01:05:26.320
one of the bravest engineering decisions undertaking really
link |
01:05:33.000
ever in the automotive industry to me, software wise,
link |
01:05:36.040
starting from scratch.
link |
01:05:37.440
It's a really brave engineering decision.
link |
01:05:39.200
So my one question there is, what was that like?
link |
01:05:42.600
What was the challenge of that?
link |
01:05:43.920
Do you mean the career decision of jumping
link |
01:05:45.720
from a comfortable good job into the unknown, or?
link |
01:05:48.800
That combined, so at the individual level,
link |
01:05:51.480
you making that decision.
link |
01:05:54.560
And then when you show up, it's a really hard engineering
link |
01:05:57.960
problem.
link |
01:05:58.760
So you could just stay, maybe slow down,
link |
01:06:03.560
say hardware one, or those kinds of decisions.
link |
01:06:06.680
Just taking it full on, let's do this from scratch.
link |
01:06:10.160
What was that like?
link |
01:06:11.080
Well, so I mean, I don't think Tesla
link |
01:06:12.640
has a culture of taking things slow and seeing how it goes.
link |
01:06:16.080
And one of the things that attracted me about Tesla
link |
01:06:18.080
is it's very much a gung ho, let's change the world,
link |
01:06:20.020
let's figure it out kind of a place.
link |
01:06:21.520
And so I have a huge amount of respect for that.
link |
01:06:25.640
Tesla has done very smart things with hardware one
link |
01:06:28.680
in particular.
link |
01:06:29.400
And the hardware one design was originally
link |
01:06:32.200
designed to be very simple automation features
link |
01:06:36.560
in the car for like traffic aware cruise control and things
link |
01:06:39.360
like that.
link |
01:06:39.840
And the fact that they were able to effectively feature creep
link |
01:06:42.920
it into lane holding and a very useful driver assistance
link |
01:06:47.720
feature is pretty astounding, particularly given
link |
01:06:50.120
the details of the hardware.
link |
01:06:52.560
Hardware two built on that in a lot of ways.
link |
01:06:54.640
And the challenge there was that they
link |
01:06:56.180
were transitioning from a third party provided vision stack
link |
01:07:00.040
to an in house built vision stack.
link |
01:07:01.720
And so for the first step, which I mostly helped with,
link |
01:07:05.680
was getting onto that new vision stack.
link |
01:07:08.480
And that was very challenging.
link |
01:07:10.800
And it was time critical for various reasons,
link |
01:07:14.000
and it was a big leap.
link |
01:07:14.960
But it was fortunate that it built
link |
01:07:16.640
on a lot of the knowledge and expertise and the team
link |
01:07:18.800
that had built hardware one's driver assistance features.
link |
01:07:22.920
So you spoke in a collected and kind way
link |
01:07:25.360
about your time at Tesla, but it was ultimately not a good fit.
link |
01:07:28.960
Elon Musk, we've talked on this podcast,
link |
01:07:31.840
several guests to the course, Elon Musk
link |
01:07:33.880
continues to do some of the most bold and innovative engineering
link |
01:07:36.880
work in the world, at times at the cost
link |
01:07:39.560
some of the members of the Tesla team.
link |
01:07:41.280
What did you learn about working in this chaotic world
link |
01:07:45.080
with Elon?
link |
01:07:46.720
Yeah, so I guess I would say that when I was at Tesla,
link |
01:07:50.560
I experienced and saw the highest degree of turnover
link |
01:07:54.440
I'd ever seen in a company, which was a bit of a shock.
link |
01:07:58.240
But one of the things I learned and I came to respect
link |
01:08:00.520
is that Elon's able to attract amazing talent because he
link |
01:08:03.760
has a very clear vision of the future,
link |
01:08:05.660
and he can get people to buy into it
link |
01:08:07.200
because they want that future to happen.
link |
01:08:09.840
And the power of vision is something
link |
01:08:11.840
that I have a tremendous amount of respect for.
link |
01:08:14.240
And I think that Elon is fairly singular
link |
01:08:17.040
in the world in terms of the things
link |
01:08:20.120
he's able to get people to believe in.
link |
01:08:22.360
And there are many people that stand in the street corner
link |
01:08:27.360
and say, ah, we're going to go to Mars, right?
link |
01:08:30.200
But then there are a few people that
link |
01:08:31.600
can get others to buy into it and believe and build the path
link |
01:08:35.200
and make it happen.
link |
01:08:36.160
And so I respect that.
link |
01:08:39.120
I don't respect all of his methods,
link |
01:08:41.880
but I have a huge amount of respect for that.
link |
01:08:45.000
You've mentioned in a few places,
link |
01:08:46.920
including in this context, working hard.
link |
01:08:50.440
What does it mean to work hard?
link |
01:08:52.000
And when you look back at your life,
link |
01:08:53.520
what were some of the most brutal periods
link |
01:08:57.080
of having to really put everything
link |
01:09:00.760
you have into something?
link |
01:09:03.360
Yeah, good question.
link |
01:09:05.040
So working hard can be defined a lot of different ways,
link |
01:09:07.440
so a lot of hours, and so that is true.
link |
01:09:12.480
The thing to me that's the hardest
link |
01:09:14.520
is both being short term focused on delivering and executing
link |
01:09:18.760
and making a thing happen while also thinking
link |
01:09:21.120
about the longer term and trying to balance that.
link |
01:09:24.400
Because if you are myopically focused on solving a task
link |
01:09:28.520
and getting that done and only think
link |
01:09:31.240
about that incremental next step,
link |
01:09:32.600
you will miss the next big hill you should jump over to.
link |
01:09:36.440
And so I've been really fortunate that I've
link |
01:09:39.600
been able to kind of oscillate between the two.
link |
01:09:42.120
And historically at Apple, for example, that
link |
01:09:45.480
was made possible because I was able to work with some really
link |
01:09:47.920
amazing people and build up teams and leadership
link |
01:09:50.360
structures and allow them to grow in their careers
link |
01:09:55.280
and take on responsibility, thereby freeing up
link |
01:09:58.280
me to be a little bit crazy and thinking about the next thing.
link |
01:10:02.960
And so it's a lot of that.
link |
01:10:04.640
But it's also about with experience,
link |
01:10:06.760
you make connections that other people don't necessarily make.
link |
01:10:10.080
And so I think that's a big part as well.
link |
01:10:12.880
But the bedrock is just a lot of hours.
link |
01:10:16.000
And that's OK with me.
link |
01:10:19.600
There's different theories on work life balance.
link |
01:10:21.480
And my theory for myself, which I do not project onto the team,
link |
01:10:25.200
but my theory for myself is that I
link |
01:10:28.520
want to love what I'm doing and work really hard.
link |
01:10:30.400
And my purpose, I feel like, and my goal is to change the world
link |
01:10:35.000
and make it a better place.
link |
01:10:36.280
And that's what I'm really motivated to do.
link |
01:10:40.000
So last question, LLVM logo is a dragon.
link |
01:10:44.760
You explain that this is because dragons have connotations
link |
01:10:47.880
of power, speed, intelligence.
link |
01:10:50.320
It can also be sleek, elegant, and modular,
link |
01:10:53.320
though you remove the modular part.
link |
01:10:56.280
What is your favorite dragon related character
link |
01:10:58.920
from fiction, video, or movies?
link |
01:11:01.440
So those are all very kind ways of explaining it.
link |
01:11:03.840
Do you want to know the real reason it's a dragon?
link |
01:11:06.200
Yeah.
link |
01:11:07.000
Is that better?
link |
01:11:07.920
So there is a seminal book on compiler design
link |
01:11:11.040
called The Dragon Book.
link |
01:11:12.520
And so this is a really old now book on compilers.
link |
01:11:16.320
And so the dragon logo for LLVM came about because at Apple,
link |
01:11:22.080
we kept talking about LLVM related technologies
link |
01:11:24.720
and there's no logo to put on a slide.
link |
01:11:26.960
And so we're like, what do we do?
link |
01:11:28.480
And somebody's like, well, what kind of logo
link |
01:11:30.480
should a compiler technology have?
link |
01:11:32.160
And I'm like, I don't know.
link |
01:11:33.360
I mean, the dragon is the best thing that we've got.
link |
01:11:37.320
And Apple somehow magically came up with the logo.
link |
01:11:41.520
And it was a great thing.
link |
01:11:42.680
And the whole community rallied around it.
link |
01:11:44.520
And then it got better as other graphic designers
link |
01:11:46.760
got involved.
link |
01:11:47.360
But that's originally where it came from.
link |
01:11:49.360
The story.
link |
01:11:50.160
Is there dragons from fiction that you
link |
01:11:51.960
connect with, that Game of Thrones, Lord of the Rings,
link |
01:11:57.240
that kind of thing?
link |
01:11:58.080
Lord of the Rings is great.
link |
01:11:59.200
I also like role playing games and things
link |
01:12:00.760
like computer role playing games.
link |
01:12:02.240
And so dragons often show up in there.
link |
01:12:04.280
But really, it comes back to the book.
link |
01:12:07.160
Oh, no, we need a thing.
link |
01:12:09.960
And hilariously, one of the funny things about LLVM
link |
01:12:13.720
is that my wife, who's amazing, runs the LLVM Foundation.
link |
01:12:19.520
And she goes to Grace Hopper and is
link |
01:12:21.080
trying to get more women involved in the.
link |
01:12:23.360
She's also a compiler engineer.
link |
01:12:24.640
So she's trying to get other women
link |
01:12:26.080
to get interested in compilers and things like this.
link |
01:12:28.020
And so she hands out the stickers.
link |
01:12:30.000
And people like the LLVM sticker because of Game of Thrones.
link |
01:12:34.320
And so sometimes culture has this helpful effect
link |
01:12:36.880
to get the next generation of compiler engineers
link |
01:12:39.960
engaged with the cause.
link |
01:12:42.400
OK, awesome.
link |
01:12:43.320
Chris, thanks so much for talking with us.
link |
01:12:44.800
It's been great talking with you.