back to index

Chris Lattner: Compilers, LLVM, Swift, TPU, and ML Accelerators | Lex Fridman Podcast #21


small model | large model

link |
00:00:00.000
The following is a conversation with Chris Latner.
link |
00:00:02.680
Currently, he's a senior director of Google working
link |
00:00:05.680
on several projects, including CPU, GPU, TPU accelerators
link |
00:00:09.560
for TensorFlow, Swift for TensorFlow,
link |
00:00:12.080
and all kinds of machine learning compiler magic
link |
00:00:14.400
going on behind the scenes.
link |
00:00:16.360
He's one of the top experts in the world
link |
00:00:18.440
on compiler technologies, which means
link |
00:00:20.640
he deeply understands the intricacies of how
link |
00:00:23.960
hardware and software come together
link |
00:00:26.280
to create efficient code.
link |
00:00:27.960
He created the LLVM compiler infrastructure project
link |
00:00:31.480
and the Clang compiler.
link |
00:00:33.400
He led major engineering efforts at Apple,
link |
00:00:36.040
including the creation of the Swift programming language.
link |
00:00:39.040
He also briefly spent time at Tesla as vice president
link |
00:00:42.600
of autopilot software during the transition from autopilot
link |
00:00:45.920
hardware one to hardware two, when Tesla essentially
link |
00:00:49.640
started from scratch to build an in house software
link |
00:00:52.640
infrastructure for autopilot.
link |
00:00:54.880
I could have easily talked to Chris for many more hours.
link |
00:00:58.040
Compiling code down across the level's abstraction
link |
00:01:01.240
is one of the most fundamental and fascinating aspects
link |
00:01:04.120
of what computers do, and he is one of the world experts
link |
00:01:07.160
in this process.
link |
00:01:08.640
It's rigorous science, and it's messy, beautiful art.
link |
00:01:12.920
This conversation is part of the Artificial Intelligence
link |
00:01:15.920
podcast.
link |
00:01:16.760
If you enjoy it, subscribe on YouTube,
link |
00:01:18.920
iTunes, or simply connect with me on Twitter
link |
00:01:21.560
at Lex Friedman, spelled F R I D. And now, here's
link |
00:01:25.760
my conversation with Chris Ladner.
link |
00:01:29.400
What was the first program you've ever written?
link |
00:01:33.240
My first program back.
link |
00:01:34.680
And when was it?
link |
00:01:35.400
I think I started as a kid, and my parents
link |
00:01:39.160
got a basic programming book.
link |
00:01:41.640
And so when I started, it was typing out programs from a book
link |
00:01:45.400
and seeing how they worked, and then typing them in wrong
link |
00:01:49.320
and trying to figure out why they were not working right,
link |
00:01:51.600
and that kind of stuff.
link |
00:01:52.960
So basic, what was the first language
link |
00:01:54.880
that you remember yourself maybe falling in love with,
link |
00:01:58.360
like really connecting with?
link |
00:01:59.960
I don't know.
link |
00:02:00.480
I mean, I feel like I've learned a lot along the way,
link |
00:02:02.480
and each of them have a different, special thing about them.
link |
00:02:06.720
So I started in basic, and then went like GW basic, which
link |
00:02:09.880
was the thing back in the DOS days,
link |
00:02:11.440
and then upgraded to Q basic, and eventually Quick basic,
link |
00:02:15.320
which are all slightly more fancy versions
link |
00:02:17.760
of Microsoft basic, made the jump to Pascal,
link |
00:02:20.920
and started doing machine language programming and assembly
link |
00:02:23.960
in Pascal, which was really cool.
link |
00:02:25.280
Turbo Pascal was amazing for its day.
link |
00:02:28.120
Eventually, gone to C, C++, and then kind of did
link |
00:02:31.600
lots of other weird things.
link |
00:02:33.440
I feel like you took the dark path, which is the,
link |
00:02:36.640
you could have gone Lisp, you could have gone a higher level
link |
00:02:41.520
sort of functional, philosophical, hippie route.
link |
00:02:44.600
Instead, you went into like the dark arts of the C.
link |
00:02:48.040
It was straight into the machine.
link |
00:02:49.720
Straight into the machine.
link |
00:02:50.680
So started with basic Pascal and then assembly,
link |
00:02:53.880
and then wrote a lot of assembly, and eventually did
link |
00:02:58.120
small talk and other things like that,
link |
00:03:00.120
but that was not the starting point.
link |
00:03:01.920
But so what is this journey to see?
link |
00:03:05.080
Is that in high school?
link |
00:03:06.360
Is that in college?
link |
00:03:07.560
That was in high school, yeah.
link |
00:03:08.800
So, and then that was really about trying
link |
00:03:13.760
to be able to do more powerful things than what Pascal
link |
00:03:16.120
could do and also to learn a different world.
link |
00:03:18.920
So he was really confusing to me with pointers
link |
00:03:20.720
and the syntax and everything, and it took a while,
link |
00:03:22.800
but Pascal's much more principled in various ways,
link |
00:03:28.720
sees more, I mean, it has its historical roots,
link |
00:03:33.360
but it's not as easy to learn.
link |
00:03:35.440
With pointers, there's this memory management thing
link |
00:03:39.840
that you have to become conscious of.
link |
00:03:41.640
Is that the first time you start to understand
link |
00:03:43.880
that there's resources that you're supposed to manage?
link |
00:03:46.480
Well, so you have that in Pascal as well,
link |
00:03:48.480
but in Pascal, the carrot instead of the star,
link |
00:03:51.920
there's some small differences like that,
link |
00:03:53.160
but it's not about pointer arithmetic.
link |
00:03:55.640
And see, you end up thinking about how things get laid
link |
00:03:58.920
out in memory a lot more.
link |
00:04:00.800
And so in Pascal, you have allocating and deallocating
link |
00:04:04.160
and owning the memory, but just the programs are simpler
link |
00:04:07.520
and you don't have to, well, for example,
link |
00:04:10.880
Pascal has a string type.
link |
00:04:12.600
And so you can think about a string
link |
00:04:14.040
instead of an array of characters
link |
00:04:15.880
which are consecutive in memory.
link |
00:04:17.680
So it's a little bit of a higher level abstraction.
link |
00:04:20.360
So let's get into it.
link |
00:04:22.760
Let's talk about LLVM, Selang, and compilers.
link |
00:04:25.600
Sure.
link |
00:04:26.520
So can you tell me first what LLVM and Selang are,
link |
00:04:32.120
and how is it that you find yourself the creator
link |
00:04:34.560
and lead developer, one of the most powerful
link |
00:04:37.720
compiler optimization systems in use today?
link |
00:04:40.080
Sure, so I guess they're different things.
link |
00:04:43.240
So let's start with what is a compiler?
link |
00:04:47.040
Is that a good place to start?
link |
00:04:48.840
What are the phases of a compiler?
link |
00:04:50.200
Where are the parts?
link |
00:04:51.040
Yeah, what is it?
link |
00:04:51.880
So what is even a compiler used for?
link |
00:04:53.400
So the way I look at this is you have a two sided problem
link |
00:04:57.560
of you have humans that need to write code.
link |
00:05:00.440
And then you have machines that need to run the program
link |
00:05:02.360
that the human wrote.
link |
00:05:03.360
And for lots of reasons, the humans don't want to be
link |
00:05:05.720
writing in binary and want to think about every piece
link |
00:05:08.320
of hardware.
link |
00:05:09.160
So at the same time that you have lots of humans,
link |
00:05:12.080
you also have lots of kinds of hardware.
link |
00:05:14.760
And so compilers are the art of allowing humans
link |
00:05:17.760
to think at a level of abstraction
link |
00:05:19.200
that they want to think about.
link |
00:05:20.880
And then get that program, get the thing that they wrote
link |
00:05:23.600
to run on a specific piece of hardware.
link |
00:05:26.040
And the interesting and exciting part of all this
link |
00:05:29.480
is that there's now lots of different kinds of hardware,
link |
00:05:31.960
chips like x86 and PowerPC and ARM and things like that,
link |
00:05:35.880
but also high performance accelerators for machine
link |
00:05:37.720
learning and other things like that are also just different
link |
00:05:39.920
kinds of hardware, GPUs, these are new kinds of hardware.
link |
00:05:42.880
And at the same time on the programming side of it,
link |
00:05:45.600
you have basic, you have C, you have JavaScript,
link |
00:05:48.640
you have Python, you have Swift, you have like lots
link |
00:05:51.480
of other languages that are all trying to talk to the human
link |
00:05:54.440
in a different way to make them more expressive
link |
00:05:57.040
and capable and powerful.
link |
00:05:58.320
And so compilers are the thing that goes from one
link |
00:06:02.080
to the other now.
link |
00:06:03.440
And to end from the very beginning to the very end.
link |
00:06:05.200
And to end, and so you go from what the human wrote
link |
00:06:08.120
and programming languages end up being about expressing intent,
link |
00:06:12.600
not just for the compiler and the hardware,
link |
00:06:15.960
but the programming language's job is really to capture
link |
00:06:20.320
an expression of what the programmer wanted
link |
00:06:22.640
that then can be maintained and adapted
link |
00:06:25.080
and evolved by other humans, as well as by the,
link |
00:06:28.240
interpreted by the compiler.
link |
00:06:29.680
So when you look at this problem,
link |
00:06:31.520
you have on the one hand humans, which are complicated,
link |
00:06:34.160
you have hardware, which is complicated.
link |
00:06:36.720
And so compilers typically work in multiple phases.
link |
00:06:39.880
And so the software engineering challenge
link |
00:06:42.720
that you have here is try to get maximum reuse
link |
00:06:44.960
out of the amount of code that you write
link |
00:06:47.080
because these compilers are very complicated.
link |
00:06:49.720
And so the way it typically works out is that
link |
00:06:51.840
you have something called a front end or a parser
link |
00:06:54.440
that is language specific.
link |
00:06:56.600
And so you'll have a C parser, that's what Clang is,
link |
00:07:00.360
or C++ or JavaScript or Python or whatever,
link |
00:07:03.440
that's the front end.
link |
00:07:04.960
Then you'll have a middle part,
link |
00:07:07.080
which is often the optimizer.
link |
00:07:09.000
And then you'll have a late part,
link |
00:07:11.120
which is hardware specific.
link |
00:07:13.320
And so compilers end up, there's many different layers often,
link |
00:07:16.680
but these three big groups are very common in compilers.
link |
00:07:20.880
And what LLVM is trying to do is trying to standardize
link |
00:07:23.440
that middle and last part.
link |
00:07:25.360
And so one of the cool things about LLVM
link |
00:07:27.880
is that there are a lot of different languages
link |
00:07:29.760
that compile through to it.
link |
00:07:31.080
And so things like Swift, but also Julia, Rust,
link |
00:07:36.520
Clang for C, C++, Subjective C,
link |
00:07:39.120
like these are all very different languages
link |
00:07:40.920
and they can all use the same optimization infrastructure,
link |
00:07:43.800
which gets better performance,
link |
00:07:45.400
and the same code generation infrastructure
link |
00:07:47.240
for hardware support.
link |
00:07:48.800
And so LLVM is really that layer that is common,
link |
00:07:52.240
that all these different specific compilers can use.
link |
00:07:55.560
And is it a standard, like a specification,
link |
00:07:59.280
or is it literally an implementation?
link |
00:08:01.160
It's an implementation.
link |
00:08:02.120
And so it's, I think there's a couple of different ways
link |
00:08:05.880
of looking at it, right?
link |
00:08:06.720
Because it depends on which angle you're looking at it from.
link |
00:08:09.680
LLVM ends up being a bunch of code, okay?
link |
00:08:12.600
So it's a bunch of code that people reuse
link |
00:08:14.440
and they build compilers with.
link |
00:08:16.520
We call it a compiler infrastructure
link |
00:08:18.040
because it's kind of the underlying platform
link |
00:08:20.000
that you build a concrete compiler on top of.
link |
00:08:22.520
But it's also a community.
link |
00:08:23.680
And the LLVM community is hundreds of people
link |
00:08:26.800
that all collaborate.
link |
00:08:27.920
And one of the most fascinating things about LLVM
link |
00:08:30.560
over the course of time is that we've managed somehow
link |
00:08:34.280
to successfully get harsh competitors
link |
00:08:37.080
in the commercial space to collaborate
link |
00:08:39.080
on shared infrastructure.
link |
00:08:41.120
And so you have Google and Apple.
link |
00:08:43.880
You have AMD and Intel.
link |
00:08:45.880
You have NVIDIA and AMD on the graphics side.
link |
00:08:48.880
You have Cray and everybody else doing these things.
link |
00:08:52.640
And like all these companies are collaborating together
link |
00:08:55.400
to make that shared infrastructure
link |
00:08:57.480
really, really great.
link |
00:08:58.520
And they do this not out of the goodness of their heart
link |
00:09:01.400
but they do it because it's in their commercial interest
link |
00:09:03.440
of having really great infrastructure
link |
00:09:05.160
that they can build on top of.
link |
00:09:06.800
And facing the reality that it's so expensive
link |
00:09:09.120
that no one company, even the big companies,
link |
00:09:11.200
no one company really wants to implement it all themselves.
link |
00:09:14.600
Expensive or difficult?
link |
00:09:16.120
Both.
link |
00:09:16.960
That's a great point because it's also about the skill sets.
link |
00:09:20.600
And these, the skill sets are very hard to find.
link |
00:09:25.600
How big is the LLVM?
link |
00:09:27.960
It always seems like with open source projects,
link |
00:09:30.400
the kind, and LLVM is open source?
link |
00:09:33.480
Yes, it's open source.
link |
00:09:34.440
It's about, it's 19 years old now.
link |
00:09:36.320
So it's fairly old.
link |
00:09:38.640
It seems like the magic often happens
link |
00:09:40.960
within a very small circle of people.
link |
00:09:43.040
Yes.
link |
00:09:43.880
At least at early birth and whatever.
link |
00:09:46.080
Yes.
link |
00:09:46.920
So the LLVM came from a university project.
link |
00:09:49.640
And so I was at the University of Illinois.
link |
00:09:51.640
And there it was myself, my advisor,
link |
00:09:53.880
and then a team of two or three research students
link |
00:09:57.480
in the research group.
link |
00:09:58.360
And we built many of the core pieces initially.
link |
00:10:02.080
I then graduated and went to Apple.
link |
00:10:03.720
And then Apple brought it to the products,
link |
00:10:06.480
first in the OpenGL graphics stack,
link |
00:10:09.320
but eventually to the C compiler realm
link |
00:10:11.600
and eventually built Clang
link |
00:10:12.760
and eventually built Swift and these things.
link |
00:10:14.640
Along the way, building a team of people
link |
00:10:16.360
that are really amazing compiler engineers
link |
00:10:18.600
that helped build a lot of that.
link |
00:10:20.120
And so as it was gaining momentum
link |
00:10:21.840
and as Apple was using it, being open source and public
link |
00:10:24.800
and encouraging contribution, many others,
link |
00:10:27.040
for example, at Google, came in and started contributing.
link |
00:10:30.400
And in some cases, Google effectively owns Clang now
link |
00:10:33.680
because it cares so much about C++
link |
00:10:35.520
and the evolution of that ecosystem.
link |
00:10:37.280
And so it's investing a lot in the C++ world
link |
00:10:41.400
and the tooling and things like that.
link |
00:10:42.960
And so likewise, NVIDIA cares a lot about CUDA.
link |
00:10:47.840
And so CUDA uses Clang and uses LVM for graphics and GPGPU.
link |
00:10:54.960
And so when you first started as a master's project, I guess,
link |
00:10:59.880
did you think it was gonna go as far as it went?
link |
00:11:02.920
Were you crazy ambitious about it?
link |
00:11:06.280
No.
link |
00:11:07.120
It seems like a really difficult undertaking, a brave one.
link |
00:11:09.760
Yeah, no, no, it was nothing like that.
link |
00:11:11.280
So I mean, my goal when I went to University of Illinois
link |
00:11:13.640
was to get in and out with the non thesis masters
link |
00:11:16.120
in a year and get back to work.
link |
00:11:18.680
So I was not planning to stay for five years
link |
00:11:22.160
and build this massive infrastructure.
link |
00:11:24.440
I got nerd sniped into staying.
link |
00:11:27.400
And a lot of it was because LVM was fun
link |
00:11:29.480
and I was building cool stuff
link |
00:11:30.920
and learning really interesting things
link |
00:11:33.400
and facing both software engineering challenges
link |
00:11:36.880
but also learning how to work in a team
link |
00:11:38.520
and things like that.
link |
00:11:40.120
I had worked at many companies as interns before that,
link |
00:11:43.600
but it was really a different thing
link |
00:11:45.840
to have a team of people that were working together
link |
00:11:48.120
and trying to collaborate in version control
link |
00:11:50.480
and it was just a little bit different.
link |
00:11:52.400
Like I said, I just talked to Don Knuth
link |
00:11:54.080
and he believes that 2% of the world population
link |
00:11:56.840
have something weird with their brain, that they're geeks,
link |
00:11:59.600
they understand computers, they're connected with computers.
link |
00:12:02.560
He put it at exactly 2%.
link |
00:12:04.360
Okay, so...
link |
00:12:05.560
Is this a specific act?
link |
00:12:06.560
It's very specific.
link |
00:12:08.760
Well, he says, I can't prove it,
link |
00:12:10.200
but it's very empirically there.
link |
00:12:13.040
Is there something that attracts you
link |
00:12:14.480
to the idea of optimizing code?
link |
00:12:16.920
And he seems like that's one of the biggest,
link |
00:12:19.120
coolest things about LVM.
link |
00:12:20.920
Yeah, that's one of the major things it does.
link |
00:12:22.480
So I got into that because of a person, actually.
link |
00:12:26.440
So when I was in my undergraduate,
link |
00:12:28.200
I had an advisor or a professor named Steve Vegdahl
link |
00:12:32.040
and I went to this little tiny private school.
link |
00:12:35.760
There were like seven or nine people
link |
00:12:38.280
in my computer science department,
link |
00:12:40.320
students in my class.
link |
00:12:43.080
So it was a very tiny, very small school.
link |
00:12:47.440
It was kind of a work on the side of the math department
link |
00:12:49.960
kind of a thing at the time.
link |
00:12:51.240
I think it's evolved a lot in the many years since then,
link |
00:12:53.800
but Steve Vegdahl was a compiler guy
link |
00:12:58.280
and he was super passionate
link |
00:12:59.600
and his passion rubbed off on me
link |
00:13:02.720
and one of the things I like about compilers
link |
00:13:04.440
is that they're large, complicated software pieces.
link |
00:13:09.120
And so one of the culminating classes
link |
00:13:12.920
that many computer science departments
link |
00:13:14.520
at least at the time did was to say
link |
00:13:16.680
that you would take algorithms and data structures
link |
00:13:18.400
and all these core classes,
link |
00:13:19.480
but then the compilers class
link |
00:13:20.720
was one of the last classes you take
link |
00:13:22.160
because it pulls everything together
link |
00:13:24.360
and then you work on one piece of code
link |
00:13:27.000
over the entire semester.
link |
00:13:28.680
And so you keep building on your own work,
link |
00:13:32.080
which is really interesting and it's also very challenging
link |
00:13:34.800
because in many classes, if you don't get a project done,
link |
00:13:37.520
you just forget about it and move on to the next one
link |
00:13:39.320
and get your B or whatever it is,
link |
00:13:41.320
but here you have to live with the decisions you make
link |
00:13:43.880
and continue to reinvest in it.
link |
00:13:45.280
And I really like that.
link |
00:13:46.880
And so I did a extra study project with him
link |
00:13:51.080
the following semester and he was just really great
link |
00:13:53.960
and he was also a great mentor in a lot of ways.
link |
00:13:56.920
And so from him and from his advice,
link |
00:13:59.560
he encouraged me to go to graduate school.
link |
00:14:01.520
I wasn't super excited about going to grad school.
link |
00:14:03.200
I wanted the master's degree,
link |
00:14:05.240
but I didn't want to be an academic.
link |
00:14:09.000
But like I said, I kind of got tricked into saying
link |
00:14:11.160
I was having a lot of fun and I definitely do not regret it.
link |
00:14:14.560
Well, the aspects of compilers
link |
00:14:15.840
were the things you connected with.
link |
00:14:17.960
So LVM, there's also the other part
link |
00:14:22.120
that's just really interesting
link |
00:14:23.440
if you're interested in languages is parsing and just analyzing
link |
00:14:27.640
like, yeah, analyzing the language,
link |
00:14:29.640
breaking it down, parsing and so on.
link |
00:14:31.240
Was that interesting to you
link |
00:14:32.280
or were you more interested in optimization?
link |
00:14:34.080
For me, it was more, so I'm not really a math person.
link |
00:14:37.400
I can do math, I understand some bits of it
link |
00:14:39.600
when I get into it,
link |
00:14:41.600
but math is never the thing that attracted me.
link |
00:14:43.960
And so a lot of the parser part of the compiler
link |
00:14:46.160
has a lot of good formal theories that Dawn, for example,
link |
00:14:48.960
knows quite well.
link |
00:14:50.440
Still waiting for his book on that.
link |
00:14:51.920
But I just like building a thing
link |
00:14:56.080
and seeing what it could do and exploring
link |
00:14:59.200
and getting it to do more things
link |
00:15:00.800
and then setting new goals and reaching for them.
link |
00:15:02.880
And in the case of LVM, when I started working on that,
link |
00:15:08.880
my research advisor that I was working for was a compiler guy.
link |
00:15:13.360
And so he and I specifically found each other
link |
00:15:15.600
because we both interested in compilers
link |
00:15:16.920
and so I started working with them and taking his class.
link |
00:15:19.480
And a lot of LVM initially was it's fun
link |
00:15:21.800
implementing all the standard algorithms
link |
00:15:23.560
and all the things that people had been talking about
link |
00:15:26.360
and were well known and they were in the curricula
link |
00:15:28.920
for advanced studies in compilers.
link |
00:15:31.320
And so just being able to build that was really fun
link |
00:15:34.560
and I was learning a lot
link |
00:15:36.160
by instead of reading about it, just building.
link |
00:15:38.640
And so I enjoyed that.
link |
00:15:40.200
So you said compilers are these complicated systems.
link |
00:15:42.800
Can you even just with language try to describe
link |
00:15:48.240
how you turn a C++ program into code?
link |
00:15:52.240
Like what are the hard parts?
link |
00:15:53.480
Why is it so hard?
link |
00:15:54.640
So I'll give you examples of the hard parts along the way.
link |
00:15:56.840
So C++ is a very complicated programming language.
link |
00:16:01.040
It's something like 1,400 pages in the spec.
link |
00:16:03.480
So C++ by itself is crazy complicated.
link |
00:16:06.120
Can we just, sorry, pause.
link |
00:16:07.160
What makes the language complicated
link |
00:16:08.720
in terms of what's syntactically?
link |
00:16:11.520
Like, so it's what they call syntax.
link |
00:16:14.320
So the actual how the characters are arranged.
link |
00:16:16.280
Yes, it's also semantics, how it behaves.
link |
00:16:20.080
It's also in the case of C++.
link |
00:16:21.720
There's a huge amount of history.
link |
00:16:23.400
C++ build on top of C.
link |
00:16:25.560
You play that forward and then a bunch of suboptimal
link |
00:16:28.720
in some cases decisions were made
link |
00:16:30.360
and they compound and then more and more and more things
link |
00:16:33.400
keep getting added to C++
link |
00:16:35.080
and it will probably never stop.
link |
00:16:37.040
But the language is very complicated from that perspective.
link |
00:16:39.440
And so the interactions between subsystems
link |
00:16:41.200
is very complicated.
link |
00:16:42.360
There's just a lot there.
link |
00:16:43.560
And when you talk about the front end,
link |
00:16:45.640
one of the major challenges which playing as a project,
link |
00:16:48.560
the C++ compiler that I built, I and many people built.
link |
00:16:53.320
One of the challenges we took on was we looked at GCC.
link |
00:16:57.560
I think GCC at the time was like a really good
link |
00:17:01.120
industry standardized compiler that had really consolidated
link |
00:17:05.320
a lot of the other compilers in the world
link |
00:17:06.760
and was a standard.
link |
00:17:08.360
But it wasn't really great for research.
link |
00:17:10.640
The design was very difficult to work with
link |
00:17:12.600
and it was full of global variables and other things
link |
00:17:16.640
that made it very difficult to reuse
link |
00:17:18.120
in ways that it wasn't originally designed for.
link |
00:17:20.400
And so with Clang, one of the things that we wanted to do
link |
00:17:22.560
is push forward on better user interface.
link |
00:17:25.520
So make error messages that are just better than GCCs.
link |
00:17:28.160
And that's actually hard because you have to do
link |
00:17:29.920
a lot of bookkeeping in an efficient way
link |
00:17:32.800
to be able to do that.
link |
00:17:33.640
We want to make compile time better.
link |
00:17:35.160
And so compile time is about making it efficient,
link |
00:17:37.520
which is also really hard when you're keeping
link |
00:17:38.920
track of extra information.
link |
00:17:40.520
We wanted to make new tools available.
link |
00:17:43.400
So refactoring tools and other analysis tools
link |
00:17:46.400
that the GCC never supported,
link |
00:17:48.400
also leveraging the extra information we kept,
link |
00:17:52.200
but enabling those new classes of tools
link |
00:17:54.080
that then get built into IDEs.
link |
00:17:55.960
And so that's been one of the areas
link |
00:17:58.560
that Clang has really helped push the world forward in
link |
00:18:01.320
is in the tooling for C and C++ and things like that.
link |
00:18:05.080
But C++ in the front end piece is complicated
link |
00:18:07.760
and you have to build syntax trees
link |
00:18:09.040
and you have to check every rule in the spec
link |
00:18:11.360
and you have to turn that back into an error message
link |
00:18:14.000
to the human that the human can understand
link |
00:18:16.040
when they do something wrong.
link |
00:18:17.840
But then you start doing what's called lowering.
link |
00:18:20.760
So going from C++ in the way that it represents code
link |
00:18:23.440
down to the machine.
link |
00:18:24.960
And when you do that,
link |
00:18:25.800
there's many different phases you go through.
link |
00:18:29.640
Often there are, I think LVM has something like 150
link |
00:18:33.040
different, what are called passes in the compiler
link |
00:18:36.240
that the code passes through
link |
00:18:38.760
and these get organized in very complicated ways,
link |
00:18:41.880
which affect the generated code and the performance
link |
00:18:44.360
and compile time and many of the things.
link |
00:18:46.000
What are they passing through?
link |
00:18:47.320
So after you do the Clang parsing,
link |
00:18:51.840
what's the graph?
link |
00:18:53.960
What does it look like?
link |
00:18:54.800
What's the data structure here?
link |
00:18:55.960
Yeah, so in the parser, it's usually a tree
link |
00:18:59.040
and it's called an abstract syntax tree.
link |
00:19:01.040
And so the idea is you have a node for the plus
link |
00:19:04.560
that the human wrote in their code
link |
00:19:06.800
or the function call, you'll have a node for call
link |
00:19:09.000
with the function that they call in the arguments they pass.
link |
00:19:11.840
Things like that.
link |
00:19:14.440
This then gets lowered into what's called
link |
00:19:16.840
an intermediate representation
link |
00:19:18.600
and intermediate representations are like LVM has one.
link |
00:19:22.080
And there it's a, it's what's called a control flow graph.
link |
00:19:26.920
And so you represent each operation in the program
link |
00:19:31.200
as a very simple, like this is gonna add two numbers.
link |
00:19:34.480
This is gonna multiply two things.
link |
00:19:35.880
This maybe we'll do a call,
link |
00:19:37.480
but then they get put in what are called blocks.
link |
00:19:40.280
And so you get blocks of these straight line operations
link |
00:19:43.600
where instead of being nested like in a tree,
link |
00:19:45.320
it's straight line operations.
link |
00:19:46.920
And so there's a sequence
link |
00:19:47.920
in ordering to these operations.
link |
00:19:49.760
So within the block or outside the block?
link |
00:19:51.840
That's within the block.
link |
00:19:53.240
And so it's a straight line sequence of operations
link |
00:19:55.000
within the block.
link |
00:19:55.840
And then you have branches,
link |
00:19:57.520
like conditional branches between blocks.
link |
00:20:00.160
And so when you write a loop, for example,
link |
00:20:04.120
in a syntax tree, you would have a four node
link |
00:20:07.080
like for a four statement in a C like language,
link |
00:20:09.080
you'd have a four node.
link |
00:20:10.840
And you have a pointer to the expression
link |
00:20:12.200
for the initializer, a pointer to the expression
link |
00:20:14.120
for the increment, a pointer to the expression
link |
00:20:15.840
for the comparison, a pointer to the body.
link |
00:20:18.720
Okay, and these are all nested underneath it.
link |
00:20:21.040
In a control flow graph, you get a block
link |
00:20:22.880
for the code that runs before the loop.
link |
00:20:25.960
So the initializer code.
link |
00:20:27.600
Then you have a block for the body of the loop.
link |
00:20:30.280
And so the body of the loop code goes in there,
link |
00:20:33.760
but also the increment and other things like that.
link |
00:20:35.520
And then you have a branch that goes back to the top
link |
00:20:37.800
and a comparison and a branch that goes out.
link |
00:20:39.840
And so it's more of a assembly level kind of representation.
link |
00:20:44.000
But the nice thing about this level of representation
link |
00:20:46.040
is it's much more language independent.
link |
00:20:48.680
And so there's lots of different kinds of languages
link |
00:20:51.880
with different kinds of, you know,
link |
00:20:54.520
JavaScript has a lot of different ideas
link |
00:20:56.600
of what is false, for example,
link |
00:20:58.160
and all that can stay in the front end,
link |
00:21:00.760
but then that middle part can be shared across all of those.
link |
00:21:04.200
How close is that intermediate representation
link |
00:21:07.520
to new networks, for example?
link |
00:21:10.280
Are they, because everything you describe as a kind of
link |
00:21:14.320
a close of a neural network graph,
link |
00:21:16.080
are they neighbors or what?
link |
00:21:18.920
They're quite different in details,
link |
00:21:20.960
but they're very similar in idea.
link |
00:21:22.480
So one of the things that normal networks do
link |
00:21:24.000
is they learn representations for data
link |
00:21:26.880
at different levels of abstraction, right?
link |
00:21:29.120
And then they transform those through layers, right?
link |
00:21:33.920
So the compiler does very similar things,
link |
00:21:35.680
but one of the things the compiler does
link |
00:21:37.120
is it has relatively few different representations.
link |
00:21:40.640
Where a neural network, often as you get deeper,
link |
00:21:42.480
for example, you get many different representations
link |
00:21:44.800
and each, you know, layer or set of ops
link |
00:21:47.400
is transforming between these different representations.
link |
00:21:50.200
In a compiler, often you get one representation
link |
00:21:53.080
and they do many transformations to it.
link |
00:21:55.240
And these transformations are often applied iteratively.
link |
00:21:59.520
And for programmers, they're familiar types of things.
link |
00:22:02.920
For example, trying to find expressions inside of a loop
link |
00:22:06.160
and pulling them out of a loop.
link |
00:22:07.320
So if they execute fairer times
link |
00:22:08.560
or find redundant computation
link |
00:22:10.760
or find constant folding or other simplifications
link |
00:22:15.360
turning, you know, two times X into X shift left by one
link |
00:22:19.040
and things like this are all the examples
link |
00:22:21.960
of the things that happen.
link |
00:22:23.360
But compilers end up getting a lot of theorem proving
link |
00:22:26.200
and other kinds of algorithms
link |
00:22:27.640
that try to find higher level properties of the program
link |
00:22:29.960
that then can be used by the optimizer.
link |
00:22:32.320
Cool, so what's like the biggest bang for the buck
link |
00:22:35.920
with optimization?
link |
00:22:37.680
What's a day?
link |
00:22:38.720
Yeah.
link |
00:22:39.560
Well, no, not even today.
link |
00:22:40.920
At the very beginning, the 80s, I don't know.
link |
00:22:42.800
Yeah, so for the 80s,
link |
00:22:43.960
a lot of it was things like register allocation.
link |
00:22:46.440
So the idea of in a modern, like a microprocessor,
link |
00:22:51.000
what you'll end up having is you'll end up having memory,
link |
00:22:52.760
which is relatively slow.
link |
00:22:54.320
And then you have registers relatively fast,
link |
00:22:57.080
but registers, you don't have very many of them.
link |
00:22:59.920
Okay, and so when you're writing a bunch of code,
link |
00:23:02.600
you're just saying like, compute this,
link |
00:23:04.200
put in temporary variable, compute this,
link |
00:23:05.520
compute this, put in temporary variable,
link |
00:23:07.800
I have a loop, I have some other stuff going on.
link |
00:23:09.760
Well, now you're running on an x86,
link |
00:23:11.680
like a desktop PC or something.
link |
00:23:13.920
Well, it only has, in some cases,
link |
00:23:16.160
some modes, eight registers, right?
link |
00:23:18.720
And so now the compiler has to choose
link |
00:23:20.800
what values get put in what registers,
link |
00:23:22.800
at what points in the program.
link |
00:23:24.840
And this is actually a really big deal.
link |
00:23:26.480
So if you think about, you have a loop,
link |
00:23:28.560
an inner loop that executes millions of times maybe.
link |
00:23:31.640
If you're doing loads and stores inside that loop,
link |
00:23:33.600
then it's gonna be really slow.
link |
00:23:34.920
But if you can somehow fit all the values
link |
00:23:37.080
inside that loop in registers, now it's really fast.
link |
00:23:40.200
And so getting that right requires a lot of work,
link |
00:23:43.400
because there's many different ways to do that.
link |
00:23:44.960
And often what the compiler ends up doing
link |
00:23:47.000
is it ends up thinking about things
link |
00:23:48.880
in a different representation than what the human wrote.
link |
00:23:51.920
All right, you wrote into x.
link |
00:23:53.320
Well, the compiler thinks about that as four different values,
link |
00:23:56.800
each which have different lifetimes
link |
00:23:58.360
across the function that it's in.
link |
00:24:00.400
And each of those could be put in a register
link |
00:24:02.640
or memory or different memory, or maybe in some parts
link |
00:24:05.840
of the code, recompute it instead of stored and reloaded.
link |
00:24:08.760
And there are many of these different kinds
link |
00:24:10.000
of techniques that can be used.
link |
00:24:11.440
So it's adding almost like a time dimension
link |
00:24:14.840
to it's trying to optimize across time.
link |
00:24:18.320
So it's considering when you're programming,
link |
00:24:20.360
you're not thinking in that way.
link |
00:24:21.920
Yeah, absolutely.
link |
00:24:23.200
And so the risk era made things, so risk chips, RISC,
link |
00:24:28.200
RISC, the risk chips as opposed to SISC chips,
link |
00:24:33.680
the risk chips made things more complicated
link |
00:24:36.000
for the compiler because what they ended up doing
link |
00:24:39.720
is ending up adding pipelines to the processor
link |
00:24:42.360
where the processor can do more than one thing at a time.
link |
00:24:45.000
But this means that the order of operations matters a lot.
link |
00:24:47.600
And so one of the classical compiler techniques
link |
00:24:49.720
that you use is called scheduling.
link |
00:24:52.000
And so moving the instructions around
link |
00:24:54.200
so that the processor can like keep its pipelines full
link |
00:24:57.400
instead of stalling and getting blocked.
link |
00:24:59.600
And so there's a lot of things like that
link |
00:25:00.960
that are kind of bread and butter or compiler techniques
link |
00:25:03.600
that have been studied a lot over the course of decades now.
link |
00:25:06.240
But the engineering side of making them real
link |
00:25:08.520
is also still quite hard.
link |
00:25:10.680
And you talk about machine learning,
link |
00:25:12.400
this is a huge opportunity for machine learning
link |
00:25:14.400
because many of these algorithms
link |
00:25:16.520
are full of these like hokey hand rolled heuristics
link |
00:25:19.120
which work well on specific benchmarks
link |
00:25:20.880
that don't generalize and full of magic numbers.
link |
00:25:23.920
And I hear there's some techniques
link |
00:25:26.520
that are good at handling that.
link |
00:25:28.000
So what would be the,
link |
00:25:29.880
if you were to apply machine learning to this,
link |
00:25:33.040
what's the thing you try to optimize?
link |
00:25:34.720
Is it ultimately the running time?
link |
00:25:38.080
Yeah, you can pick your metric
link |
00:25:39.960
and there's running time, there's memory use,
link |
00:25:42.240
there's lots of different things that you can optimize
link |
00:25:44.760
for code size is another one that some people care about
link |
00:25:47.200
in the embedded space.
link |
00:25:48.800
Is this like the thinking into the future
link |
00:25:51.680
or has somebody actually been crazy enough to try
link |
00:25:55.600
to have machine learning based parameter tuning
link |
00:25:59.080
for optimization of compilers?
link |
00:26:01.040
So this is something that is, I would say research right now.
link |
00:26:04.840
There are a lot of research systems
link |
00:26:06.800
that have been applying search in various forms
link |
00:26:09.080
and using reinforcement learning as one form,
link |
00:26:11.440
but also brute force search has been tried for quite a while.
link |
00:26:14.400
And usually these are in small problem spaces.
link |
00:26:18.160
So find the optimal way to code generate
link |
00:26:21.480
a matrix multiply for a GPU, right?
link |
00:26:23.680
Something like that where you say,
link |
00:26:25.480
there there's a lot of design space
link |
00:26:28.080
of do you unroll loops a lot?
link |
00:26:29.920
Do you execute multiple things in parallel?
link |
00:26:32.600
And there's many different confounding factors here
link |
00:26:35.320
because graphics cards have different numbers of threads
link |
00:26:38.120
and registers and execution ports and memory bandwidth
link |
00:26:41.040
and many different constraints to interact
link |
00:26:42.760
in nonlinear ways.
link |
00:26:44.280
And so search is very powerful for that
link |
00:26:46.480
and it gets used in certain ways,
link |
00:26:49.840
but it's not very structured.
link |
00:26:51.240
This is something that we need,
link |
00:26:52.640
we as an industry need to fix.
link |
00:26:54.520
So you said 80s, but like,
link |
00:26:56.240
so have there been like big jumps in improvement
link |
00:26:59.960
and optimization?
link |
00:27:01.280
Yeah.
link |
00:27:02.360
Yeah, since then, what's the coolest thing about it?
link |
00:27:05.320
It's largely been driven by hardware.
link |
00:27:07.120
So hardware and software.
link |
00:27:09.880
So in the mid 90s, Java totally changed the world, right?
link |
00:27:13.880
And I'm still amazed by how much change was introduced
link |
00:27:17.520
by Java in a good way or in a good way.
link |
00:27:19.320
So like reflecting back,
link |
00:27:20.600
Java introduced things like all at once introduced things
link |
00:27:23.800
like JIT compilation.
link |
00:27:25.680
None of these were novel,
link |
00:27:26.920
but it pulled it together and made it mainstream
link |
00:27:28.640
and made people invest in it.
link |
00:27:30.600
JIT compilation, garbage collection,
link |
00:27:32.680
portable code, safe code, like memory safe code,
link |
00:27:37.680
like a very dynamic dispatch execution model.
link |
00:27:41.480
Like many of these things,
link |
00:27:42.680
which had been done in research systems
link |
00:27:44.120
and had been done in small ways in various places,
link |
00:27:46.960
really came to the forefront
link |
00:27:48.040
and really changed how things worked.
link |
00:27:49.840
And therefore changed the way people thought
link |
00:27:52.040
about the problem.
link |
00:27:53.120
JavaScript was another major world change
link |
00:27:56.360
based on the way it works.
link |
00:27:59.320
But also on the hardware side of things,
link |
00:28:02.240
multi core and vector instructions
link |
00:28:05.200
really change the problem space
link |
00:28:07.520
and are very, they don't remove any of the problems
link |
00:28:10.800
that compilers faced in the past,
link |
00:28:12.360
but they add new kinds of problems
link |
00:28:14.560
of how do you find enough work
link |
00:28:16.400
to keep a four wide vector busy, right?
link |
00:28:20.040
Or if you're doing a matrix multiplication,
link |
00:28:22.640
how do you do different columns out of that matrix
link |
00:28:25.360
in at the same time?
link |
00:28:26.680
And how do you maximum utilize the arithmetic compute
link |
00:28:30.160
that one core has?
link |
00:28:31.440
And then how do you take it to multiple cores?
link |
00:28:33.480
How did the whole virtual machine thing
link |
00:28:35.040
change the compilation pipeline?
link |
00:28:37.960
Yeah, so what the Java virtual machine does
link |
00:28:40.440
is it splits, just like I was talking about before,
link |
00:28:44.160
where you have a front end that parses the code
link |
00:28:46.280
and then you have an intermediate representation
link |
00:28:47.960
that gets transformed.
link |
00:28:49.400
What Java did was they said,
link |
00:28:50.960
we will parse the code and then compile
link |
00:28:52.720
to what's known as Java bytecode.
link |
00:28:55.480
And that bytecode is now a portable code representation
link |
00:28:58.560
that is industry standard and locked down and can't change.
link |
00:29:02.400
And then the back part of the compiler
link |
00:29:05.040
that does optimization and code generation
link |
00:29:07.280
can now be built by different vendors.
link |
00:29:09.440
Okay, and Java bytecode can be shipped around
link |
00:29:12.080
across the wire, it's memory safe and relatively trusted.
link |
00:29:16.840
And because of that it can run in the browser.
link |
00:29:18.680
And that's why it runs in the browser, right?
link |
00:29:20.480
And so that way you can be in, you know,
link |
00:29:22.960
again, back in the day, you would write a Java applet
link |
00:29:25.000
and you'd use it as a web developer,
link |
00:29:27.720
you'd build this mini app that would run on a web page.
link |
00:29:30.840
Well, a user of that is running a web browser
link |
00:29:33.600
on their computer, you download that Java bytecode,
link |
00:29:36.160
which can be trusted, and then you do
link |
00:29:39.280
all the compiler stuff on your machine
link |
00:29:41.040
so that you know that you trust that.
link |
00:29:42.400
Is that a good idea or a bad idea?
link |
00:29:44.080
It's a great idea, I mean,
link |
00:29:44.920
it's a great idea for certain problems.
link |
00:29:46.200
And I'm very much a believer
link |
00:29:48.200
that the technology is itself neither good nor bad,
link |
00:29:50.480
it's how you apply it.
link |
00:29:52.920
You know, this would be a very, very bad thing
link |
00:29:54.600
for very low levels of the software stack,
link |
00:29:56.960
but in terms of solving some of these software portability
link |
00:30:00.280
and transparency or portability problems,
link |
00:30:02.760
I think it's been really good.
link |
00:30:04.200
Now Java ultimately didn't win out on the desktop
link |
00:30:06.560
and like there are good reasons for that,
link |
00:30:09.400
but it's been very successful on servers and in many places,
link |
00:30:13.200
it's been a very successful thing over decades.
link |
00:30:16.280
So what has been LLVM's and Selang's improvements
link |
00:30:24.480
and optimization that throughout its history,
link |
00:30:28.720
what are some moments we had set back
link |
00:30:31.080
and really proud of what's been accomplished?
link |
00:30:33.280
Yeah, I think that the interesting thing about LLVM
link |
00:30:36.200
is not the innovations in compiler research,
link |
00:30:40.120
it has very good implementations
link |
00:30:41.880
of very important algorithms, no doubt.
link |
00:30:43.880
And a lot of really smart people have worked on it,
link |
00:30:48.280
but I think that the thing that's most profound about LLVM
link |
00:30:50.560
is that through standardization,
link |
00:30:52.600
it made things possible that otherwise wouldn't have happened.
link |
00:30:55.720
Okay.
link |
00:30:56.560
And so interesting things that have happened with LLVM,
link |
00:30:59.120
for example, Sony has picked up LLVM
link |
00:31:01.280
and used it to do all the graphics compilation
link |
00:31:03.920
in their movie production pipeline.
link |
00:31:06.080
And so now they're able to have better special effects
link |
00:31:07.920
because of LLVM.
link |
00:31:09.680
That's kind of cool.
link |
00:31:11.200
That's not what it was designed for, right?
link |
00:31:13.000
But that's the sign of good infrastructure
link |
00:31:15.480
when it can be used in ways it was never designed for
link |
00:31:18.800
because it has good layering and software engineering
link |
00:31:20.960
and it's composable and things like that.
link |
00:31:23.440
Which is where, as you said, it differs from GCC.
link |
00:31:26.120
Yes, GCC is also great in various ways,
link |
00:31:28.240
but it's not as good as infrastructure technology.
link |
00:31:31.800
It's really a C compiler, or it's a 4 train compiler.
link |
00:31:36.120
It's not infrastructure in the same way.
link |
00:31:39.200
Is it, now you can tell,
link |
00:31:40.400
I don't know what I'm talking about
link |
00:31:41.560
because I keep saying C lang.
link |
00:31:44.520
You can always tell when a person is closed,
link |
00:31:48.080
by the way, pronounce something.
link |
00:31:49.400
I don't think, have I ever used Clang?
link |
00:31:52.600
Entirely possible.
link |
00:31:53.440
Have you, well, so you've used code,
link |
00:31:55.680
it's generated probably.
link |
00:31:58.200
So Clang is an LLVM or used to compile
link |
00:32:01.760
all the apps on the iPhone effectively and the OSes.
link |
00:32:05.240
It compiles Google's production server applications.
link |
00:32:09.360
It's used to build GameCube games and PlayStation 4
link |
00:32:14.880
and things like that.
link |
00:32:16.720
Those are the user I have,
link |
00:32:17.920
but just everything I've done that I experienced
link |
00:32:20.800
with Linux has been, I believe, always GCC.
link |
00:32:23.600
Yeah, I think Linux still defaults to GCC.
link |
00:32:25.720
And is there a reason for that?
link |
00:32:27.840
Or is it, I mean, is there a reason?
link |
00:32:29.480
It's a combination of technical and social reasons.
link |
00:32:32.080
Many Linux developers do use Clang,
link |
00:32:36.000
but the distributions, for lots of reasons,
link |
00:32:40.600
use GCC historically and they've not switched, yeah.
link |
00:32:44.280
Because it's just anecdotally online,
link |
00:32:46.680
it seems that LLVM has either reached the level of GCC
link |
00:32:50.680
or superseded on different features or whatever.
link |
00:32:53.560
The way I would say it is that they're so close
link |
00:32:55.240
it doesn't matter.
link |
00:32:56.080
Yeah, exactly.
link |
00:32:56.920
Like they're slightly better in some ways,
link |
00:32:58.160
slightly worse than otherwise,
link |
00:32:59.200
but it doesn't actually really matter anymore at that level.
link |
00:33:03.320
So in terms of optimization, breakthroughs,
link |
00:33:06.320
it's just been solid incremental work.
link |
00:33:09.200
Yeah, yeah, which describes a lot of compilers.
link |
00:33:12.200
The hard thing about compilers,
link |
00:33:14.360
in my experience, is the engineering,
link |
00:33:16.000
the software engineering, making it
link |
00:33:18.680
so that you can have hundreds of people collaborating
link |
00:33:20.920
on really detailed low level work and scaling that.
link |
00:33:25.400
And that's really hard.
link |
00:33:27.880
And that's one of the things I think LLVM has done well.
link |
00:33:30.720
And that kind of goes back to the original design goals
link |
00:33:34.160
with it to be modular and things like that.
link |
00:33:37.160
And incidentally, I don't want to take all the credit
link |
00:33:38.840
for this, right?
link |
00:33:39.680
I mean, some of the best parts about LLVM
link |
00:33:41.760
is that it was designed to be modular.
link |
00:33:43.600
And when I started, I would write,
link |
00:33:44.960
for example, a register allocator,
link |
00:33:46.840
and then somebody much smarter than me would come in
link |
00:33:49.040
and pull it out and replace it with something else
link |
00:33:51.320
that they would come up with.
link |
00:33:52.640
And because it's modular, they were able to do that.
link |
00:33:55.160
And that's one of the challenges with GCC, for example,
link |
00:33:58.240
is replacing subsystems is incredibly difficult.
link |
00:34:01.240
It can be done, but it wasn't designed for that.
link |
00:34:04.640
And that's one of the reasons that LLVM has been
link |
00:34:06.040
very successful in the research world as well.
link |
00:34:08.720
But in the community sense,
link |
00:34:11.040
Guido van Rasen, right?
link |
00:34:12.960
From Python, just retired from,
link |
00:34:18.080
what is it, benevolent, dictated for life, right?
link |
00:34:20.480
So in managing this community of brilliant compiler folks,
link |
00:34:24.720
is there, did it, for a time at least,
link |
00:34:28.640
fall on you to approve things?
link |
00:34:31.480
Oh yeah, so I mean, I still have something like
link |
00:34:34.240
an order of magnitude more patches in LLVM
link |
00:34:38.000
than anybody else.
link |
00:34:40.000
And many of those I wrote myself.
link |
00:34:42.760
But you're still right.
link |
00:34:43.840
I mean, you're still close to the,
link |
00:34:48.360
I don't know what the expression is to the metal.
link |
00:34:50.040
You're still right, Ko.
link |
00:34:51.040
Yeah, I'm still right, Ko.
link |
00:34:52.200
Not as much as I was able to in grad school,
link |
00:34:54.240
but that's an important part of my identity.
link |
00:34:56.760
But the way that LLVM has worked over time
link |
00:34:58.880
is that when I was a grad student,
link |
00:35:00.440
I could do all the work and steer everything
link |
00:35:03.000
and review every patch and make sure everything was done
link |
00:35:05.800
exactly the way my opinionated sense
link |
00:35:09.040
felt like it should be done.
link |
00:35:10.640
And that was fine.
link |
00:35:11.760
But as things scale, you can't do that, right?
link |
00:35:14.320
And so what ends up happening is LLVM has a hierarchical
link |
00:35:18.040
system of what's called code owners.
link |
00:35:20.520
These code owners are given the responsibility
link |
00:35:22.880
not to do all the work,
link |
00:35:24.920
not necessarily to review all the patches,
link |
00:35:26.680
but to make sure that the patches do get reviewed
link |
00:35:28.840
and make sure that the right thing's happening
link |
00:35:30.360
architecturally in their area.
link |
00:35:32.200
And so what you'll see is you'll see
link |
00:35:34.200
that for example, hardware manufacturers
link |
00:35:37.760
end up owning the hardware specific parts
link |
00:35:40.920
of their hardware, that's very common.
link |
00:35:45.560
Leaders in the community that have done really good work
link |
00:35:47.760
naturally become the de facto owner of something.
link |
00:35:50.920
And then usually somebody else is like,
link |
00:35:53.440
how about we make them the official code owner?
link |
00:35:55.520
And then we'll have somebody to make sure
link |
00:35:58.600
that all the patches get reviewed in a timely manner.
link |
00:36:00.320
And then everybody's like, yes, that's obvious.
link |
00:36:02.080
And then it happens, right?
link |
00:36:03.240
And usually this is a very organic thing, which is great.
link |
00:36:06.080
And so I'm nominally the top of that stack still,
link |
00:36:08.720
but I don't spend a lot of time reviewing patches.
link |
00:36:11.560
What I do is I help negotiate a lot of the technical
link |
00:36:16.520
disagreements that end up happening
link |
00:36:18.080
and making sure that the community as a whole
link |
00:36:19.680
makes progress and is moving in the right direction
link |
00:36:22.080
and doing that.
link |
00:36:23.960
So we also started a nonprofit six years ago,
link |
00:36:28.280
seven years ago, time's gone away.
link |
00:36:30.880
And the LVM Foundation nonprofit helps oversee
link |
00:36:34.640
all the business sides of things and make sure
link |
00:36:36.480
that the events that the LVM community has are funded
link |
00:36:39.680
and set up and run correctly and stuff like that.
link |
00:36:42.840
But the foundation is very much stays out
link |
00:36:45.200
of the technical side of where the project is going.
link |
00:36:49.080
Right, so it sounds like a lot of it is just organic, just.
link |
00:36:53.200
Yeah, well, and this is LVM is almost 20 years old,
link |
00:36:55.720
which is hard to believe.
link |
00:36:56.640
Somebody pointed out to me recently that LVM is now older
link |
00:37:00.360
than GCC was when LVM started, right?
link |
00:37:04.640
So time has a way of getting away from you.
link |
00:37:06.880
But the good thing about that is it has a really robust,
link |
00:37:10.440
really amazing community of people that are
link |
00:37:13.560
in their professional lives,
link |
00:37:14.720
spread across lots of different companies,
link |
00:37:16.320
but it's a community of people
link |
00:37:19.320
that are interested in similar kinds of problems
link |
00:37:21.160
and have been working together effectively for years
link |
00:37:23.720
and have a lot of trust and respect for each other.
link |
00:37:26.480
And even if they don't always agree that, you know,
link |
00:37:28.960
we're able to find a path forward.
link |
00:37:31.200
So then in a slightly different flavor of effort,
link |
00:37:34.520
you started at Apple in 2005 with the task of making,
link |
00:37:38.920
I guess, LVM production ready.
link |
00:37:41.840
And then eventually 2013 through 2017,
link |
00:37:44.680
leading the entire developer tools department.
link |
00:37:48.400
We're talking about LLVM, Xcode, Objective C to Swift.
link |
00:37:53.960
So in a quick overview of your time there,
link |
00:37:58.600
what were the challenges?
link |
00:37:59.640
First of all, leading such a huge group of developers.
link |
00:38:03.280
What was the big motivator dream mission
link |
00:38:06.560
behind creating Swift, the early birth of it
link |
00:38:11.440
from Objective C and so on and Xcode?
link |
00:38:13.440
What are some challenges?
link |
00:38:14.280
So these are different questions.
link |
00:38:15.920
Yeah, I know.
link |
00:38:16.760
But I want to talk about the other stuff too.
link |
00:38:19.560
I'll stay on the technical side,
link |
00:38:21.240
then we can talk about the big team pieces.
link |
00:38:23.440
That's okay?
link |
00:38:24.280
Sure.
link |
00:38:25.120
So it's to really oversimplify many years of hard work.
link |
00:38:27.760
LVM started, joined Apple, became a thing,
link |
00:38:32.440
became successful and became deployed.
link |
00:38:34.600
But then there was a question about
link |
00:38:36.760
how do we actually parse the source code?
link |
00:38:38.880
So LVM is that back part,
link |
00:38:40.320
the optimizer and the code generator.
link |
00:38:42.320
And LVM is really good for Apple as it went through
link |
00:38:44.640
a couple of hardware transitions.
link |
00:38:46.040
I joined right at the time of the Intel transition,
link |
00:38:47.920
for example, and 64 bit transitions
link |
00:38:51.800
and then the transition to ARM with the iPhone.
link |
00:38:53.480
And so LVM was very useful
link |
00:38:54.680
for some of these kinds of things.
link |
00:38:56.920
But at the same time,
link |
00:38:57.760
there's a lot of questions around developer experience.
link |
00:39:00.080
And so if you're a programmer pounding out
link |
00:39:01.880
at the time Objective C code,
link |
00:39:04.400
the error message you get, the compile time,
link |
00:39:06.440
the turnaround cycle, the tooling and the IDE
link |
00:39:09.680
were not great, were not as good as they could be.
link |
00:39:12.960
And so, as I occasionally do, I'm like,
link |
00:39:17.960
well, okay, how hard is it to write a C compiler?
link |
00:39:20.080
Right.
link |
00:39:20.920
And so I'm not gonna commit to anybody.
link |
00:39:22.520
I'm not gonna tell anybody.
link |
00:39:23.360
I'm just gonna just do it on nights and weekends
link |
00:39:25.960
and start working on it.
link |
00:39:27.400
And then I built up and see there's this thing
link |
00:39:30.120
called the preprocessor, which people don't like,
link |
00:39:32.960
but it's actually really hard and complicated
link |
00:39:35.440
and includes a bunch of really weird things
link |
00:39:37.640
like try graphs and other stuff like that
link |
00:39:39.240
that are really nasty.
link |
00:39:40.880
And it's the crux of a bunch of the performance issues
link |
00:39:44.000
in the compiler, start working on the parser
link |
00:39:46.560
and kind of got to the point where I'm like,
link |
00:39:47.720
oh, you know what, we could actually do this.
link |
00:39:49.840
Everybody's saying that this is impossible to do,
link |
00:39:51.400
but it's actually just hard.
link |
00:39:52.800
It's not impossible.
link |
00:39:53.880
And eventually told my manager about it
link |
00:39:57.520
and he's like, oh, wow, this is great.
link |
00:39:59.160
We do need to solve this problem.
link |
00:40:00.280
Oh, this is great.
link |
00:40:01.120
We can get you one other person to work with you on this.
link |
00:40:04.360
And so the team is formed and it starts taking off.
link |
00:40:08.240
And C++, for example, huge complicated language.
link |
00:40:11.960
People always assume that it's impossible to implement
link |
00:40:14.280
and it's very nearly impossible,
link |
00:40:16.160
but it's just really, really hard.
link |
00:40:18.640
And the way to get there is to build it
link |
00:40:20.760
one piece at a time incrementally.
link |
00:40:22.360
And that was only possible because we were lucky
link |
00:40:26.360
to hire some really exceptional engineers
link |
00:40:28.080
that knew various parts of it very well
link |
00:40:30.280
and could do great things.
link |
00:40:32.600
Swift was kind of a similar thing.
link |
00:40:34.360
So Swift came from, we were just finishing off
link |
00:40:39.080
the first version of C++ support in Clang.
link |
00:40:42.520
And C++ is a very formidable and very important language,
link |
00:40:47.160
but it's also ugly in lots of ways.
link |
00:40:49.240
And you can't implement C++ without thinking
link |
00:40:52.280
there has to be a better thing, right?
link |
00:40:54.320
And so I started working on Swift again
link |
00:40:56.080
with no hope or ambition that would go anywhere.
link |
00:40:58.520
Just let's see what could be done.
link |
00:41:00.760
Let's play around with this thing.
link |
00:41:02.560
It was me in my spare time,
link |
00:41:04.800
not telling anybody about it kind of a thing.
link |
00:41:08.160
And it made some good progress.
link |
00:41:09.360
I'm like, actually, it would make sense to do this.
link |
00:41:11.240
At the same time, I started talking with the senior VP
link |
00:41:14.760
of software at the time, a guy named Bertrand Sirle,
link |
00:41:17.680
and Bertrand was very encouraging.
link |
00:41:19.240
He was like, well, let's have fun, let's talk about this.
link |
00:41:22.040
And he was a little bit of a language guy.
link |
00:41:23.400
And so he helped guide some of the early work
link |
00:41:26.120
and encouraged me and got things off the ground.
link |
00:41:30.360
And eventually, I told my manager and told other people.
link |
00:41:34.240
And it started making progress.
link |
00:41:38.760
The complicating thing with Swift
link |
00:41:40.920
was that the idea of doing a new language
link |
00:41:43.840
is not obvious to anybody, including myself.
link |
00:41:47.760
And the tone at the time was that the iPhone
link |
00:41:50.160
was successful because of Objective C, right?
link |
00:41:53.360
Oh, interesting.
link |
00:41:54.360
In Objective C.
link |
00:41:55.200
Not despite of or just because of.
link |
00:41:57.080
And you have to understand that at the time,
link |
00:42:01.080
Apple was hiring software people that loved Objective C, right?
link |
00:42:05.360
And it wasn't that they came despite Objective C.
link |
00:42:07.920
They loved Objective C, and that's why they got hired.
link |
00:42:10.160
And so you had a software team that the leadership in many cases
link |
00:42:13.680
went all the way back to Next, where Objective C really became
link |
00:42:18.440
real.
link |
00:42:19.320
And so they, quote unquote, grew up writing Objective C.
link |
00:42:23.200
And many of the individual engineers
link |
00:42:25.680
all were hired because they loved Objective C.
link |
00:42:28.280
And so this notion of, OK, let's do new language
link |
00:42:30.520
was kind of heretical in many ways, right?
link |
00:42:34.040
Meanwhile, my sense was that the outside community wasn't really
link |
00:42:36.960
in love with Objective C. Some people were.
link |
00:42:38.520
And some of the most outspoken people were.
link |
00:42:40.200
But other people were hitting challenges
link |
00:42:42.600
because it has very sharp corners and it's difficult to learn.
link |
00:42:46.760
And so one of the challenges of making Swift happen
link |
00:42:50.040
that was totally non technical is the social part
link |
00:42:54.640
of what do we do?
link |
00:42:57.760
If we do a new language, which at Apple, many things
link |
00:43:00.280
happen that don't ship, right?
link |
00:43:02.200
So if we ship it, what is the metrics of success?
link |
00:43:05.520
Why would we do this?
link |
00:43:06.360
Why wouldn't we make Objective C better?
link |
00:43:07.920
If Objective C has problems, let's
link |
00:43:09.760
file off those rough corners and edges.
link |
00:43:12.120
And one of the major things that became the reason to do this
link |
00:43:15.600
was this notion of safety, memory safety.
link |
00:43:18.960
And the way Objective C works is that a lot of the object
link |
00:43:22.880
system and everything else is built on top of pointers
link |
00:43:26.440
in C. Objective C is an extension on top of C.
link |
00:43:29.920
And so pointers are unsafe.
link |
00:43:32.640
And if you get rid of the pointers,
link |
00:43:34.600
it's not Objective C anymore.
link |
00:43:36.400
And so fundamentally, that was an issue
link |
00:43:39.040
that you could not fix safety or memory safety
link |
00:43:42.160
without fundamentally changing the language.
link |
00:43:45.560
And so once we got through that part of the mental process
link |
00:43:49.880
and the thought process, it became a design process of saying,
link |
00:43:53.480
OK, well, if we're going to do something new, what is good?
link |
00:43:56.240
Like, how do we think about this?
link |
00:43:57.400
And what are we like, and what are we looking for?
link |
00:43:59.960
And that was a very different phase of it.
link |
00:44:02.400
So what are some design choices early on in Swift?
link |
00:44:05.880
Like, we're talking about braces.
link |
00:44:09.720
Are you making a type language or not?
link |
00:44:12.040
All those kinds of things.
link |
00:44:13.200
Yeah, so some of those were obvious given the context.
link |
00:44:16.000
So a type language, for example, Objective C
link |
00:44:18.240
is a type language, and going with an untyped language
link |
00:44:22.480
wasn't really seriously considered.
link |
00:44:24.280
We wanted the performance, and we wanted refactoring tools
link |
00:44:26.920
and other things like that that go with type languages.
link |
00:44:29.600
Quick dumb question.
link |
00:44:30.800
Yeah.
link |
00:44:31.400
Was it obvious?
link |
00:44:32.920
I think this would be a dumb question.
link |
00:44:34.600
But was it obvious that the language has
link |
00:44:36.520
to be a compiled language?
link |
00:44:38.920
Not an?
link |
00:44:40.120
Yes, that's not a dumb question.
link |
00:44:42.040
Earlier, I think late 90s, Apple
link |
00:44:44.000
had seriously considered moving its development experience to Java.
link |
00:44:48.960
But Swift started in 2010, which was several years
link |
00:44:53.120
after the iPhone.
link |
00:44:53.800
It was when the iPhone was definitely on an upper trajectory.
link |
00:44:56.600
And the iPhone was still extremely
link |
00:44:58.680
and is still a bit memory constrained.
link |
00:45:01.760
And so being able to compile the code and then ship it
link |
00:45:05.480
and then having standalone code that is not JIT compiled
link |
00:45:09.720
is a very big deal.
link |
00:45:11.320
And it's very much part of the Apple value system.
link |
00:45:15.200
Now, JavaScript's also a thing.
link |
00:45:17.520
I mean, it's not that this is exclusive,
link |
00:45:19.360
and technologies are good, depending on how they're applied.
link |
00:45:23.880
But in the design of Swift, saying how can we make
link |
00:45:27.200
Objective C better, Objective C was statically compiled,
link |
00:45:29.560
and that was the contiguous natural thing to do.
link |
00:45:32.480
Just skip ahead a little bit.
link |
00:45:34.640
Right back, just as a question, as you think about today
link |
00:45:37.600
in 2019, in your work at Google, TensorFlow, and so on,
link |
00:45:42.400
is, again, compilation, static compilation,
link |
00:45:47.480
still the right thing.
link |
00:45:49.480
Yeah, so the funny thing after working on compilers
link |
00:45:52.560
for a really long time is that, and this
link |
00:45:56.480
is one of the things that LLVM has helped with,
link |
00:45:59.080
is that I don't look at compilations
link |
00:46:01.480
being static or dynamic or interpreted or not.
link |
00:46:05.320
This is a spectrum, and one of the cool things about Swift
link |
00:46:09.160
is that Swift is not just statically compiled.
link |
00:46:12.200
It's actually dynamically compiled as well.
link |
00:46:14.160
And it can also be interpreted, though nobody's actually
link |
00:46:16.000
done that.
link |
00:46:17.560
And so what ends up happening when
link |
00:46:20.360
you use Swift in a workbook, for example,
link |
00:46:22.760
in Colab or in Jupyter, is it's actually dynamically
link |
00:46:25.320
compiling the statements as you execute them.
link |
00:46:28.320
And so this gets back to the software engineering problems,
link |
00:46:32.840
where if you layer the stack properly,
link |
00:46:34.960
you can actually completely change
link |
00:46:37.280
how and when things get compiled because you
link |
00:46:39.320
have the right abstractions there.
link |
00:46:41.120
And so the way that a Colab workbook works with Swift
link |
00:46:44.800
is that when you start typing into it,
link |
00:46:47.720
it creates a process, a UNIX process.
link |
00:46:50.320
And then each line of code you type in,
link |
00:46:52.240
it compiles it through the Swift compiler, the front end part,
link |
00:46:56.240
and then sends it through the optimizer,
link |
00:46:58.400
JIT compiles machine code, and then
link |
00:47:01.120
injects it into that process.
link |
00:47:03.920
And so as you're typing new stuff,
link |
00:47:06.560
it's like squirting in new code and overwriting and replacing
link |
00:47:09.360
and updating code in place.
link |
00:47:11.240
And the fact that it can do this is not an accident.
link |
00:47:13.520
Like Swift was designed for this.
link |
00:47:15.560
But it's an important part of how the language was set up
link |
00:47:18.120
and how it's layered.
link |
00:47:18.960
And this is a non obvious piece.
link |
00:47:21.360
And one of the things with Swift that was, for me,
link |
00:47:24.640
a very strong design point is to make it so that you
link |
00:47:27.040
can learn it very quickly.
link |
00:47:29.680
And so from a language design perspective,
link |
00:47:32.080
the thing that I always come back to is this UI principle
link |
00:47:34.520
of progressive disclosure of complexity.
link |
00:47:37.880
And so in Swift, you can start by saying print, quote,
link |
00:47:41.680
hello world, quote.
link |
00:47:43.960
And there's no slash n, just like Python, one line of code,
link |
00:47:47.160
no main, no header files, no public static class void,
link |
00:47:51.560
blah, blah, blah string, like Java has, one line of code.
link |
00:47:55.600
And you can teach that and it works great.
link |
00:47:58.280
Then you can say, well, let's introduce variables.
link |
00:48:00.280
And so you can declare a variable with var.
link |
00:48:02.400
So var x equals four.
link |
00:48:03.760
What is a variable?
link |
00:48:04.680
You can use x, x plus one.
link |
00:48:06.280
This is what it means.
link |
00:48:07.720
Then you can say, well, how about control flow?
link |
00:48:09.480
Well, this is one if statement is.
link |
00:48:10.840
This is what a for statement is.
link |
00:48:12.240
This is what a while statement is.
link |
00:48:15.320
Then you can say, let's introduce functions.
link |
00:48:17.280
And many languages like Python have
link |
00:48:20.000
had this kind of notion of let's introduce small things.
link |
00:48:22.800
And then you can add complexity.
link |
00:48:24.360
Then you can introduce classes.
link |
00:48:25.720
And then you can add generics in the case of Swift.
link |
00:48:28.040
And then you can build in modules and build out in terms
link |
00:48:30.600
of the things that you're expressing.
link |
00:48:32.200
But this is not very typical for compiled languages.
link |
00:48:35.800
And so this was a very strong design point.
link |
00:48:38.000
And one of the reasons that Swift in general
link |
00:48:40.960
is designed with this factoring of complexity in mind
link |
00:48:43.480
so that the language can express powerful things.
link |
00:48:46.440
You can write firmware in Swift if you want to.
link |
00:48:49.240
But it has a very high level feel, which is really
link |
00:48:52.800
this perfect blend.
link |
00:48:53.760
Because often you have very advanced library writers
link |
00:48:57.440
that want to be able to use the nitty gritty details.
link |
00:49:00.520
But then other people just want to use the libraries
link |
00:49:02.960
and work at a higher abstraction level.
link |
00:49:04.880
It's kind of cool that I saw that you can just
link |
00:49:07.200
enter a probability.
link |
00:49:09.200
I don't think I pronounced that word enough.
link |
00:49:11.320
But you can just drag in Python.
link |
00:49:14.920
It's just a string.
link |
00:49:15.960
You can import like, I saw this in the demo,
link |
00:49:18.840
import number.
link |
00:49:19.600
How do you make that happen?
link |
00:49:20.760
Yeah, well.
link |
00:49:21.240
What's up with that?
link |
00:49:22.520
Yeah.
link |
00:49:23.240
Is that as easy as it looks?
link |
00:49:24.960
Or is it?
link |
00:49:25.520
Yes, as easy as it looks.
link |
00:49:26.560
That's not a stage magic hack or anything like that.
link |
00:49:29.440
I don't mean from the user perspective.
link |
00:49:31.400
I mean from the implementation perspective
link |
00:49:33.200
to make it happen.
link |
00:49:34.120
So it's easy once all the pieces are in place.
link |
00:49:37.000
The way it works.
link |
00:49:37.920
So if you think about a dynamically typed language
link |
00:49:39.560
like Python, you can think about it in two different ways.
link |
00:49:42.160
You can say it has no types, which
link |
00:49:45.800
is what most people would say.
link |
00:49:47.480
Or you can say it has one type.
link |
00:49:50.440
And you can say it has one type and it's the Python object.
link |
00:49:53.360
And the Python object is passed around.
link |
00:49:55.040
And because there's only one type,
link |
00:49:56.280
it's implicit.
link |
00:49:58.240
And so what happens with Swift and Python talking to each other,
link |
00:50:01.320
Swift has lots of types, has arrays,
link |
00:50:03.320
and it has strings and all classes and that kind of stuff.
link |
00:50:07.040
But it now has a Python object type.
link |
00:50:11.120
So there is one Python object type.
link |
00:50:12.800
And so when you say import numpy, what you get
link |
00:50:16.440
is a Python object, which is the numpy module.
link |
00:50:19.880
And then you say np.array.
link |
00:50:22.160
It says, OK, hey Python object, I have no idea what you are.
link |
00:50:24.960
Give me your array member.
link |
00:50:27.280
OK, cool.
link |
00:50:27.960
And it just uses dynamic stuff, talks to the Python interpreter
link |
00:50:31.160
and says, hey Python, what's the dot array member
link |
00:50:33.680
in that Python object?
link |
00:50:35.680
It gives you back another Python object.
link |
00:50:37.400
And now you say, parentheses for the call
link |
00:50:39.480
and the arguments are going to pass.
link |
00:50:40.960
And so then it says, hey, a Python object that
link |
00:50:43.640
is the result of np.array, call with these arguments.
link |
00:50:48.040
Again, calling into the Python interpreter to do that work.
link |
00:50:50.320
And so right now, this is all really simple.
link |
00:50:53.680
And if you dive into the code, what you'll see
link |
00:50:55.960
is that the Python module in Swift
link |
00:50:58.440
is something like 1,200 lines of code or something.
link |
00:51:01.400
It's written in pure Swift.
link |
00:51:02.360
It's super simple.
link |
00:51:03.560
And it's built on top of the C interoperability
link |
00:51:06.560
because it just talks to the Python interpreter.
link |
00:51:09.520
But making that possible required us
link |
00:51:11.200
to add two major language features to Swift
link |
00:51:13.480
to be able to express these dynamic calls
link |
00:51:15.400
and the dynamic member lookups.
link |
00:51:17.200
And so what we've done over the last year
link |
00:51:19.480
is we've proposed, implement, standardized,
link |
00:51:23.080
and contributed new language features to the Swift language
link |
00:51:26.160
in order to make it so it is really trivial.
link |
00:51:29.560
And this is one of the things about Swift
link |
00:51:31.360
that is critical to the Swift for TensorFlow work, which
link |
00:51:35.000
is that we can actually add new language features.
link |
00:51:37.200
And the bar for adding those is high,
link |
00:51:39.160
but it's what makes it possible.
link |
00:51:42.160
So you're now at Google doing incredible work
link |
00:51:45.240
on several things, including TensorFlow.
link |
00:51:47.680
So TensorFlow 2.0 or whatever leading up to 2.0
link |
00:51:52.240
has, by default, in 2.0, has eager execution in yet
link |
00:51:57.360
in order to make code optimized for GPU or GPU
link |
00:52:00.480
or some of these systems computation
link |
00:52:04.080
needs to be converted to a graph.
link |
00:52:05.960
So what's that process like?
link |
00:52:07.400
What are the challenges there?
link |
00:52:08.920
Yeah, so I'm tangentially involved in this.
link |
00:52:11.680
But the way that it works with Autograph
link |
00:52:15.240
is that you mark your function with a decorator.
link |
00:52:21.600
And when Python calls it, that decorator is invoked.
link |
00:52:24.280
And then it says, before I call this function,
link |
00:52:28.240
you can transform it.
link |
00:52:29.480
And so the way Autograph works is, as far as I understand,
link |
00:52:32.400
is it actually uses the Python parser
link |
00:52:34.440
to go parse that, turn into a syntax tree,
link |
00:52:37.160
and now apply compiler techniques to, again,
link |
00:52:39.400
transform this down into TensorFlow graphs.
link |
00:52:42.320
And so you can think of it as saying, hey, I have an if statement.
link |
00:52:45.880
I'm going to create an if node in the graph, like you say,
link |
00:52:48.800
tf.cond.
link |
00:52:51.080
You have a multiply.
link |
00:52:53.000
Well, I'll turn that into a multiply node in the graph.
link |
00:52:55.320
And it becomes this tree transformation.
link |
00:52:57.720
So where does the Swift for TensorFlow come in?
link |
00:53:01.280
Which is parallels.
link |
00:53:04.720
For one, Swift is an interface.
link |
00:53:06.960
Like Python is an interface with TensorFlow.
link |
00:53:09.200
But it seems like there's a lot more going on
link |
00:53:11.200
than just a different language interface.
link |
00:53:13.120
There's optimization methodology.
link |
00:53:15.240
So the TensorFlow world has a couple of different, what
link |
00:53:19.560
I'd call front end technologies.
link |
00:53:21.240
And so Swift, and Python, and Go, and Rust, and Julian,
link |
00:53:25.400
all these things share the TensorFlow graphs
link |
00:53:29.360
and all the runtime and everything that's later.
link |
00:53:32.760
And so Swift for TensorFlow is merely another front end
link |
00:53:36.680
for TensorFlow, just like any of these other systems are.
link |
00:53:40.680
There's a major difference between, I would say,
link |
00:53:43.120
three camps of technologies here.
link |
00:53:44.640
There's Python, which is a special case,
link |
00:53:46.920
because the vast majority of the community efforts
link |
00:53:49.280
go into the Python interface.
link |
00:53:51.160
And Python has its own approaches
link |
00:53:53.000
for automatic differentiation, has its own APIs,
link |
00:53:55.800
and all this kind of stuff.
link |
00:53:58.200
There's Swift, which I'll talk about in a second.
link |
00:54:00.240
And then there's kind of everything else.
link |
00:54:02.080
And so the everything else are effectively language bindings.
link |
00:54:05.440
So they call into the TensorFlow runtime.
link |
00:54:08.000
But they usually don't have automatic differentiation,
link |
00:54:10.960
or they usually don't provide anything other than APIs that
link |
00:54:14.760
call the C APIs in TensorFlow.
link |
00:54:16.480
And so they're kind of wrappers for that.
link |
00:54:18.400
Swift is really kind of special.
link |
00:54:19.840
And it's a very different approach.
link |
00:54:22.800
Swift for TensorFlow, that is, is a very different approach,
link |
00:54:25.360
because there we're saying, let's
link |
00:54:26.920
look at all the problems that need
link |
00:54:28.440
to be solved in the full stack of the TensorFlow compilation
link |
00:54:34.120
process, if you think about it that way.
link |
00:54:35.680
Because TensorFlow is fundamentally a compiler.
link |
00:54:38.200
It takes models, and then it makes them go fast on hardware.
link |
00:54:42.760
That's what a compiler does.
link |
00:54:43.800
And it has a front end, it has an optimizer,
link |
00:54:47.560
and it has many back ends.
link |
00:54:49.320
And so if you think about it the right way,
link |
00:54:51.680
or if you look at it in a particular way,
link |
00:54:54.760
it is a compiler.
link |
00:54:59.280
And so Swift is merely another front end.
link |
00:55:02.120
But it's saying, and the design principle is saying,
link |
00:55:05.560
let's look at all the problems that we face as machine
link |
00:55:08.200
learning practitioners, and what is the best possible way
link |
00:55:11.200
we can do that, given the fact that we can change literally
link |
00:55:13.840
anything in this entire stack.
link |
00:55:15.920
And Python, for example, where the vast majority
link |
00:55:18.440
of the engineering and effort has gone into,
link |
00:55:22.600
is constrained by being the best possible thing you can do
link |
00:55:25.280
with a Python library.
link |
00:55:27.280
There are no Python language features
link |
00:55:29.280
that are added because of machine learning that I'm aware of.
link |
00:55:32.520
They added a matrix multiplication operator with that,
link |
00:55:35.080
but that's as close as you get.
link |
00:55:38.280
And so with Swift, it's hard, but you
link |
00:55:41.400
can add language features to the language,
link |
00:55:43.800
and there's a community process for that.
link |
00:55:46.080
And so we look at these things and say,
link |
00:55:48.000
well, what is the right division of labor
link |
00:55:49.680
between the human programmer and the compiler?
link |
00:55:52.000
And Swift has a number of things that shift that balance.
link |
00:55:55.280
So because it has a type system, for example,
link |
00:56:00.520
it makes certain things possible for analysis of the code,
link |
00:56:03.280
and the compiler can automatically
link |
00:56:05.520
build graphs for you without you thinking about them.
link |
00:56:08.800
That's a big deal for a programmer.
link |
00:56:10.520
You just get free performance.
link |
00:56:11.640
You get clustering and fusion and optimization,
link |
00:56:14.360
things like that, without you as a programmer having
link |
00:56:17.440
to manually do it because the compiler can do it for you.
link |
00:56:20.040
Automatic differentiation is another big deal,
link |
00:56:22.200
and I think one of the key contributions of the Swift
link |
00:56:25.440
for TensorFlow project is that there's
link |
00:56:29.600
this entire body of work on automatic differentiation that
link |
00:56:32.200
dates back to the Fortran days.
link |
00:56:34.240
People doing a tremendous amount of numerical computing
link |
00:56:36.360
in Fortran used to write what they call source to source
link |
00:56:39.800
translators, where you take a bunch of code, shove it
link |
00:56:43.600
into a mini compiler, and it would push out more Fortran
link |
00:56:47.280
code, but it would generate the backwards passes
link |
00:56:50.200
for your functions for you, the derivatives.
link |
00:56:53.000
And so in that work in the 70s, a tremendous number
link |
00:56:57.840
of optimizations, a tremendous number of techniques
link |
00:57:01.160
for fixing numerical instability,
link |
00:57:02.920
and other kinds of problems were developed,
link |
00:57:05.080
but they're very difficult to port into a world
link |
00:57:07.600
where in eager execution, you get an op by op at a time.
link |
00:57:11.280
Like, you need to be able to look at an entire function
link |
00:57:13.280
and be able to reason about what's going on.
link |
00:57:15.600
And so when you have a language integrated
link |
00:57:18.360
automatic differentiation, which is one of the things
link |
00:57:20.480
that the Swift project is focusing on,
link |
00:57:22.760
you can open all these techniques and reuse them
link |
00:57:25.720
in familiar ways, but the language integration piece
link |
00:57:30.160
has a bunch of design room in it, and it's also complicated.
link |
00:57:33.280
The other piece of the puzzle here,
link |
00:57:34.920
this kind of interesting is TPUs at Google.
link |
00:57:37.040
Yes.
link |
00:57:37.880
So, you know, we're in a new world with deep learning.
link |
00:57:40.200
It's constantly changing, and I imagine
link |
00:57:43.000
without disclosing anything, I imagine, you know,
link |
00:57:46.400
you're still innovating on the TPU front too.
link |
00:57:48.480
Indeed.
link |
00:57:49.320
So how much sort of interplays there are
link |
00:57:52.280
between software and hardware and trying to figure out
link |
00:57:54.440
how to gather, move towards an optimized solution.
link |
00:57:56.760
There's an incredible amount.
link |
00:57:57.800
So we're on our third generation of TPUs,
link |
00:57:59.520
which are now 100 petaflops in a very large
link |
00:58:02.800
liquid cooled box, virtual box with no cover.
link |
00:58:07.800
And as you might imagine, we're not out of ideas yet.
link |
00:58:11.320
The great thing about TPUs is that they're
link |
00:58:14.400
a perfect example of hardware software co design.
link |
00:58:17.640
And so it's about saying, what hardware
link |
00:58:19.840
do we build to solve certain classes of machine learning
link |
00:58:23.280
problems?
link |
00:58:23.920
Well, the algorithms are changing.
link |
00:58:26.360
Like the hardware takes some cases years to produce, right?
link |
00:58:30.480
And so you have to make bets and decide what is going to happen.
link |
00:58:34.160
And so what is the best way to spend the transistors
link |
00:58:37.280
to get the maximum performance per watt or area per cost
link |
00:58:41.560
or whatever it is that you're optimizing for?
link |
00:58:44.120
And so one of the amazing things about TPUs
link |
00:58:46.600
is this numeric format called B Float 16.
link |
00:58:50.040
B Float 16 is a compressed 16 bit floating point format,
link |
00:58:54.160
but it puts the bits in different places.
link |
00:58:56.120
In numeric terms, it has a smaller mantissa
link |
00:58:59.000
and a larger exponent.
link |
00:59:00.480
That means that it's less precise,
link |
00:59:03.000
but it can represent larger ranges of values, which
link |
00:59:06.120
in the machine learning context is really important and useful.
link |
00:59:08.640
Because sometimes you have very small gradients
link |
00:59:13.120
you want to accumulate and very, very small numbers that
link |
00:59:17.600
are important to move things as you're learning.
link |
00:59:20.520
But sometimes you have very large magnitude numbers as well.
link |
00:59:23.240
And B Float 16 is not as precise.
link |
00:59:26.880
The mantissa is small.
link |
00:59:28.240
But it turns out the machine learning algorithms actually
link |
00:59:30.360
want to generalize.
link |
00:59:31.640
And so there's theories that this actually
link |
00:59:34.320
increases the ability for the network
link |
00:59:36.440
to generalize across data sets.
link |
00:59:38.040
And regardless of whether it's good or bad,
link |
00:59:41.160
it's much cheaper at the hardware level
link |
00:59:42.640
to implement because the area and time of a multiplier
link |
00:59:48.160
is n squared in the number of bits in the mantissa,
link |
00:59:50.880
but it's linear with size of the exponent.
link |
00:59:53.360
And you're connected to both efforts here, both on the hardware
link |
00:59:56.360
and the software side?
link |
00:59:57.200
Yeah, and so that was a breakthrough coming
link |
00:59:59.240
from the research side and people working
link |
01:00:01.800
on optimizing network transport of weights
link |
01:00:06.000
across a network originally and trying
link |
01:00:08.280
to find ways to compress that.
link |
01:00:10.160
But then it got burned into silicon.
link |
01:00:12.160
And it's a key part of what makes TPU performance so amazing.
link |
01:00:15.320
And great.
link |
01:00:17.880
Now, TPUs have many different aspects that are important.
link |
01:00:20.640
But the co design between the low level compiler bits
link |
01:00:25.080
and the software bits and the algorithms
link |
01:00:27.360
is all super important.
link |
01:00:28.640
And it's this amazing trifecta that only Google can do.
link |
01:00:32.880
Yeah, that's super exciting.
link |
01:00:34.360
So can you tell me about MLIR project,
link |
01:00:38.440
previously the secretive one?
link |
01:00:41.400
Yeah, so MLIR is a project that we
link |
01:00:43.000
announced at a compiler conference three weeks ago
link |
01:00:46.960
or something at the Compilers for Machine Learning Conference.
link |
01:00:50.880
Basically, again, if you look at TensorFlow as a compiler
link |
01:00:53.280
stack, it has a number of compiler algorithms
link |
01:00:55.040
within it.
link |
01:00:56.000
It also has a number of compilers
link |
01:00:57.480
that get embedded into it.
link |
01:00:58.880
And they're made by different vendors.
link |
01:01:00.320
For example, Google has XLA, which is a great compiler system.
link |
01:01:04.640
NVIDIA has TensorFlow RT, Intel has NGraph.
link |
01:01:08.680
There's a number of these different compiler systems.
link |
01:01:10.640
And they're very hardware specific.
link |
01:01:13.600
And they're trying to solve different parts of the problems.
link |
01:01:16.280
But they're all kind of similar in a sense
link |
01:01:18.920
of they want to integrate with TensorFlow.
link |
01:01:20.680
Now, TensorFlow has an optimizer.
link |
01:01:22.720
And it has these different code generation technologies
link |
01:01:25.480
built in.
link |
01:01:26.360
The idea of MLIR is to build a common infrastructure
link |
01:01:28.680
to support all these different subsystems.
link |
01:01:31.040
And initially, it's to be able to make it so that they all
link |
01:01:34.120
plug in together.
link |
01:01:34.840
And they can share a lot more code and can be reusable.
link |
01:01:37.800
But over time, we hope that the industry will start
link |
01:01:40.960
collaborating and sharing code.
link |
01:01:42.440
And instead of reinventing the same things over and over again,
link |
01:01:45.200
that we can actually foster some of that working together
link |
01:01:49.240
to solve common problem energy that
link |
01:01:51.520
has been useful in the compiler field before.
link |
01:01:54.440
Beyond that, MLIR is some people have
link |
01:01:57.000
joked that it's kind of LLVM2.
link |
01:01:59.240
It learns a lot about what LLVM has been good
link |
01:02:01.760
and what LLVM has done wrong.
link |
01:02:04.280
And it's a chance to fix that.
link |
01:02:06.800
And also, there are challenges in the LLVM ecosystem
link |
01:02:09.320
as well, where LLVM is very good at the thing
link |
01:02:11.840
it was designed to do.
link |
01:02:12.680
But 20 years later, the world has changed.
link |
01:02:15.480
And people are trying to solve higher level problems.
link |
01:02:17.560
And we need some new technology.
link |
01:02:20.280
And what's the future of open source in this context?
link |
01:02:24.680
Very soon.
link |
01:02:25.680
So it is not yet open source.
link |
01:02:27.440
But it will be, hopefully, the next couple of months.
link |
01:02:29.360
So you still believe in the value of open source
link |
01:02:30.960
and these kinds of kinds?
link |
01:02:31.560
Oh, yeah, absolutely.
link |
01:02:32.400
And I think that the TensorFlow community at large
link |
01:02:36.080
fully believes in open source.
link |
01:02:37.640
So I mean, there is a difference between Apple,
link |
01:02:40.080
where you were previously in Google, now in spirit and culture.
link |
01:02:43.480
And I would say the open sourcing of TensorFlow
link |
01:02:45.440
was a seminal moment in the history of software.
link |
01:02:48.360
Because here's this large company releasing
link |
01:02:51.640
a very large code base that's open sourcing.
link |
01:02:55.880
What are your thoughts on that?
link |
01:02:57.880
How happy or not were you to see that kind
link |
01:03:00.800
of degree of open sourcing?
link |
01:03:02.880
So between the two, I prefer the Google approach,
link |
01:03:05.320
if that's what you're saying.
link |
01:03:07.800
The Apple approach makes sense given the historical context
link |
01:03:12.360
that Apple came from.
link |
01:03:13.360
But that's been 35 years ago.
link |
01:03:15.720
And I think that Apple is definitely adapting.
link |
01:03:18.160
And the way I look at it is that there's
link |
01:03:20.240
different kinds of concerns in the space, right?
link |
01:03:23.120
It is very rational for a business
link |
01:03:24.840
to care about making money.
link |
01:03:28.680
That fundamentally is what a business is about, right?
link |
01:03:31.600
But I think it's also incredibly realistic
link |
01:03:34.280
to say it's not your string library that's
link |
01:03:36.320
the thing that's going to make you money.
link |
01:03:38.040
It's going to be the amazing UI product differentiating
link |
01:03:41.440
features and other things like that
link |
01:03:42.880
that you build on top of your string library.
link |
01:03:45.200
And so keeping your string library proprietary and secret
link |
01:03:49.480
and things like that isn't maybe not the important thing
link |
01:03:53.480
anymore, right?
link |
01:03:54.680
Or before, platforms were different, right?
link |
01:03:57.720
And even 15 years ago, things were a little bit different.
link |
01:04:01.480
But the world is changing.
link |
01:04:02.880
So Google strikes a very good balance, I think.
link |
01:04:05.280
And I think that TensorFlow being open source
link |
01:04:08.680
really changed the entire machine learning field
link |
01:04:12.000
and it caused a revolution in its own right.
link |
01:04:14.080
And so I think it's amazingly forward looking
link |
01:04:17.560
because I could have imagined, and I wasn't at Google at the time,
link |
01:04:21.520
but I could imagine a different context in a different world
link |
01:04:23.760
where a company says, machine learning is critical
link |
01:04:26.520
to what we're doing, we're not going
link |
01:04:27.960
to give it to other people, right?
link |
01:04:29.600
And so that decision is a profoundly brilliant insight
link |
01:04:35.840
that I think has really led to the world being better
link |
01:04:38.320
and better for Google as well.
link |
01:04:40.160
And has all kinds of ripple effects.
link |
01:04:42.200
I think it is really, I mean, you can't
link |
01:04:45.400
understate Google deciding how profound that is for software.
link |
01:04:49.800
It's awesome.
link |
01:04:50.840
Well, and again, I can understand the concern
link |
01:04:54.880
about if we release our machine learning software,
link |
01:04:57.640
our competitors could go faster.
link |
01:05:00.400
But on the other hand, I think that open sourcing TensorFlow
link |
01:05:02.480
has been fantastic for Google.
link |
01:05:03.960
And I'm sure that decision was very nonobvious at the time,
link |
01:05:09.080
but I think it's worked out very well.
link |
01:05:11.480
So let's try this real quick.
link |
01:05:13.200
You were at Tesla for five months
link |
01:05:15.600
as the VP of autopilot software.
link |
01:05:17.600
You led the team during the transition from H Hardware
link |
01:05:20.480
1 to Hardware 2.
link |
01:05:22.320
I have a couple of questions.
link |
01:05:23.480
So one, first of all, to me, that's
link |
01:05:26.320
one of the bravest engineering decisions
link |
01:05:28.520
undertaking sort of like, undertaking really ever
link |
01:05:33.320
in the automotive industry to me, software wise,
link |
01:05:36.000
starting from scratch.
link |
01:05:37.440
It's a really brave engineering decision.
link |
01:05:39.320
So my one question is there is, what was that like?
link |
01:05:42.760
What was the challenge of that?
link |
01:05:43.960
Do you mean the career decision of jumping
link |
01:05:45.760
from a comfortable good job into the unknown?
link |
01:05:48.880
That combined, so at the individual level,
link |
01:05:51.560
you making that decision.
link |
01:05:54.640
And then when you show up, it's a really hard engineering
link |
01:05:58.040
problem.
link |
01:05:58.840
So you could just stay, maybe slow down, say, Hardware 1,
link |
01:06:04.880
or those kinds of decisions.
link |
01:06:06.560
So just taking it full on, let's do this from scratch.
link |
01:06:10.160
What was that like?
link |
01:06:11.080
Well, so I mean, I don't think Tesla
link |
01:06:12.680
has a culture of taking things slow and seeing how it goes.
link |
01:06:15.720
So one of the things that attracted me about Tesla
link |
01:06:18.080
is it's very much a gung ho.
link |
01:06:19.240
Let's change the world.
link |
01:06:20.200
Let's figure it out kind of a place.
link |
01:06:21.640
And so I have a huge amount of respect for that.
link |
01:06:25.680
Tesla has done very smart things with Hardware 1
link |
01:06:28.720
in particular.
link |
01:06:29.440
And the Hardware 1 design was originally designed
link |
01:06:32.760
to be very simple automation features in the car
link |
01:06:37.280
for like traffic aware cruise control and things like that.
link |
01:06:39.840
And the fact that they were able to effectively feature
link |
01:06:42.680
creep it into lane holding and a very useful driver assistance
link |
01:06:47.760
feature is pretty astounding, particularly given
link |
01:06:50.120
the details of the hardware.
link |
01:06:52.560
Hardware 2 built on that in a lot of ways.
link |
01:06:54.640
And the challenge there was that they were transitioning
link |
01:06:56.800
from a third party provided vision stack
link |
01:07:00.080
to an in house built vision stack.
link |
01:07:01.760
And so for the first step, which I mostly helped with,
link |
01:07:05.680
was getting onto that new vision stack.
link |
01:07:08.520
And that was very challenging.
link |
01:07:10.880
And it was time critical for various reasons.
link |
01:07:14.000
And it was a big leap.
link |
01:07:15.000
But it was fortunate that it built on a lot of the knowledge
link |
01:07:17.560
and expertise in the team that had built Hardware 1's
link |
01:07:20.880
driver assistance features.
link |
01:07:22.920
So you spoke in a collected and kind way
link |
01:07:25.400
about your time at Tesla.
link |
01:07:26.720
But it was ultimately not a good fit Elon Musk.
link |
01:07:30.280
We've talked on this podcast, several guests of the course.
link |
01:07:33.440
Elon Musk continues to do some of the most bold and innovative
link |
01:07:36.480
engineering work in the world at times
link |
01:07:38.800
at the cost to some of the members of the Tesla team.
link |
01:07:41.320
What did you learn about this working in this chaotic world
link |
01:07:45.120
with Elon?
link |
01:07:46.720
Yeah, so I guess I would say that when I was at Tesla,
link |
01:07:50.560
I experienced and saw the highest degree of turnover
link |
01:07:54.480
I'd ever seen in a company, which was a bit of a shock.
link |
01:07:58.280
But one of the things I learned and I came to respect
link |
01:08:00.520
is that Elon's able to attract amazing talent
link |
01:08:03.400
because he has a very clear vision of the future.
link |
01:08:05.640
And he can get people to buy into it
link |
01:08:07.200
because they want that future to happen.
link |
01:08:09.840
And the power of vision is something
link |
01:08:11.840
that I have a tremendous amount of respect for.
link |
01:08:14.200
And I think that Elon is fairly singular in the world
link |
01:08:17.600
in terms of the things he's able to get people to believe in.
link |
01:08:22.320
And there are many people that stand in the street corner
link |
01:08:27.360
and say, ah, we're going to go to Mars, right?
link |
01:08:29.320
But then there are a few people that
link |
01:08:31.600
can get others to buy into it and believe in, build the path
link |
01:08:35.200
and make it happen.
link |
01:08:36.160
And so I respect that.
link |
01:08:39.120
I don't respect all of his methods,
link |
01:08:41.000
but I have a huge amount of respect for that.
link |
01:08:44.960
You've mentioned in a few places,
link |
01:08:46.840
including in this context, working hard.
link |
01:08:50.400
What does it mean to work hard?
link |
01:08:51.960
And when you look back at your life,
link |
01:08:53.480
what were some of the most brutal periods of having
link |
01:08:59.040
to really put everything you have into something?
link |
01:09:03.360
Yeah, good question.
link |
01:09:05.040
So working hard can be defined a lot of different ways.
link |
01:09:07.480
So a lot of hours.
link |
01:09:08.680
And so that is true.
link |
01:09:12.440
The thing to me that's the hardest
link |
01:09:14.480
is both being short term focused on delivering and executing
link |
01:09:18.720
and making a thing happen, while also thinking
link |
01:09:21.080
about the longer term and trying to balance that, right?
link |
01:09:24.360
Because if you are myopically focused on solving a task
link |
01:09:28.480
and getting that done and only think about that incremental
link |
01:09:31.920
next step, you will miss the next big hill
link |
01:09:34.640
you should jump over to, right?
link |
01:09:36.360
And so I've been really fortunate
link |
01:09:38.000
that I've been able to kind of oscillate between the two.
link |
01:09:42.080
And historically at Apple, for example,
link |
01:09:45.600
that was made possible because I was
link |
01:09:47.080
able to work with some really amazing people and build up
link |
01:09:49.080
teams and leadership structures and allow
link |
01:09:53.760
them to grow in their careers and take on responsibility,
link |
01:09:57.120
thereby freeing up me to be a little bit crazy
link |
01:10:00.080
and thinking about the next thing.
link |
01:10:02.960
And so it's a lot of that.
link |
01:10:04.640
But it's also about with the experience
link |
01:10:06.760
you make connections that other people don't necessarily make.
link |
01:10:10.120
And so I think that's a big part as well.
link |
01:10:12.960
But the bedrock is just a lot of hours.
link |
01:10:16.040
And that's OK with me.
link |
01:10:19.720
There's different theories on work life balance.
link |
01:10:21.480
And my theory for myself, which I do not project onto the team,
link |
01:10:25.160
but my theory for myself is that I
link |
01:10:28.480
want to love what I'm doing and work really hard.
link |
01:10:30.400
And my purpose, I feel like, and my goal
link |
01:10:33.960
is to change the world and make it a better place.
link |
01:10:36.240
And that's what I'm really motivated to do.
link |
01:10:40.000
So last question, LLVM logo is a dragon.
link |
01:10:44.760
You explained that this is because dragons
link |
01:10:46.760
have connotations of power, speed, intelligence.
link |
01:10:50.320
It can also be sleek, elegant, and modular,
link |
01:10:53.320
though you remove the modular part.
link |
01:10:56.280
What is your favorite dragon related character
link |
01:10:58.920
from fiction, video, or movies?
link |
01:11:01.480
So those are all very kind ways of explaining it.
link |
01:11:03.840
Do you want to know the real reason it's a dragon?
link |
01:11:06.200
Yeah.
link |
01:11:07.000
Is that better?
link |
01:11:07.920
So there is a seminal book on compiler design
link |
01:11:11.040
called The Dragon Book.
link |
01:11:12.480
And so this is a really old now book on compilers.
link |
01:11:16.280
And so the Dragon logo for LLVM came about because at Apple,
link |
01:11:22.040
we kept talking about LLVM related technologies,
link |
01:11:24.720
and there's no logo to put on a slide.
link |
01:11:26.960
And we're like, what do we do?
link |
01:11:28.440
And somebody's like, well, what kind of logo
link |
01:11:30.000
should a compiler technology have?
link |
01:11:32.160
And I'm like, I don't know.
link |
01:11:33.320
I mean, the dragon is the best thing that we've got.
link |
01:11:37.280
And Apple somehow magically came up with the logo.
link |
01:11:40.600
And it was a great thing, and the whole community
link |
01:11:43.240
rallied around it.
link |
01:11:44.000
And then it got better as other graphic designers got
link |
01:11:46.840
involved.
link |
01:11:47.320
But that's originally where it came from.
link |
01:11:49.280
The story.
link |
01:11:50.080
Is there dragons from fiction that you connect with?
link |
01:11:53.960
That Game of Thrones, Lord of the Rings, that kind of thing?
link |
01:11:58.000
Lord of the Rings is great.
link |
01:11:59.120
I also like role playing games and things like computer
link |
01:12:01.440
role playing games.
link |
01:12:02.160
And so dragons often show up in there.
link |
01:12:03.600
But it really comes back to the book.
link |
01:12:07.080
Oh, no, we need a thing.
link |
01:12:08.480
We need a lot to do.
link |
01:12:09.880
And hilariously, one of the funny things about LLVM
link |
01:12:13.640
is that my wife, who's amazing, runs the LLVM foundation.
link |
01:12:19.400
And she goes to Grace Hopper, and is
link |
01:12:21.040
trying to get more women involved.
link |
01:12:22.480
And she's also a compiler engineer.
link |
01:12:24.600
So she's trying to get other women
link |
01:12:26.040
to get interested in compilers and things like this.
link |
01:12:28.120
And so she hands out the stickers.
link |
01:12:29.960
And people like the LLVM sticker because of Game of Thrones.
link |
01:12:34.240
And so sometimes culture has this helpful effect
link |
01:12:36.800
to get the next generation of compiler engineers engaged
link |
01:12:41.040
with the cause.
link |
01:12:42.320
OK, awesome.
link |
01:12:43.240
Grace, thanks so much for talking with us.
link |
01:12:44.680
It's been great talking with you.