back to indexGeorge Hotz: Comma.ai, OpenPilot, and Autonomous Vehicles | Lex Fridman Podcast #31
link |
The following is a conversation with George Hotz.
link |
He's the founder of Kama AI,
link |
a machine learning based vehicle automation company.
link |
He is most certainly an outspoken personality
link |
in the field of AI and technology in general.
link |
He first gained recognition for being the first person
link |
to carry or unlock an iPhone.
link |
And since then, he's done quite a few interesting things
link |
at the intersection of hardware and software.
link |
This is the Artificial Intelligence Podcast.
link |
If you enjoy it, subscribe on YouTube,
link |
give it five stars on iTunes, support it on Patreon,
link |
or simply connect with me on Twitter
link |
at Lex Friedman, spelled F R I D M A N.
link |
And I'd like to give a special thank you
link |
to Jennifer from Canada
link |
for her support of the podcast on Patreon.
link |
Merci beaucoup, Jennifer.
link |
She's been a friend and an engineering colleague
link |
for many years since I was in grad school.
link |
Your support means a lot
link |
and inspires me to keep this series going.
link |
And now, here's my conversation with George Hotz.
link |
Do you think we're living in a simulation?
link |
Yes, but it may be unfalsifiable.
link |
What do you mean by unfalsifiable?
link |
So if the simulation is designed in such a way
link |
that they did like a formal proof
link |
to show that no information can get in and out,
link |
and if their hardware is designed
link |
for the anything in the simulation
link |
to always keep the hardware in spec,
link |
it may be impossible to prove
link |
whether we're in a simulation or not.
link |
So they've designed it such that it's a closed system
link |
you can't get outside the system.
link |
Well, maybe it's one of three worlds.
link |
We're either in a simulation which can be exploited,
link |
we're in a simulation which not only can't be exploited,
link |
but like the same thing's true about VMs.
link |
A really well designed VM,
link |
you can't even detect if you're in a VM or not.
link |
So the simulation is running on a virtual machine.
link |
But now in reality, all VMs have ways to detect.
link |
I mean, you've done quite a bit of hacking yourself.
link |
So you should know that really any complicated system
link |
will have ways in and out.
link |
So this isn't necessarily true going forward.
link |
I spent my time away from Comma,
link |
I learned Coq, it's a dependently typed,
link |
it's a language for writing math proofs in.
link |
And if you write code that compiles in a language like that,
link |
it is correct by definition.
link |
The types check its correctness.
link |
So it's possible that the simulation
link |
is written in a language like this, in which case, yeah.
link |
Yeah, but that can't be sufficiently expressive
link |
a language like that.
link |
Okay, well, so all right, so.
link |
The simulation doesn't have to be Turing complete
link |
if it has a scheduled end date.
link |
Looks like it does actually with entropy.
link |
I mean, I don't think that a simulation
link |
that results in something as complicated as the universe
link |
would have a form of proof of correctness, right?
link |
It's possible, of course.
link |
We have no idea how good their tooling is.
link |
And we have no idea how complicated
link |
the universe computer really is.
link |
It may be quite simple.
link |
It's just very large, right?
link |
It's very, it's definitely very large.
link |
But the fundamental rules might be super simple.
link |
Yeah, Conway's getting a life kind of stuff.
link |
So if you could hack,
link |
so imagine a simulation that is hackable,
link |
if you could hack it,
link |
what would you change about the,
link |
like how would you approach hacking a simulation?
link |
The reason I gave that talk.
link |
By the way, I'm not familiar with the talk you gave.
link |
I just read that you talked about escaping the simulation
link |
or something like that.
link |
So maybe you can tell me a little bit about the theme
link |
and the message there too.
link |
It wasn't a very practical talk
link |
about how to actually escape a simulation.
link |
It was more about a way of restructuring
link |
an us versus them narrative.
link |
we continue on the path we're going with technology,
link |
I think we're in big trouble,
link |
like as a species and not just as a species,
link |
but even as me as an individual member of the species.
link |
So if we could change rhetoric
link |
to be more like to think upwards,
link |
like to think about that we're in a simulation
link |
and how we could get out,
link |
already we'd be on the right path.
link |
What you actually do once you do that,
link |
well, I assume I would have acquired way more intelligence
link |
in the process of doing that.
link |
So I'll just ask that.
link |
So the thinking upwards,
link |
what kind of ideas,
link |
what kind of breakthrough ideas
link |
do you think thinking in that way could inspire?
link |
And why did you say upwards?
link |
Are you thinking sort of exploration in all forms?
link |
The space narrative
link |
that held for the modernist generation
link |
doesn't hold as well for the postmodern generation.
link |
What's the space narrative?
link |
Are we talking about the same space,
link |
the three dimensional space?
link |
No, no, no, space, like going on space,
link |
like building like Elon Musk,
link |
like we're going to build rockets,
link |
we're going to go to Mars,
link |
we're going to colonize the universe.
link |
And the narrative you're referring,
link |
I was born in the Soviet Union,
link |
you're referring to the race to space.
link |
The race to space, yeah.
link |
That was a great modernist narrative.
link |
It doesn't seem to hold the same weight in today's culture.
link |
I'm hoping for good postmodern narratives that replace it.
link |
So let's think, so you work a lot with AI.
link |
So AI is one formulation of that narrative.
link |
There could be also,
link |
I don't know how much you do in VR and AR.
link |
That's another, I know less about it,
link |
but every time I play with it in our research,
link |
it's fascinating, that virtual world.
link |
Are you interested in the virtual world?
link |
I would like to move to virtual reality.
link |
In terms of your work?
link |
No, I would like to physically move there.
link |
The apartment I can rent in the cloud
link |
is way better than the apartment
link |
I can rent in the real world.
link |
Well, it's all relative, isn't it?
link |
Because others will have very nice apartments too,
link |
so you'll be inferior in the virtual world as well.
link |
No, but that's not how I view the world, right?
link |
I don't view the world,
link |
I mean, it's a very almost zero sum ish way
link |
to view the world.
link |
Say like, my great apartment isn't great
link |
because my neighbor has one too.
link |
No, my great apartment is great
link |
because look at this dishwasher, man.
link |
You just touch the dish and it's washed, right?
link |
And that is great in and of itself
link |
if I have the only apartment
link |
or if everybody had the apartment.
link |
So you have fundamental gratitude.
link |
The world first learned of George Hots
link |
in August 2007, maybe before then,
link |
but certainly in August 2007
link |
when you were the first person to unlock,
link |
carry unlock an iPhone.
link |
How did you get into hacking?
link |
What was the first system
link |
you discovered vulnerabilities for and broke into?
link |
So that was really kind of the first thing.
link |
I had a book in 2006 called Grey Hat Hacking.
link |
And I guess I realized that if you acquired
link |
these sort of powers, you could control the world.
link |
But I didn't really know that much
link |
about computers back then.
link |
I started with electronics.
link |
The first iPhone hack was physical.
link |
You had to open it up and pull an address line high.
link |
And it was because I didn't really know
link |
about software exploitation.
link |
I learned that all in the next few years
link |
and I got very good at it.
link |
But back then I knew about like how memory chips
link |
are connected to processors and stuff.
link |
You knew about software and programming.
link |
You just didn't know.
link |
So your view of the world and computers
link |
was physical, was hardware.
link |
Actually, if you read the code that I released
link |
with that in August 2007, it's atrocious.
link |
What language was it?
link |
And in a broken sort of state machine ask C.
link |
I didn't know how to program.
link |
So how did you learn to program?
link |
What was your journey?
link |
Cause I mean, we'll talk about it.
link |
You've live streamed some of your programming.
link |
This chaotic, beautiful mess.
link |
How did you arrive at that?
link |
Years and years of practice.
link |
I interned at Google after the summer
link |
after the iPhone unlock.
link |
And I did a contract for them where I built hardware
link |
for Street View and I wrote a software library
link |
to interact with it.
link |
And it was terrible code.
link |
And for the first time I got feedback from people
link |
who I respected saying, no, like don't write code like this.
link |
Now, of course, just getting that feedback is not enough.
link |
The way that I really got good was I wanted to write
link |
this thing like that could emulate and then visualize
link |
like arm binaries.
link |
Cause I wanted to hack the iPhone better.
link |
And I didn't like that I couldn't like see
link |
what the, I couldn't single step through the processor
link |
because I had no debugger on there,
link |
especially for the low level things like the boot rum
link |
and the bootloader.
link |
So I tried to build this tool to do it.
link |
And I built the tool once and it was terrible.
link |
I built the tool a second time, it was terrible.
link |
I built the tool a third time.
link |
This was by the time I was at Facebook, it was kind of okay.
link |
And then I built the tool a fourth time
link |
when I was a Google intern again in 2014.
link |
And that was the first time I was like,
link |
this is finally usable.
link |
How do you pronounce this Kira?
link |
So it's essentially the most efficient way
link |
to visualize the change of state of the computer
link |
as the program is running.
link |
That's what you mean by debugger.
link |
Yeah, it's a timeless debugger.
link |
So you can rewind just as easily as going forward.
link |
Think about if you're using GDB,
link |
you have to put a watch on a variable.
link |
If you wanna see if that variable changes.
link |
In Kira, you can just click on that variable
link |
and then it shows every single time
link |
when that variable was changed or accessed.
link |
Think about it like Git for your computers, the run log.
link |
So there's like a deep log of the state of the computer
link |
as the program runs and you can rewind.
link |
Why isn't that, maybe it is, maybe you can educate me.
link |
Why isn't that kind of debugging used more often?
link |
Cause the tooling's bad.
link |
One, if you're trying to debug Chrome,
link |
Chrome is a 200 megabyte binary
link |
that runs slowly on desktops.
link |
So that's gonna be really hard to use for that.
link |
But it's really good to use for like CTFs
link |
and for boot roms and for small parts of code.
link |
So it's hard if you're trying to debug like massive systems.
link |
What's a CTF and what's a boot rom?
link |
A boot rom is the first code that executes
link |
the minute you give power to your iPhone.
link |
And CTF where these competitions
link |
that I played capture the flag.
link |
Capture the flag, I was gonna ask you about that.
link |
What are those, look at,
link |
I watched a couple of videos on YouTube,
link |
those look fascinating.
link |
What have you learned about maybe
link |
at the high level of vulnerability of systems
link |
from these competitions?
link |
I feel like in the heyday of CTFs,
link |
you had all of the best security people in the world
link |
challenging each other and coming up
link |
with new toy exploitable things over here.
link |
And then everybody, okay, who can break it?
link |
And when you break it, you get like,
link |
there's like a file on the server called flag.
link |
And then there's a program running,
link |
listening on a socket that's vulnerable.
link |
So you write an exploit, you get a shell,
link |
and then you cat flag, and then you type the flag
link |
into like a web based scoreboard and you get points.
link |
So the goal is essentially,
link |
to find an exploit in the system
link |
that allows you to run shell,
link |
to run arbitrary code on that system.
link |
That's one of the categories.
link |
That's like the pwnable category.
link |
It's like, you know, you pwn the program.
link |
It's a program that's, yeah.
link |
Yeah, you know, first of all, I apologize.
link |
I'm gonna say it's because I'm Russian,
link |
but maybe you can help educate me.
link |
Some video game like misspelled own way back in the day.
link |
Yeah, and it's just, I wonder if there's a definition.
link |
I'll have to go to Urban Dictionary for it.
link |
It'll be interesting to see what it says.
link |
Okay, so what was the heyday of CTF, by the way?
link |
But was it, what decade are we talking about?
link |
I think like, I mean, maybe unbiased
link |
because it's the era that I played,
link |
but like 2011 to 2015,
link |
because the modern CTF scene
link |
is similar to the modern competitive programming scene.
link |
You have people who like do drills.
link |
You have people who practice.
link |
And then once you've done that,
link |
you've turned it less into a game of generic computer skill
link |
and more into a game of, okay,
link |
you drill on these five categories.
link |
And then before that, it wasn't,
link |
it didn't have like as much attention as it had.
link |
I don't know, they were like,
link |
I won $30,000 once in Korea for one of these competitions.
link |
Yeah, they were, they were, that was.
link |
So that means, I mean, money is money,
link |
but that means there was probably good people there.
link |
Are the challenges human constructed
link |
or are they grounded in some real flaws and real systems?
link |
Usually they're human constructed,
link |
but they're usually inspired by real flaws.
link |
What kind of systems are imagined
link |
is really focused on mobile.
link |
Like what has vulnerabilities these days?
link |
Is it primarily mobile systems like Android?
link |
Oh, everything does.
link |
Still. Yeah, of course.
link |
The price has kind of gone up
link |
because less and less people can find them.
link |
And what's happened in security
link |
is now if you want to like jailbreak an iPhone,
link |
you don't need one exploit anymore, you need nine.
link |
Nine chained together, what would it mean?
link |
Okay, so it's really,
link |
what's the benefit speaking higher level
link |
philosophically about hacking?
link |
I mean, it sounds from everything I've seen about you,
link |
you just love the challenge
link |
and you don't want to do anything.
link |
You don't want to bring that exploit out into the world
link |
and do any actual, let it run wild.
link |
You just want to solve it
link |
and then you go on to the next thing.
link |
Oh yeah, I mean, doing criminal stuff's not really worth it.
link |
And I'll actually use the same argument
link |
for why I don't do defense for why I don't do crime.
link |
If you want to defend a system,
link |
say the system has 10 holes, right?
link |
If you find nine of those holes as a defender,
link |
you still lose because the attacker
link |
gets in through the last one.
link |
If you're an attacker,
link |
you only have to find one out of the 10.
link |
But if you're a criminal,
link |
if you log on with a VPN nine out of the 10 times,
link |
but one time you forget, you're done.
link |
Because you're caught, okay.
link |
Because you only have to mess up once
link |
to be caught as a criminal.
link |
That's why I'm not a criminal.
link |
because I was having a discussion with somebody
link |
just at a high level about nuclear weapons actually,
link |
why we're having blown ourselves up yet.
link |
And my feeling is all the smart people in the world,
link |
if you look at the distribution of smart people,
link |
smart people are generally good.
link |
And then this other person I was talking to,
link |
Sean Carroll, the physicist,
link |
and he was saying, no, good and bad people
link |
are evenly distributed amongst everybody.
link |
My sense was good hackers are in general good people
link |
and they don't want to mess with the world.
link |
What's your sense?
link |
I'm not even sure about that.
link |
I have a nice life.
link |
Crime wouldn't get me anything.
link |
But if you're good and you have these skills,
link |
you probably have a nice life too, right?
link |
Right, you can use it for other things.
link |
But is there an ethical,
link |
is there a little voice in your head that says,
link |
well, yeah, if you could hack something
link |
to where you could hurt people
link |
and you could earn a lot of money doing it though,
link |
not hurt physically perhaps,
link |
but disrupt their life in some kind of way,
link |
isn't there a little voice that says?
link |
One, I don't really care about money.
link |
So like the money wouldn't be an incentive.
link |
The thrill might be an incentive.
link |
But when I was 19, I read Crime and Punishment.
link |
And that was another great one
link |
that talked me out of ever really doing crime.
link |
Cause it's like, that's gonna be me.
link |
I'd get away with it, but it would just run through my head.
link |
Even if I got away with it, you know?
link |
And then you do crime for long enough,
link |
you'll never get away with it.
link |
In the end, that's a good reason to be good.
link |
I wouldn't say I'm good.
link |
I would just say I'm not bad.
link |
You're a talented programmer and a hacker
link |
in a good positive sense of the word.
link |
You've played around,
link |
found vulnerabilities in various systems.
link |
What have you learned broadly
link |
about the design of systems and so on
link |
from that whole process?
link |
You learn to not take things
link |
for what people say they are,
link |
but you look at things for what they actually are.
link |
I understand that's what you tell me it is,
link |
but what does it do?
link |
And you have nice visualization tools
link |
to really know what it's really doing.
link |
I'm a better programmer now than I was in 2014.
link |
I said, Kira, that was the first tool
link |
that I wrote that was usable.
link |
I wouldn't say the code was great.
link |
I still wouldn't say my code is great.
link |
So how was your evolution as a programmer except practice?
link |
So you started with C.
link |
At which point did you pick up Python?
link |
Because you're pretty big in Python now.
link |
Now, yeah, in college.
link |
I went to Carnegie Mellon when I was 22.
link |
I'm like, all right,
link |
I'm gonna take all your hardest CS courses.
link |
We'll see how I do, right?
link |
Like, did I miss anything
link |
by not having a real undergraduate education?
link |
Took operating systems, compilers, AI,
link |
and they're like a freshman wheat or math course.
link |
Operating systems, some of those classes
link |
you mentioned are pretty tough, actually.
link |
At least the 2012, circa 2012,
link |
operating systems and compilers were two of the,
link |
they were the best classes I've ever taken in my life.
link |
Because you write an operating system
link |
and you write a compiler.
link |
I wrote my operating system in C
link |
and I wrote my compiler in Haskell,
link |
but somehow I picked up Python that semester as well.
link |
I started using it for the CTFs, actually.
link |
That's when I really started to get into CTFs
link |
and CTFs, you're all, it's a race against the clock.
link |
So I can't write things in C.
link |
Oh, there's a clock component.
link |
So you really want to use the programming languages
link |
so you can be fastest.
link |
48 hours, pwn as many of these challenges as you can.
link |
Yeah, you got like a hundred points of challenge.
link |
Whatever team gets the most.
link |
You were both at Facebook and Google for a brief stint.
link |
With Project Zero actually at Google for five months
link |
where you developed Kira.
link |
What was Project Zero about in general?
link |
What, I'm just curious about the security efforts
link |
in these companies.
link |
Well, Project Zero started the same time I went there.
link |
What years are there?
link |
So that was right at the beginning of Project Zero.
link |
It's Google's offensive security team.
link |
I'll try to give the best public facing explanation
link |
So the idea is basically these vulnerabilities
link |
exist in the world.
link |
Nation states have them.
link |
Some high powered bad actors have them.
link |
Sometime people will find these vulnerabilities
link |
and submit them in bug bounties to the companies.
link |
But a lot of the companies don't really care.
link |
They don't even fix the bug.
link |
It doesn't hurt for there to be a vulnerability.
link |
So Project Zero is like, we're going to do it different.
link |
We're going to announce a vulnerability
link |
and we're going to give them 90 days to fix it.
link |
And then whether they fix it or not,
link |
we're going to drop the zero day.
link |
We're going to drop the weapon.
link |
I love the deadlines.
link |
Oh, that's so cool.
link |
Give them real deadlines.
link |
And I think it's done a lot for moving the industry forward.
link |
I watched your coding sessions on the streamed online.
link |
You code things up, the basic projects,
link |
usually from scratch.
link |
I would say sort of as a programmer myself,
link |
just watching you that you type really fast
link |
and your brain works in both brilliant and chaotic ways.
link |
I don't know if that's always true,
link |
but certainly for the live streams.
link |
So it's interesting to me because I'm more,
link |
I'm much slower and systematic and careful.
link |
And you just move, I mean,
link |
probably in order of magnitude faster.
link |
So I'm curious, is there a method to your madness?
link |
Is it just who you are?
link |
There's pros and cons.
link |
There's pros and cons to my programming style.
link |
And I'm aware of them.
link |
Like if you ask me to like get something up
link |
and working quickly with like an API
link |
that's kind of undocumented,
link |
I will do this super fast
link |
because I will throw things at it until it works.
link |
If you ask me to take a vector and rotate it 90 degrees
link |
and then flip it over the XY plane,
link |
I'll spam program for two hours and won't get it.
link |
Oh, because it's something that you could do
link |
with a sheet of paper, think through design,
link |
and then just, do you really just throw stuff at the wall
link |
and you get so good at it that it usually works?
link |
I should become better at the other kind as well.
link |
Sometimes I'll do things methodically.
link |
It's nowhere near as entertaining on the Twitch streams.
link |
I do exaggerate it a bit on the Twitch streams as well.
link |
The Twitch streams, I mean,
link |
what do you want to see a game or you want to see
link |
actions per minute, right?
link |
I'll show you APM for programming too.
link |
Yeah, I recommend people go to it.
link |
I think I watched, I watched probably several hours
link |
of you, like I've actually left you programming
link |
in the background while I was programming
link |
because you made me, it was like watching
link |
a really good gamer.
link |
It's like energizes you because you're like moving so fast.
link |
It's so, it's awesome.
link |
It's inspiring and it made me jealous that like,
link |
because my own programming is inadequate
link |
in terms of speed.
link |
So I'm twice as frantic on the live streams
link |
as I am when I code without them.
link |
It's super entertaining.
link |
So I wasn't even paying attention to what you were coding,
link |
It's just watching you switch windows and Vim I guess
link |
Yeah, there's Vim on screen.
link |
I've developed the workload at Facebook
link |
and stuck with it.
link |
How do you learn new programming tools,
link |
ideas, techniques these days?
link |
What's your like a methodology for learning new things?
link |
So I wrote for comma, the distributed file systems
link |
out in the world are extremely complex.
link |
Like if you want to install something like like like Ceph,
link |
Ceph is I think the like open infrastructure
link |
distributed file system,
link |
or there's like newer ones like seaweed FS,
link |
but these are all like 10,000 plus line projects.
link |
I think some of them are even a hundred thousand line
link |
and just configuring them as a nightmare.
link |
So I wrote, I wrote one, it's 200 lines
link |
and it's, it uses like NGINX and volume servers
link |
and has this little master server that I wrote in Go.
link |
And the way I go, this,
link |
if I would say that I'm proud per line of any code I wrote,
link |
maybe there's some exploits that I think are beautiful.
link |
And then this, this is 200 lines.
link |
And just the way that I thought about it,
link |
I think was very good.
link |
And the reason it's very good is because
link |
that was the fourth version of it that I wrote.
link |
And I had three versions that I threw away.
link |
You mentioned, did you say Go?
link |
I wrote in Go, yeah.
link |
Is that a functional language?
link |
I forget what Go is.
link |
Go is Google's language.
link |
It's not functional.
link |
It's some, it's like in a way it's C++, but easier.
link |
It's, it's strongly typed.
link |
It has a nice ecosystem around it.
link |
When I first looked at it, I was like, this is like Python,
link |
but it takes twice as long to do anything.
link |
Now that I've, OpenPilot is migrating to C,
link |
but it still has large Python components.
link |
I now understand why Python doesn't work
link |
for large code bases and why you want something like Go.
link |
So why, why doesn't Python work for,
link |
so even most, speaking for myself at least,
link |
like we do a lot of stuff,
link |
basically demo level work with autonomous vehicles
link |
and most of the work is Python.
link |
Why doesn't Python work for large code bases?
link |
Because, well, lack of type checking is a big part.
link |
So errors creep in.
link |
And like, you don't know,
link |
the compiler can tell you like nothing, right?
link |
So everything is either, you know,
link |
like, like syntax errors, fine.
link |
But if you misspell a variable in Python,
link |
the compiler won't catch that.
link |
There's like linters that can catch it some of the time.
link |
This is really the biggest downside.
link |
And then, well, Python's slow, but that's not related to it.
link |
Well, maybe it's kind of related to it, so it's lack of.
link |
So what's, what's in your toolbox these days?
link |
I need to move to something else.
link |
My adventure into dependently typed languages,
link |
I love these languages.
link |
They just have like syntax from the 80s.
link |
What do you think about JavaScript?
link |
ES6, like the modern, or TypeScript?
link |
the whole ecosystem is unbelievably confusing.
link |
NPM updates a package from 0.2.2 to 0.2.5,
link |
and that breaks your Babel linter,
link |
which translates your ES5 into ES6,
link |
which doesn't run on, so.
link |
Why do I have to compile my JavaScript again, huh?
link |
It may be the future, though.
link |
You think about, I mean,
link |
I've embraced JavaScript recently,
link |
just because, just like I've continually embraced PHP,
link |
it seems that these worst possible languages
link |
live on for the longest, like cockroaches never die.
link |
Well, it's in the browser, and it's fast.
link |
It's in the browser, and compute might stay,
link |
become, you know, the browser.
link |
It's unclear what the role of the browser is
link |
in terms of distributed computation in the future, so.
link |
JavaScript is definitely here to stay.
link |
It's interesting if autonomous vehicles
link |
will run on JavaScript one day.
link |
I mean, you have to consider these possibilities.
link |
Well, all our debug tools are JavaScript.
link |
We actually just open sourced them.
link |
We have a tool, Explorer,
link |
which you can annotate your disengagements,
link |
and we have a tool, Cabana,
link |
which lets you analyze the can traffic from the car.
link |
So basically, anytime you're visualizing something
link |
about the log, you're using JavaScript.
link |
Well, the web is the best UI toolkit by far, so.
link |
And then, you know what?
link |
You're coding in JavaScript.
link |
We have a React guy.
link |
Let's get into it.
link |
So let's talk autonomous vehicles.
link |
You founded Comma AI.
link |
Let's, at a high level,
link |
how did you get into the world of vehicle automation?
link |
Can you also just, for people who don't know,
link |
tell the story of Comma AI?
link |
So I was working at this AI startup,
link |
and a friend approached me,
link |
and he's like, dude, I don't know where this is going,
link |
but the coolest applied AI problem today
link |
is self driving cars.
link |
I'm like, well, absolutely.
link |
You want to meet with Elon Musk,
link |
and he's looking for somebody to build a vision system
link |
This is when they were still on AP1.
link |
They were still using Mobileye.
link |
Elon, back then, was looking for a replacement,
link |
and he brought me in,
link |
and we talked about a contract
link |
where I would deliver something
link |
that meets Mobileye level performance.
link |
I would get paid $12 million if I could deliver it tomorrow,
link |
and I would lose $1 million
link |
for every month I didn't deliver.
link |
So I was like, okay, this is a great deal.
link |
This is a super exciting challenge.
link |
Even if it takes me 10 months,
link |
Maybe I can finish up in five.
link |
Maybe I don't finish it at all,
link |
and I get paid nothing,
link |
and I can still work for 12 months for free.
link |
So maybe just take a pause on that.
link |
I'm also curious about this
link |
because I've been working in robotics for a long time,
link |
and I'm curious to see a person like you
link |
just step in and sort of somewhat naive,
link |
but brilliant, right?
link |
So that's the best place to be
link |
because you basically full steam take on a problem.
link |
How confident, how, from that time,
link |
because you know a lot more now,
link |
at that time, how hard do you think it is
link |
to solve all of autonomous driving?
link |
I remember I suggested to Elon in the meeting
link |
putting a GPU behind each camera
link |
to keep the compute local.
link |
This is an incredibly stupid idea.
link |
I leave the meeting 10 minutes later,
link |
and I'm like, I could have spent a little bit of time
link |
thinking about this problem before I went in.
link |
Why is it a stupid idea?
link |
Oh, just send all your cameras to one big GPU.
link |
You're much better off doing that.
link |
You said behind every camera have a GPU.
link |
Every camera have a small GPU.
link |
I was like, oh, I'll put the first few layers
link |
of my comms there.
link |
Ugh, why'd I say that?
link |
It's possible, but it's a bad idea.
link |
It's not obviously a bad idea.
link |
Pretty obviously bad,
link |
but whether it's actually a bad idea or not,
link |
I left that meeting with Elon, beating myself up.
link |
I'm like, why'd I say something stupid?
link |
Yeah, you haven't at least thought through
link |
every aspect of it, yeah.
link |
He's very sharp too.
link |
Usually in life, I get away with saying stupid things
link |
and then kind of course,
link |
oh, right away he called me out about it.
link |
And usually in life, I get away with saying stupid things
link |
and then a lot of times people don't even notice
link |
and I'll correct it and bring the conversation back.
link |
But with Elon, it was like, nope, okay, well.
link |
That's not at all why the contract fell through.
link |
I was much more prepared the second time I met him.
link |
Yeah, but in general, how hard did you think it is?
link |
Like 12 months is a tough timeline.
link |
Oh, I just thought I'd clone Mobileye IQ3.
link |
I didn't think I'd solve level five self driving
link |
So the goal there was to do lane keeping, good lane keeping.
link |
I saw, my friend showed me the outputs from a Mobileye
link |
and the outputs from a Mobileye was just basically
link |
two lanes at a position of a lead car.
link |
I'm like, I can gather a data set and train this net
link |
in weeks and I did.
link |
Well, first time I tried the implementation of Mobileye
link |
in a Tesla, I was really surprised how good it is.
link |
It's going incredibly good.
link |
Cause I thought it's just cause I've done a lot
link |
of computer vision, I thought it'd be a lot harder
link |
to create a system that that's stable.
link |
So I was personally surprised, you know,
link |
Cause I was kind of skeptical before trying it.
link |
Cause I thought it would go in and out a lot more.
link |
It would get disengaged a lot more and it's pretty robust.
link |
So what, how hard is the problem when you tackled it?
link |
So I think AP1 was great.
link |
Like Elon talked about disengagements on the 405 down in LA
link |
with like the lane marks are kind of faded
link |
and the Mobileye system would drop out.
link |
Like I had something up and working that I would say
link |
was like the same quality in three months.
link |
Same quality, but how do you know?
link |
You say stuff like that confidently, but you can't,
link |
and I love it, but the question is you can't,
link |
you're kind of going by feel cause you test it out.
link |
Absolutely, absolutely.
link |
Like I would take, I borrowed my friend's Tesla.
link |
I would take AP1 out for a drive
link |
and then I would take my system out for a drive.
link |
And it seems reasonably like the same.
link |
So the 405, how hard is it to create something
link |
that could actually be a product that's deployed?
link |
I mean, I've read an article where Elon,
link |
this respondent said something about you saying
link |
that to build autopilot is more complicated
link |
than a single George Hodge level job.
link |
How hard is that job to create something
link |
that would work across globally?
link |
Why don't think globally is the challenge?
link |
But Elon followed that up by saying
link |
it's gonna take two years in a company of 10 people.
link |
And here I am four years later with a company of 12 people.
link |
And I think we still have another two to go.
link |
Two years, so yeah.
link |
So what do you think about how Tesla is progressing
link |
with autopilot of V2, V3?
link |
I think we've kept pace with them pretty well.
link |
I think navigate and autopilot is terrible.
link |
We had some demo features internally of the same stuff
link |
and we would test it.
link |
And I'm like, I'm not shipping this
link |
even as like open source software to people.
link |
Why do you think it's terrible?
link |
Consumer Reports does a great job of describing it.
link |
Like when it makes a lane change,
link |
it does it worse than a human.
link |
You shouldn't ship things like autopilot, open pilot.
link |
They lane keep better than a human.
link |
If you turn it on for a stretch of a highway,
link |
like an hour long, it's never gonna touch a lane line.
link |
Human will touch probably a lane line twice.
link |
You just inspired me.
link |
I don't know if you're grounded in data on that.
link |
I read your paper.
link |
Okay, but that's interesting.
link |
I wonder actually how often we touch lane lines
link |
in general, like a little bit,
link |
I could answer that question pretty easily
link |
with the common data set.
link |
Yeah, I'm curious.
link |
I've never answered it.
link |
I just, two is like my personal.
link |
That's interesting.
link |
Because every time you touch a lane,
link |
that's a source of a little bit of stress
link |
and kind of lane keeping is removing that stress.
link |
That's ultimately the biggest value add honestly
link |
is just removing the stress of having to stay in lane.
link |
And I think honestly, I don't think people fully realize,
link |
first of all, that that's a big value add,
link |
but also that that's all it is.
link |
And that not only, I find it a huge value add.
link |
I drove down when we moved to San Diego,
link |
I drove down in a enterprise rental car and I missed it.
link |
So I missed having the system so much.
link |
It's so much more tiring to drive without it.
link |
It is that lane centering.
link |
That's the key feature.
link |
And in a way, it's the only feature
link |
that actually adds value to people's lives
link |
in autonomous vehicles today.
link |
Waymo does not add value to people's lives.
link |
It's a more expensive, slower Uber.
link |
Maybe someday it'll be this big cliff where it adds value,
link |
but I don't usually believe it.
link |
It is fascinating.
link |
I haven't talked to, this is good.
link |
Cause I haven't, I have intuitively,
link |
but I think we're making it explicit now.
link |
I actually believe that really good lane keeping
link |
is a reason to buy a car.
link |
Will be a reason to buy a car and it's a huge value add.
link |
I've never, until we just started talking about it,
link |
I haven't really quite realized it.
link |
That I've felt with Elon's chase of level four
link |
is not the correct chase.
link |
It was on, cause you should just say Tesla has the best
link |
as if from a Tesla perspective, say,
link |
Tesla has the best lane keeping.
link |
Comma AI should say, Comma AI is the best lane keeping.
link |
You have to do the longitudinal as well.
link |
You can't just lane keep.
link |
You have to do ACC,
link |
but ACC is much more forgiving than lane keep,
link |
especially on the highway.
link |
By the way, are you Comma AI's camera only, correct?
link |
No, we use the radar.
link |
From the car, you're able to get the, okay.
link |
We can do a camera only now.
link |
It's gotten to the point,
link |
but we leave the radar there as like a, it's fusion now.
link |
Okay, so let's maybe talk through some of the system specs
link |
What's the hardware side of what you're providing?
link |
What's the capabilities on the software side
link |
with OpenPilot and so on?
link |
So OpenPilot, as the box that we sell, that it runs on,
link |
it's a phone in a plastic case.
link |
It's nothing special.
link |
We sell it without the software.
link |
So you buy the phone, it's just easy.
link |
It'll be easy set up, but it's sold with no software.
link |
OpenPilot right now is about to be 0.6.
link |
When it gets to 1.0,
link |
I think we'll be ready for a consumer product.
link |
We're not gonna add any new features.
link |
We're just gonna make the lane keeping really, really good.
link |
So what do we have right now?
link |
It's a Snapdragon 820.
link |
It's a Sony IMX 298 forward facing camera.
link |
Driver monitoring camera,
link |
it's just a selfie camera on the phone.
link |
And a CAN transceiver,
link |
maybe there's a little thing called PANDAS.
link |
And they talk over USB to the phone
link |
and then they have three CAN buses
link |
that they talk to the car.
link |
One of those CAN buses is the radar CAN bus.
link |
One of them is the main car CAN bus
link |
and the other one is the proxy camera CAN bus.
link |
We leave the existing camera in place
link |
so we don't turn AEB off.
link |
Right now, we still turn AEB off
link |
if you're using our longitudinal,
link |
but we're gonna fix that before 1.0.
link |
And it's CAN both ways.
link |
So how are you able to control vehicles?
link |
the vehicles that we work with
link |
already have a lane keeping assist system.
link |
So lane keeping assist can mean a huge variety of things.
link |
It can mean it will apply a small torque to the wheel
link |
after you've already crossed a lane line by a foot,
link |
which is the system in the older Toyotas
link |
versus like, I think Tesla still calls it
link |
lane keeping assist,
link |
where it'll keep you perfectly
link |
in the center of the lane on the highway.
link |
You can control, like with the joystick, the car.
link |
So these cars already have the capability of drive by wire.
link |
So is it trivial to convert a car that it operates with?
link |
OpenPILOT is able to control the steering?
link |
Oh, a new car or a car that we,
link |
so we have support now for 45 different makes of cars.
link |
What are the cars in general?
link |
Mostly Hondas and Toyotas.
link |
We support almost every Honda and Toyota made this year.
link |
And then a bunch of GMs, a bunch of Subarus,
link |
a bunch of Chevys.
link |
It doesn't have to be like a Prius,
link |
it could be a Corolla as well.
link |
Oh, the 2020 Corolla is the best car with OpenPILOT.
link |
The actuator has less lag than the older Corolla.
link |
I think I started watching a video with your,
link |
I mean, the way you make videos is awesome.
link |
You're just literally at the dealership streaming.
link |
Yeah, I had my friend on the phone,
link |
I'm like, bro, you wanna stream for an hour?
link |
Yeah, and basically, like if stuff goes a little wrong,
link |
you're just like, you just go with it.
link |
That's so beautiful and it's so in contrast
link |
to the way other companies
link |
would put together a video like that.
link |
Kind of why I like to do it like that.
link |
I mean, if you become super rich one day and successful,
link |
I hope you keep it that way
link |
because I think that's actually what people love,
link |
that kind of genuine.
link |
Oh, it's all that has value to me.
link |
Money has no, if I sell out to like make money,
link |
I sold out, it doesn't matter.
link |
I don't want a yacht.
link |
And I think Tesla's actually has a small inkling
link |
of that as well with Autonomy Day.
link |
They did reveal more than, I mean, of course,
link |
there's marketing communications, you could tell,
link |
but it's more than most companies would reveal,
link |
which is, I hope they go towards that direction more,
link |
other companies, GM, Ford.
link |
Oh, Tesla's gonna win level five.
link |
So let's talk about it.
link |
You think, you're focused on level two currently?
link |
We're gonna be one to two years behind Tesla
link |
getting to level five.
link |
We're Android, right?
link |
I'm just saying, once Tesla gets it,
link |
we're one to two years behind.
link |
I'm not making any timeline on when Tesla's
link |
gonna get it. That's right.
link |
You did, that was brilliant.
link |
I'm sorry, Tesla investors,
link |
if you think you're gonna have an autonomous
link |
Robo Taxi fleet by the end of the year.
link |
I'll bet against that.
link |
So what do you think about this?
link |
The most level four companies
link |
are kind of just doing their usual safety driver,
link |
doing full autonomy kind of testing.
link |
And then Tesla does basically trying to go
link |
from lane keeping to full autonomy.
link |
What do you think about that approach?
link |
How successful would it be?
link |
It's a ton better approach.
link |
Because Tesla is gathering data on a scale
link |
that none of them are.
link |
They're putting real users behind the wheel of the cars.
link |
It's, I think, the only strategy that works.
link |
Well, so there's a few components to Tesla approach
link |
that's more than just the incrementalists.
link |
What you spoke with is the ones, the software,
link |
so over the air software updates.
link |
I mean Waymo crews have those too.
link |
Those differentiate from the automakers.
link |
Right, no lane keeping systems have,
link |
no cars with lane keeping system have that except Tesla.
link |
And the other one is the data, the other direction,
link |
which is the ability to query the data.
link |
I don't think they're actually collecting
link |
as much data as people think,
link |
but the ability to turn on collection and turn it off.
link |
So I'm both in the robotics world
link |
and the psychology human factors world.
link |
Many people believe that level two autonomy is problematic
link |
because of the human factor.
link |
Like the more the task is automated,
link |
the more there's a vigilance decrement.
link |
You start to fall asleep.
link |
You start to become complacent,
link |
start texting more and so on.
link |
Do you worry about that?
link |
Cause if we're talking about transition from lane keeping
link |
to full autonomy, if you're spending 80% of the time,
link |
not supervising the machine,
link |
do you worry about what that means
link |
to the safety of the drivers?
link |
One, we don't consider open pilot to be 1.0
link |
until we have 100% driver monitoring.
link |
You can cheat right now, our driver monitoring system.
link |
There's a few ways to cheat it.
link |
They're pretty obvious.
link |
We're working on making that better.
link |
Before we ship a consumer product that can drive cars,
link |
I want to make sure that I have driver monitoring
link |
that you can't cheat.
link |
What's like a successful driver monitoring system look like?
link |
Is it all about just keeping your eyes on the road?
link |
Well, a few things.
link |
So that's what we went with at first for driver monitoring.
link |
I'm checking, I'm actually looking at
link |
where your head is looking.
link |
The camera's not that high resolution.
link |
Eyes are a little bit hard to get.
link |
Well, head is this big.
link |
And actually a lot of it, just psychology wise,
link |
to have that monitor constantly there,
link |
it reminds you that you have to be paying attention.
link |
But we want to go further.
link |
We just hired someone full time
link |
to come on to do the driver monitoring.
link |
I want to detect phone in frame
link |
and I want to make sure you're not sleeping.
link |
How much does the camera see of the body?
link |
This one, not enough.
link |
The next one, everything.
link |
Well, it's interesting, Fisheye,
link |
because we're doing just data collection, not real time.
link |
But Fisheye is a beautiful,
link |
being able to capture the body.
link |
And the smartphone is really like the biggest problem.
link |
I can show you one of the pictures from our new system.
link |
Awesome, so you're basically saying
link |
the driver monitoring will be the answer to that.
link |
I think the other point
link |
that you raised in your paper is good as well.
link |
You're not asking a human to supervise a machine
link |
without giving them the,
link |
they can take over at any time.
link |
Our safety model, you can take over.
link |
We disengage on both the gas or the brake.
link |
We don't disengage on steering.
link |
I don't feel you have to.
link |
But we disengage on gas or brake.
link |
So it's very easy for you to take over
link |
and it's very easy for you to reengage.
link |
That switching should be super cheap.
link |
The cars that require,
link |
even autopilot requires a double press.
link |
That's almost, I see, I don't like that.
link |
And then the cancel, to cancel in autopilot,
link |
you either have to press cancel,
link |
which no one knows what that is, so they press the brake.
link |
But a lot of times you don't actually want
link |
to press the brake.
link |
You want to press the gas.
link |
So you should cancel on gas.
link |
Or wiggle the steering wheel, which is bad as well.
link |
Wow, that's brilliant.
link |
I haven't heard anyone articulate that point.
link |
Oh, this is all I think about.
link |
It's the, because I think,
link |
I think actually Tesla has done a better job
link |
than most automakers at making that frictionless.
link |
But you just described that it could be even better.
link |
I love Super Cruise as an experience once it's engaged.
link |
I don't know if you've used it,
link |
but getting the thing to try to engage.
link |
Yeah, I've used the, I've driven Super Cruise a lot.
link |
So what's your thoughts on the Super Cruise system?
link |
You disengage Super Cruise and it falls back to ACC.
link |
So my car's like still accelerating.
link |
Otherwise, when you actually have Super Cruise engaged
link |
on the highway, it is phenomenal.
link |
We bought that Cadillac.
link |
But we bought it just to like experience this.
link |
And I wanted everyone in the office to be like,
link |
this is what we're striving to build.
link |
GM pioneering with the driver monitoring.
link |
You like their driver monitoring system?
link |
If there's a sun shining back here, it'll be blind to you.
link |
But overall, mostly, yeah.
link |
That's so cool that you know all this stuff.
link |
I don't often talk to people that,
link |
because it's such a rare car, unfortunately, currently.
link |
We bought one explicitly for this.
link |
We lost like 25K in the deprecation,
link |
but I feel it's worth it.
link |
I was very pleasantly surprised that GM system
link |
was so innovative and really wasn't advertised much,
link |
wasn't talked about much.
link |
And I was nervous that it would die,
link |
that it would disappear.
link |
Well, they put it on the wrong car.
link |
They should have put it on the Bolt
link |
and not some weird Cadillac that nobody bought.
link |
I think that's gonna be into,
link |
they're saying at least it's gonna be
link |
into their entire fleet.
link |
So what do you think about,
link |
as long as we're on the driver monitoring,
link |
what do you think about Elon Musk's claim
link |
that driver monitoring is not needed?
link |
Normally, I love his claims.
link |
That one is stupid.
link |
That one is stupid.
link |
And, you know, he's not gonna have his level five fleet
link |
by the end of the year.
link |
Hopefully he's like, okay, I was wrong.
link |
I'm gonna add driver monitoring.
link |
Because when these systems get to the point
link |
that they're only messing up once every thousand miles,
link |
you absolutely need driver monitoring.
link |
So let me play, cause I agree with you,
link |
but let me play devil's advocate.
link |
One possibility is that without driver monitoring,
link |
people are able to monitor, self regulate,
link |
monitor themselves.
link |
You know, that, so your idea is.
link |
You've seen all the people sleeping in Teslas?
link |
Yeah, well, I'm a little skeptical
link |
of all the people sleeping in Teslas
link |
because I've stopped paying attention to that kind of stuff
link |
because I want to see real data.
link |
It's too much glorified.
link |
It doesn't feel scientific to me.
link |
So I want to know how many people are really sleeping
link |
in Teslas versus sleeping.
link |
I was driving here sleep deprived in a car
link |
with no automation.
link |
I was falling asleep.
link |
I agree that it's hypey.
link |
It's just like, you know what?
link |
If you want to put driver monitoring,
link |
I rented a, my last autopilot experience
link |
was I rented a model three in March and drove it around.
link |
The wheel thing is annoying.
link |
And the reason the wheel thing is annoying,
link |
we use the wheel thing as well,
link |
but we don't disengage on wheel.
link |
For Tesla, you have to touch the wheel just enough
link |
to trigger the torque sensor, to tell it that you're there,
link |
but not enough as to disengage it,
link |
which don't use it for two things.
link |
Don't disengage on wheel.
link |
You don't have to.
link |
That whole experience, wow, beautifully put.
link |
All of those elements,
link |
even if you don't have driver monitoring,
link |
that whole experience needs to be better.
link |
Driver monitoring, I think would make,
link |
I mean, I think Super Cruise is a better experience
link |
once it's engaged over autopilot.
link |
I think Super Cruise is a transition
link |
to engagement and disengagement are significantly worse.
link |
Well, there's a tricky thing,
link |
because if I were to criticize Super Cruise is,
link |
it's a little too crude.
link |
And I think like six seconds or something,
link |
if you look off road, it'll start warning you.
link |
It's some ridiculously long period of time.
link |
I think it's basically, it's a binary.
link |
It should be adaptive.
link |
Yeah, it needs to learn more about you.
link |
It needs to communicate what it sees about you more.
link |
Tesla shows what it sees about the external world.
link |
It would be nice if Super Cruise would tell us
link |
what it sees about the internal world.
link |
It's even worse than that.
link |
You press the button to engage
link |
and it just says Super Cruise unavailable.
link |
Yeah, that transparency is good.
link |
We've renamed the driver monitoring packet to driver state.
link |
We have car state packet, which has the state of the car.
link |
And you have driver state packet,
link |
which has the state of the driver.
link |
Estimate their BAC.
link |
Blood alcohol content.
link |
You think that's possible with computer vision?
link |
To me, it's an open question.
link |
I haven't looked into it too much.
link |
Actually, I quite seriously looked at the literature.
link |
It's not obvious to me that from the eyes and so on,
link |
You might need stuff from the car as well.
link |
You might need how they're controlling the car, right?
link |
And that's fundamentally at the end of the day,
link |
what you care about.
link |
But I think, especially when people are really drunk,
link |
they're not controlling the car nearly as smoothly
link |
as they would look at them walking, right?
link |
The car is like an extension of the body.
link |
So I think you could totally detect.
link |
And if you could fix people who are drunk, distracted,
link |
asleep, if you fix those three.
link |
Yeah, that's huge.
link |
So what are the current limitations of open pilot?
link |
What are the main problems that still need to be solved?
link |
We're hopefully fixing a few of them in 06.
link |
We're not as good as autopilot at stop cars.
link |
So if you're coming up to a red light at 55,
link |
so it's the radar stopped car problem, which
link |
is responsible for two autopilot accidents,
link |
it's hard to differentiate a stopped car from a signpost.
link |
Yeah, a static object.
link |
So you have to fuse.
link |
You have to do this visually.
link |
There's no way from the radar data to tell the difference.
link |
Maybe you can make a map, but I don't really
link |
believe in mapping at all anymore.
link |
Wait, wait, wait, what, you don't believe in mapping?
link |
So you basically, the open pilot solution
link |
is saying react to the environment as you see it,
link |
just like human beings do.
link |
And then eventually, when you want
link |
to do navigate on open pilot, I'll
link |
train the net to look at ways.
link |
I'll run ways in the background, I'll
link |
train a confident way.
link |
Are you using GPS at all?
link |
We use it to ground truth.
link |
We use it to very carefully ground truth the paths.
link |
We have a stack which can recover relative to 10
link |
centimeters over one minute.
link |
And then we use that to ground truth exactly where
link |
the car went in that local part of the environment,
link |
but it's all local.
link |
How are you testing in general, just for yourself,
link |
like experiments and stuff?
link |
Where are you located?
link |
So you basically drive around there, collect some data,
link |
and watch the performance?
link |
We have a simulator now.
link |
And we have, our simulator is really cool.
link |
Our simulator is not, it's not like a Unity based simulator.
link |
Our simulator lets us load in real state.
link |
We can load in a drive and simulate
link |
what the system would have done on the historical data.
link |
Right now we're only using it for testing,
link |
but as soon as we start using it for training, that's it.
link |
That's all that matters.
link |
What's your feeling about the real world versus simulation?
link |
Do you like simulation for training,
link |
if this moves to training?
link |
So we have to distinguish two types of simulators, right?
link |
There's a simulator that is completely fake.
link |
I could get my car to drive around in GTA.
link |
I feel that this kind of simulator is useless.
link |
You're never, there's so many.
link |
My analogy here is like, OK, fine.
link |
You're not solving the computer vision problem,
link |
but you're solving the computer graphics problem.
link |
And you don't think you can get very far by creating
link |
ultra realistic graphics?
link |
No, because you can create ultra realistic graphics
link |
of the road, now create ultra realistic behavioral models
link |
of the other cars.
link |
Oh, well, I'll just use myself driving.
link |
You need actual human behavior, because that's
link |
what you're trying to learn.
link |
Driving does not have a spec.
link |
The definition of driving is what humans do when they drive.
link |
Whatever Waymo does, I don't think it's driving.
link |
Well, I think actually Waymo and others,
link |
if there's any use for reinforcement learning,
link |
I've seen it used quite well.
link |
I study pedestrians a lot, too, is
link |
try to train models from real data of how pedestrians move,
link |
and try to use reinforcement learning models to make
link |
pedestrians move in human like ways.
link |
By that point, you've already gone so many layers,
link |
you detected a pedestrian?
link |
Did you hand code the feature vector of their state?
link |
Did you guys learn anything from computer vision
link |
before deep learning?
link |
Well, OK, I feel like this is.
link |
So perception to you is the sticking point.
link |
I mean, what's the hardest part of the stack here?
link |
There is no human understandable feature vector separating
link |
perception and planning.
link |
That's the best way I can put that.
link |
There is no, so it's all together,
link |
and it's a joint problem.
link |
So you can take localization.
link |
Localization and planning, there is
link |
a human understandable feature vector between these two
link |
I mean, OK, so I have like three degrees position,
link |
three degrees orientation, and those derivatives,
link |
maybe those second derivatives.
link |
That's human understandable.
link |
Between perception and planning, so like Waymo
link |
has a perception stack and then a planner.
link |
And one of the things Waymo does right
link |
is they have a simulator that can separate those two.
link |
They can like replay their perception data
link |
and test their system, which is what
link |
I'm talking about about like the two
link |
different kinds of simulators.
link |
There's the kind that can work on real data,
link |
and there's the kind that can't work on real data.
link |
Now, the problem is that I don't think you can hand code
link |
a feature vector, right?
link |
Like you have some list of like, oh, here's
link |
my list of cars in the scenes.
link |
Here's my list of pedestrians in the scene.
link |
This isn't what humans are doing.
link |
What are humans doing?
link |
And you're saying that's too difficult to hand engineer.
link |
I'm saying that there is no state vector given a perfect.
link |
I could give you the best team of engineers in the world
link |
to build a perception system and the best team
link |
to build a planner.
link |
All you have to do is define the state vector
link |
that separates those two.
link |
I'm missing the state vector that separates those two.
link |
So what is the output of your perception system?
link |
Output of the perception system, it's, OK, well,
link |
there's several ways to do it.
link |
One is the SLAM components localization.
link |
The other is drivable area, drivable space.
link |
Drivable space, yeah.
link |
And then there's the different objects in the scene.
link |
And different objects in the scene over time,
link |
maybe, to give you input to then try
link |
to start modeling the trajectories of those objects.
link |
I can give you a concrete example
link |
of something you missed.
link |
So say there's a bush in the scene.
link |
Humans understand that when they see this bush
link |
that there may or may not be a car behind that bush.
link |
Drivable area and a list of objects does not include that.
link |
Humans are doing this constantly at the simplest intersections.
link |
So now you have to talk about occluded area.
link |
But even that, what do you mean by occluded?
link |
OK, so I can't see it.
link |
Well, if it's the other side of a house, I don't care.
link |
What's the likelihood that there's
link |
a car in that occluded area?
link |
And if you say, OK, we'll add that,
link |
I can come up with 10 more examples that you can't add.
link |
Certainly, occluded area would be something
link |
that Simulator would have because it's
link |
simulating the entire occlusion is part of it.
link |
Occlusion is part of a vision stack.
link |
But what I'm saying is if you have a hand engineered,
link |
if your perception system output can
link |
be written in a spec document, it is incomplete.
link |
Yeah, I mean, certainly, it's hard to argue with that
link |
because in the end, that's going to be true.
link |
Yeah, and I'll tell you what the output of our perception
link |
It's a 1,024 dimensional vector, trained by neural net.
link |
Oh, you know that.
link |
No, it's 1,024 dimensions of who knows what.
link |
Because it's operating on real data.
link |
And that's the perception.
link |
That's the perception state.
link |
Think about an autoencoder for faces.
link |
If you have an autoencoder for faces and you say
link |
it has 256 dimensions in the middle,
link |
and I'm taking a face over here and projecting it
link |
to a face over here.
link |
Can you hand label all 256 of those dimensions?
link |
Well, no, but those have to generate automatically.
link |
But even if you tried to do it by hand,
link |
could you come up with a spec between your encoder
link |
No, because it wasn't designed, but there.
link |
No, no, no, but if you could design it.
link |
If you could design a face reconstructor system,
link |
could you come up with a spec?
link |
No, but I think we're missing here a little bit.
link |
I think you're just being very poetic about expressing
link |
a fundamental problem of simulators,
link |
that they're going to be missing so much that the feature
link |
vector will just look fundamentally different
link |
in the simulated world than the real world.
link |
I'm not making a claim about simulators.
link |
I'm making a claim about the spec division
link |
between perception and planning, even in your system.
link |
If you're trying to build a car that drives,
link |
if you're trying to hand code the output of your perception
link |
system, like saying, here's a list of all the cars
link |
in the scene, here's a list of all the people,
link |
here's a list of the occluded areas,
link |
here's a vector of drivable areas, it's insufficient.
link |
And if you start to believe that,
link |
you realize that what Waymo and Cruz are doing is impossible.
link |
Currently, what we're doing is the perception problem
link |
is converting the scene into a chessboard.
link |
And then you reason some basic reasoning
link |
around that chessboard.
link |
And you're saying that really, there's a lot missing there.
link |
First of all, why are we talking about this?
link |
Because isn't this a full autonomy?
link |
Is this something you think about?
link |
Oh, I want to win self driving cars.
link |
So your definition of win includes?
link |
Level four or five.
link |
I don't think level four is a real thing.
link |
I want to build the AlphaGo of driving.
link |
So AlphaGo is really end to end.
link |
Is, yeah, it's end to end.
link |
And do you think this whole problem,
link |
is that also kind of what you're getting at
link |
with the perception and the planning?
link |
Is that this whole problem, the right way to do it
link |
is really to learn the entire thing.
link |
I'll argue that not only is it the right way,
link |
it's the only way that's going to exceed human performance.
link |
It's certainly true for Go.
link |
Everyone who tried to hand code Go things
link |
built human inferior things.
link |
And then someone came along and wrote some 10,000 line thing
link |
that doesn't know anything about Go that beat everybody.
link |
It's 10,000 lines.
link |
True, in that sense, the open question then
link |
that maybe I can ask you is driving is much harder than Go.
link |
The open question is how much harder?
link |
So how, because I think the Elon Musk approach here
link |
with planning and perception is similar
link |
to what you're describing,
link |
which is really turning into not some kind of modular thing,
link |
but really do formulate it as a learning problem
link |
and solve the learning problem with scale.
link |
So how many years, put one is how many years
link |
would it take to solve this problem
link |
or just how hard is this freaking problem?
link |
Well, the cool thing is I think there's a lot of value
link |
that we can deliver along the way.
link |
I think that you can build lane keeping assist actually
link |
plus adaptive cruise control, plus, okay, looking at ways,
link |
extends to like all of driving.
link |
Yeah, most of driving, right?
link |
Oh, your adaptive cruise control treats red lights
link |
So let's jump around.
link |
You mentioned that you didn't like navigate an autopilot.
link |
What advice, how would you make it better?
link |
Do you think as a feature that if it's done really well,
link |
it's a good feature?
link |
I think that it's too reliant on like hand coded hacks
link |
for like, how does navigate an autopilot do a lane change?
link |
It actually does the same lane change every time
link |
and it feels mechanical.
link |
Humans do different lane changes.
link |
Humans sometime will do a slow one,
link |
sometimes do a fast one.
link |
Navigate an autopilot, at least every time I use it,
link |
it is the identical lane change.
link |
I mean, this is a fundamental thing actually
link |
is the braking and then accelerating
link |
something that's still, Tesla probably does it better
link |
than most cars, but it still doesn't do a great job
link |
of creating a comfortable natural experience.
link |
And navigate an autopilot is just lane changes
link |
and extension of that.
link |
So how do you learn to do a natural lane change?
link |
So we have it and I can talk about how it works.
link |
So I feel that we have the solution for lateral.
link |
We don't yet have the solution for longitudinal.
link |
There's a few reasons longitudinal is harder than lateral.
link |
The lane change component,
link |
the way that we train on it very simply
link |
is like our model has an input
link |
for whether it's doing a lane change or not.
link |
And then when we train the end to end model,
link |
we hand label all the lane changes,
link |
cause you have to.
link |
I've struggled a long time about not wanting to do that,
link |
but I think you have to.
link |
Or the training data.
link |
For the training data, right?
link |
Oh, we actually, we have an automatic ground truther
link |
which automatically labels all the lane changes.
link |
Was that possible?
link |
To automatically label the lane changes?
link |
Yeah, detect the lane, I see when it crosses it, right?
link |
And I don't have to get that high percent accuracy,
link |
but it's like 95, good enough.
link |
Now I set the bit when it's doing the lane change
link |
in the end to end learning.
link |
And then I set it to zero when it's not doing a lane change.
link |
So now if I wanted to do a lane change at test time,
link |
I just put the bit to a one and it'll do a lane change.
link |
Yeah, but so if you look at the space of lane change,
link |
you know, some percentage, not a hundred percent
link |
that we make as humans is not a pleasant experience
link |
cause we messed some part of it up.
link |
It's nerve wracking to change the look,
link |
you have to see, it has to accelerate.
link |
How do we label the ones that are natural and feel good?
link |
You know, that's the, cause that's your ultimate criticism.
link |
The current navigate and autopilot
link |
just doesn't feel good.
link |
Well, the current navigate and autopilot
link |
is a hand coded policy written by an engineer in a room
link |
who probably went out and tested it a few times on the 280.
link |
Probably a more, a better version of that, but yes.
link |
That's how we would have written it at Comma AI.
link |
Maybe Tesla did, Tesla, they tested it in the end.
link |
That might've been two engineers.
link |
Two engineers, yeah.
link |
No, but so if you learn the lane change,
link |
if you learn how to do a lane change from data,
link |
just like you have a label that says lane change
link |
and then you put it in when you want it
link |
to do the lane change,
link |
it'll automatically do the lane change
link |
that's appropriate for the situation.
link |
Now, to get at the problem of some humans
link |
do bad lane changes,
link |
we haven't worked too much on this problem yet.
link |
It's not that much of a problem in practice.
link |
My theory is that all good drivers are good in the same way
link |
and all bad drivers are bad in different ways.
link |
And we've seen some data to back this up.
link |
Well, beautifully put.
link |
So you just basically, if that's true hypothesis,
link |
then your task is to discover the good drivers.
link |
The good drivers stand out because they're in one cluster
link |
and the bad drivers are scattered all over the place
link |
and your net learns the cluster.
link |
Yeah, that's, so you just learn from the good drivers
link |
and they're easy to cluster.
link |
In fact, we learned from all of them
link |
and the net automatically learns the policy
link |
that's like the majority,
link |
but we'll eventually probably have to filter them out.
link |
If that theory is true, I hope it's true
link |
because the counter theory is there is many clusters,
link |
maybe arbitrarily many clusters of good drivers.
link |
Because if there's one cluster of good drivers,
link |
you can at least discover a set of policies.
link |
You can learn a set of policies,
link |
which would be good universally.
link |
That would be a nice, that would be nice if it's true.
link |
And you're saying that there is some evidence that.
link |
Let's say lane changes can be clustered into four clusters.
link |
There's this finite level of.
link |
I would argue that all four of those are good clusters.
link |
All the things that are random are noise and probably bad.
link |
And which one of the four you pick,
link |
or maybe it's 10 or maybe it's 20.
link |
You can learn that.
link |
It's context dependent.
link |
It depends on the scene.
link |
And the hope is it's not too dependent on the driver.
link |
Yeah. The hope is that it all washes out.
link |
The hope is that there's, that the distribution's not bimodal.
link |
The hope is that it's a nice Gaussian.
link |
So what advice would you give to Tesla,
link |
how to fix, how to improve navigating autopilot?
link |
That's the lessons that you've learned from Comm AI?
link |
The only real advice I would give to Tesla
link |
is please put driver monitoring in your cars.
link |
With respect to improving it?
link |
You can't do that anymore.
link |
I decided to interrupt, but you know,
link |
there's a practical nature of many of hundreds of thousands
link |
of cars being produced that don't have
link |
a good driver facing camera.
link |
The Model 3 has a selfie cam.
link |
Is it not good enough?
link |
Did they not put IR LEDs for night?
link |
That's a good question.
link |
But I do know that it's fisheye
link |
and it's relatively low resolution.
link |
So it's really not designed.
link |
It wasn't designed for driver monitoring.
link |
You can hope that you can kind of scrape up
link |
and have something from it.
link |
But why didn't they put it in today?
link |
Every time I've heard Karpathy talk about the problem
link |
and talking about like software 2.0
link |
and how the machine learning is gobbling up everything,
link |
I think this is absolutely the right strategy.
link |
I think that he didn't write navigate on autopilot.
link |
I think somebody else did
link |
and kind of hacked it on top of that stuff.
link |
I think when Karpathy says, wait a second,
link |
why did we hand code this lane change policy
link |
with all these magic numbers?
link |
We're gonna learn it from data.
link |
They already know what to do there.
link |
Well, that's Andrei's job
link |
is to turn everything into a learning problem
link |
and collect a huge amount of data.
link |
The reality is though,
link |
not every problem can be turned into a learning problem
link |
in the short term.
link |
In the end, everything will be a learning problem.
link |
The reality is like if you wanna build L5 vehicles today,
link |
it will likely involve no learning.
link |
And that's the reality is,
link |
so at which point does learning start?
link |
It's the crutch statement that LiDAR is a crutch.
link |
At which point will learning
link |
get up to part of human performance?
link |
It's over human performance on ImageNet,
link |
classification, on driving, it's a question still.
link |
I'll say this, I'm here to play for 10 years.
link |
I'm not here to try to,
link |
I'm here to play for 10 years and make money along the way.
link |
I'm not here to try to promise people
link |
that I'm gonna have my L5 taxi network
link |
up and working in two years.
link |
Do you think that was a mistake?
link |
What do you think was the motivation behind saying that?
link |
Other companies are also promising L5 vehicles
link |
with very different approaches in 2020, 2021, 2022.
link |
If anybody would like to bet me
link |
that those things do not pan out, I will bet you.
link |
Even money, even money, I'll bet you as much as you want.
link |
So are you worried about what's going to happen?
link |
Cause you're not in full agreement on that.
link |
What's going to happen when 2022, 21 come around
link |
and nobody has fleets of autonomous vehicles?
link |
Well, you can look at the history.
link |
If you go back five years ago,
link |
they were all promised by 2018 and 2017.
link |
But they weren't that strong of promises.
link |
I mean, Ford really declared pretty,
link |
I think not many have declared as like definitively
link |
as they have now these dates.
link |
Well, okay, so let's separate L4 and L5.
link |
Do I think that it's possible for Waymo to continue to kind
link |
of like hack on their system
link |
until it gets to level four in Chandler, Arizona?
link |
When there's no safety driver?
link |
Chandler, Arizona?
link |
By, sorry, which year are we talking about?
link |
Oh, I even think that's possible by like 2020, 2021.
link |
But level four, Chandler, Arizona,
link |
not level five, New York City.
link |
Level four, meaning some very defined streets,
link |
it works out really well.
link |
Very defined streets.
link |
And then practically these streets are pretty empty.
link |
If most of the streets are covered in Waymo's,
link |
Waymo can kind of change the definition of what driving is.
link |
If your self driving network
link |
is the majority of cars in an area,
link |
they only need to be safe with respect to each other
link |
and all the humans will need to learn to adapt to them.
link |
Now go drive in downtown New York.
link |
Well, yeah, that's.
link |
I mean, already you can talk about autonomy
link |
and like on farms, it already works great
link |
because you can really just follow the GPS line.
link |
So what does success look like for common AI?
link |
What are the milestones?
link |
Like where you can sit back with some champagne
link |
and say, we did it, boys and girls?
link |
Well, it's never over.
link |
You must drink champagne and celebrate.
link |
So what is a good, what are some wins?
link |
A big milestone that we're hoping for
link |
by mid next year is profitability of the company.
link |
And we're gonna have to revisit the idea
link |
of selling a consumer product,
link |
but it's not gonna be like the comma one.
link |
When we do it, it's gonna be perfect.
link |
Open pilot has gotten so much better in the last two years.
link |
We're gonna have a few features.
link |
We're gonna have a hundred percent driver monitoring.
link |
We're gonna disable no safety features in the car.
link |
Actually, I think it'd be really cool
link |
what we're doing right now.
link |
Our project this week is we're analyzing the data set
link |
and looking for all the AEB triggers
link |
from the manufacturer systems.
link |
We have better data set on that than the manufacturers.
link |
How much, just how many,
link |
does Toyota have 10 million miles of real world driving
link |
to know how many times their AEB triggered?
link |
So let me give you, cause you asked, right?
link |
Cause I work with a lot of automakers
link |
and one possible source of money for you,
link |
which I'll be excited to see you take on
link |
is basically selling the data.
link |
So, which is something that most people,
link |
and not selling in a way where here, here at Automaker,
link |
but creating, we've done this actually at MIT,
link |
not for money purposes,
link |
but you could do it for significant money purposes
link |
and make the world a better place by creating a consortia
link |
where automakers would pay in
link |
and then they get to have free access to the data.
link |
And I think a lot of people are really hungry for that
link |
and would pay significant amount of money for it.
link |
Here's the problem with that.
link |
I like this idea all in theory.
link |
It'd be very easy for me to give them access to my servers
link |
and we already have all open source tools
link |
to access this data.
link |
It's in a great format.
link |
We have a great pipeline,
link |
but they're gonna put me in the room
link |
with some business development guy.
link |
And I'm gonna have to talk to this guy
link |
and he's not gonna know most of the words I'm saying.
link |
I'm not willing to tolerate that.
link |
Okay, Mick Jagger.
link |
No, no, no, no, no.
link |
I think I agree with you.
link |
I'm the same way, but you just tell them the terms
link |
and there's no discussion needed.
link |
If I could just tell them the terms,
link |
and like, all right, who wants access to my data?
link |
I will sell it to you for, let's say,
link |
you want a subscription?
link |
I'll sell to you for 100K a month.
link |
I'll give you access to this data subscription.
link |
Yeah, I think that's kind of fair.
link |
Came up with that number off the top of my head.
link |
If somebody sends me like a three line email
link |
where it's like, we would like to pay 100K a month
link |
to get access to your data.
link |
We would agree to like reasonable privacy terms
link |
of the people who are in the data set.
link |
I would be happy to do it,
link |
but that's not going to be the email.
link |
The email is going to be, hey,
link |
do you have some time in the next month
link |
where we can sit down and we can,
link |
I don't have time for that.
link |
We're moving too fast.
link |
You could politely respond to that email,
link |
but not saying, I don't have any time for your bullshit.
link |
You say, oh, well, unfortunately these are the terms.
link |
And so this is, we try to,
link |
we brought the cost down for you
link |
in order to minimize the friction and communication.
link |
Here's the, whatever it is,
link |
one, two million dollars a year and you have access.
link |
And it's not like I get that email from like,
link |
but okay, am I going to reach out?
link |
Am I going to hire a business development person
link |
who's going to reach out to the automakers?
link |
If they reached into me, I'm not going to ignore the email.
link |
I'll come back with something like,
link |
yeah, if you're willing to pay 100K a month
link |
for access to the data, I'm happy to set that up.
link |
That's worth my engineering time.
link |
That's actually quite insightful of you.
link |
Probably because many of the automakers
link |
are quite a bit old school,
link |
there will be a need to reach out and they want it,
link |
but there'll need to be some communication.
link |
Mobileye circa 2015 had the lowest R&D spend
link |
of any chip maker, like per, per,
link |
and you look at all the people who work for them
link |
and it's all business development people
link |
because the car companies are impossible to work with.
link |
So you're, you have no patience for that
link |
and you're, you're legit Android, huh?
link |
I have something to do, right?
link |
Like, like it's not like, it's not like,
link |
I don't, like, I don't mean to like be a dick
link |
and say like, I don't have patience for that,
link |
but it's like that stuff doesn't help us
link |
with our goal of winning self driving cars.
link |
If I want money in the short term,
link |
if I showed off like the actual,
link |
like the learning tech that we have,
link |
it's, it's somewhat sad.
link |
Like it's years and years ahead of everybody else's.
link |
Not to, maybe not Tesla's.
link |
I think Tesla has some more stuff to us actually.
link |
I think Tesla has similar stuff,
link |
but when you compare it to like
link |
what the Toyota Research Institute has,
link |
you're not even close to what we have.
link |
But I also can't, I have to take your comments.
link |
I intuitively believe you,
link |
but I have to take it with a grain of salt
link |
because I mean, you are an inspiration
link |
because you basically don't care about a lot of things
link |
that other companies care about.
link |
You don't try to bullshit in a sense,
link |
like make up stuff.
link |
So to drive up valuation, you're really very real
link |
and you're trying to solve the problem
link |
and admire that a lot.
link |
What I don't necessarily fully can't trust you on,
link |
with all due respect, is how good it is, right?
link |
I can only, but I also know how bad others are.
link |
I'll say two things about, trust but verify, right?
link |
I'll say two things about that.
link |
One is try, get in a 2020 Corolla
link |
and try open pilot 0.6 when it comes out next month.
link |
I think already you'll look at this
link |
and you'll be like, this is already really good.
link |
And then I could be doing that all with hand labelers
link |
and all with like the same approach that Mobileye uses.
link |
When we release a model that no longer has the lanes in it,
link |
that only outputs a path,
link |
then think about how we did that machine learning
link |
and then right away when you see,
link |
and that's gonna be an open pilot,
link |
that's gonna be an open pilot before 1.0.
link |
When you see that model,
link |
you'll know that everything I'm saying is true
link |
because how else did I get that model?
link |
You know what I'm saying is true about the simulator.
link |
Yeah, yeah, this is super exciting, that's super exciting.
link |
But like, you know, I listened to your talk with Kyle
link |
and Kyle was originally building the aftermarket system
link |
and he gave up on it because of technical challenges,
link |
because of the fact that he's gonna have to support
link |
20 to 50 cars, we support 45,
link |
because what is he gonna do
link |
when the manufacturer ABS system triggers?
link |
We have alerts and warnings to deal with all of that
link |
And how is he going to formally verify it?
link |
Well, I got 10 million miles of data,
link |
it's probably better,
link |
it's probably better verified than the spec.
link |
Yeah, I'm glad you're here talking to me.
link |
This is, I'll remember this day,
link |
because it's interesting.
link |
If you look at Kyle's from cruise,
link |
I'm sure they have a large number
link |
of business development folks
link |
and you work with, he's working with GM,
link |
you could work with Argo AI, working with Ford.
link |
It's interesting because chances that you fail,
link |
business wise, like bankrupt, are pretty high.
link |
And yet, it's the Android model,
link |
is you're actually taking on the problem.
link |
So that's really inspiring, I mean.
link |
Well, I have a long term way for Comma to make money too.
link |
And one of the nice things
link |
when you really take on the problem,
link |
which is my hope for Autopilot, for example,
link |
is things you don't expect,
link |
ways to make money or create value
link |
that you don't expect will pop up.
link |
Oh, I've known how to do it since kind of,
link |
2017 is the first time I said it.
link |
Which part, to know how to do which part?
link |
Our long term plan is to be a car insurance company.
link |
Insurance, yeah, I love it, yep, yep.
link |
I make driving twice as safe.
link |
Not only that, I have the best data
link |
such to know who statistically is the safest drivers.
link |
And oh, oh, we see you, we see you driving unsafely,
link |
we're not gonna insure you.
link |
And that causes a bifurcation in the market
link |
because the only people who can't get Comma insurance
link |
are the bad drivers, Geico can insure them,
link |
their premiums are crazy high,
link |
our premiums are crazy low.
link |
We'll win car insurance, take over that whole market.
link |
If we win, if we win.
link |
But that's what I'm saying,
link |
how do you turn Comma into a $10 billion company?
link |
So you, Elon Musk, who else?
link |
Who else is thinking like this and working like this
link |
Who are the competitors?
link |
Are there people seriously,
link |
I don't think anyone that I'm aware of
link |
is seriously taking on lane keeping,
link |
like where it's a huge business
link |
that turns eventually into full autonomy
link |
that then creates, yeah, like that creates other businesses
link |
on top of it and so on.
link |
Thinks insurance, thinks all kinds of ideas like that.
link |
Do you know anyone else thinking like this?
link |
That's interesting.
link |
I mean, my sense is everybody turns to that
link |
in like four or five years.
link |
Like Ford, once the autonomy doesn't fall through.
link |
By the way, he paved the way for all of us.
link |
It's the iOS, true.
link |
I would not be doing Comma AI today
link |
if it was not for those conversations with Elon.
link |
And if it were not for him saying like,
link |
I think he said like,
link |
well, obviously we're not gonna use LiDAR,
link |
we use cameras, humans use cameras.
link |
So what do you think about that?
link |
How important is LiDAR?
link |
Everybody else on L5 is using LiDAR.
link |
What are your thoughts on his provocative statement
link |
that LiDAR is a crutch?
link |
See, sometimes he'll say dumb things,
link |
like the driver monitoring thing,
link |
but sometimes he'll say absolutely, completely,
link |
100% obviously true things.
link |
Of course LiDAR is a crutch.
link |
It's not even a good crutch.
link |
You're not even using it.
link |
Oh, they're using it for localization.
link |
Which isn't good in the first place.
link |
If you have to localize your car to centimeters
link |
in order to drive, like that's not driving.
link |
Currently not doing much machine learning
link |
I thought for LiDAR data.
link |
Meaning like to help you in the task of,
link |
general task of perception.
link |
The main goal of those LiDARs on those cars
link |
I think is actually localization more than perception.
link |
Or at least that's what they use them for.
link |
Yeah, that's true.
link |
If you want to localize to centimeters,
link |
you can't use GPS.
link |
The fanciest GPS in the world can't do it.
link |
Especially if you're under tree cover and stuff.
link |
With LiDAR you can do this pretty easily.
link |
So you really, they're not taking on,
link |
I mean in some research they're using it for perception,
link |
but, and they're certainly not, which is sad,
link |
they're not fusing it well with vision.
link |
They do use it for perception.
link |
I'm not saying they don't use it for perception,
link |
but the thing that, they have vision based
link |
and radar based perception systems as well.
link |
You could remove the LiDAR and keep around
link |
a lot of the dynamic object perception.
link |
You want to get centimeter accurate localization?
link |
Good luck doing that with anything else.
link |
So what should Cruz, Waymo do?
link |
Like what would be your advice to them now?
link |
I mean Waymo is actually,
link |
they're, I mean they're doing, they're serious.
link |
Waymo out of the ball of them are quite
link |
so serious about the long game.
link |
If L5 is a lot, requires 50 years,
link |
I think Waymo will be the only one left standing at the end
link |
with the, given the financial backing that they have.
link |
Buku Google bucks.
link |
I'll say nice things about both Waymo and Cruz.
link |
Waymo is by far the furthest along with technology.
link |
Waymo has a three to five year lead on all the competitors.
link |
If that, if the Waymo looking stack works,
link |
maybe three year lead.
link |
If the Waymo looking stack works,
link |
they have a three year lead.
link |
Now I argue that Waymo has spent too much money
link |
to recapitalize, to gain back their losses
link |
in those three years.
link |
Also self driving cars have no network effect like that.
link |
Uber has a network effect.
link |
You have a market, you have drivers and you have riders.
link |
Self driving cars, you have capital and you have riders.
link |
There's no network effect.
link |
If I want to blanket a new city in self driving cars,
link |
I buy the off the shelf Chinese knockoff self driving cars
link |
and I buy enough of them in the city.
link |
I can't do that with drivers.
link |
And that's why Uber has a first mover advantage
link |
that no self driving car company will.
link |
Can you disentangle that a little bit?
link |
Uber, you're not talking about Uber,
link |
the autonomous vehicle Uber.
link |
You're talking about the Uber car, the, yeah.
link |
I open for business in Austin, Texas, let's say.
link |
I need to attract both sides of the market.
link |
I need to both get drivers on my platform
link |
and riders on my platform.
link |
And I need to keep them both sufficiently happy, right?
link |
Riders aren't gonna use it
link |
if it takes more than five minutes for an Uber to show up.
link |
Drivers aren't gonna use it
link |
if they have to sit around all day and there's no riders.
link |
So you have to carefully balance a market.
link |
And whenever you have to carefully balance a market,
link |
there's a great first mover advantage
link |
because there's a switching cost for everybody, right?
link |
The drivers and the riders
link |
would have to switch at the same time.
link |
Let's even say that, you know, let's say a Luber shows up
link |
and Luber somehow, you know, agrees to do things
link |
at a bigger, you know, we're just gonna,
link |
we've done it more efficiently, right?
link |
Luber is only takes 5% of a cut
link |
instead of the 10% that Uber takes.
link |
No one is gonna switch
link |
because the switching cost is higher than that 5%.
link |
So you actually can, in markets like that,
link |
you have a first mover advantage.
link |
Autonomous vehicles of the level five variety
link |
have no first mover advantage.
link |
If the technology becomes commoditized,
link |
say I wanna go to a new city, look at the scooters.
link |
It's gonna look a lot more like scooters.
link |
Every person with a checkbook
link |
can blanket a city in scooters.
link |
And that's why you have 10 different scooter companies.
link |
Which one's gonna win?
link |
It's a race to the bottom.
link |
It's a terrible market to be in
link |
because there's no market for scooters.
link |
And the scooters don't get a say
link |
in whether they wanna be bought and deployed to a city
link |
We're gonna entice the scooters
link |
with subsidies and deals and.
link |
So whenever you have to invest that capital,
link |
It doesn't come back.
link |
That can't be your main criticism of the Waymo approach.
link |
Oh, I'm saying even if it does technically work.
link |
Even if it does technically work, that's a problem.
link |
I don't know if I were to say,
link |
I would say you're already there.
link |
I haven't even thought about that,
link |
but I would say the bigger challenge
link |
is the technical approach.
link |
So Waymo's cruises.
link |
And not just the technical approach,
link |
but of creating value.
link |
I still don't understand how you beat Uber,
link |
the human driven cars.
link |
In terms of financially,
link |
it doesn't make sense to me
link |
that people wanna get in an autonomous vehicle.
link |
I don't understand how you make money.
link |
In the longterm, yes.
link |
Like real longterm.
link |
But it just feels like there's too much
link |
capital investment needed.
link |
Oh, and they're gonna be worse than Ubers
link |
because they're gonna stop for every little thing,
link |
I'll say a nice thing about cruise.
link |
That was my nice thing about Waymo.
link |
They're three years ahead.
link |
Wait, what was the nice?
link |
Oh, because they're three.
link |
They're three years technically ahead of everybody.
link |
Their tech stack is great.
link |
My nice thing about cruise is GM buying them
link |
was a great move for GM.
link |
GM bought an insurance policy against Waymo.
link |
They put, cruise is three years behind Waymo.
link |
That means Google will get a monopoly on the technology
link |
for at most three years.
link |
And if technology works,
link |
so you might not even be right about the three years,
link |
Cruise actually might not be that far behind.
link |
I don't know how much Waymo has waffled around
link |
or how much of it actually is just that long tail.
link |
If that's the best you could say in terms of nice things,
link |
that's more of a nice thing for GM
link |
that that's the smart insurance policy.
link |
It's a smart insurance policy.
link |
I mean, I think that's how,
link |
I can't see cruise working out any other.
link |
For cruise to leapfrog Waymo would really surprise me.
link |
Yeah, so let's talk about
link |
the underlying assumption of everything is.
link |
We're not gonna leapfrog Tesla.
link |
Tesla would have to seriously mess up for us
link |
Okay, so the way you leapfrog, right?
link |
Is you come up with an idea
link |
or you take a direction perhaps secretly
link |
that the other people aren't taking.
link |
And so the cruise, Waymo,
link |
I don't know Aurora, Zooks is the same stack as well.
link |
They're all the same code base even.
link |
And they're all the same DARPA Urban Challenge code base.
link |
So the question is,
link |
do you think there's a room for brilliance and innovation
link |
that will change everything?
link |
Like say, okay, so I'll give you examples.
link |
It could be if revolution and mapping, for example,
link |
that allow you to map things,
link |
do HD maps of the whole world,
link |
all weather conditions somehow really well,
link |
or revolution and simulation
link |
to where the all the way you said before becomes incorrect.
link |
That kind of thing.
link |
Any room for breakthrough innovation?
link |
What I said before about,
link |
oh, they actually get the whole thing.
link |
Well, I'll say this about,
link |
we divide driving into three problems
link |
and I actually haven't solved the third yet,
link |
but I haven't had you how to do it.
link |
So there's the static.
link |
The static driving problem is assuming
link |
you are the only car on the road, right?
link |
And this problem can be solved 100%
link |
with mapping and localization.
link |
This is why farms work the way they do.
link |
If all you have to deal with is the static problem
link |
and you can statically schedule your machines, right?
link |
It's the same as like statically scheduling processes.
link |
You can statically schedule your tractors
link |
to never hit each other on their paths, right?
link |
Cause they know the speed they go at.
link |
So that's the static driving problem.
link |
Maps only helps you with the static driving problem.
link |
Yeah, the question about static driving,
link |
you've just made it sound like it's really easy.
link |
Static driving is really easy.
link |
How, well, cause the whole drifting out of lane,
link |
when Tesla drifts out of lane,
link |
it's failing on the fundamental static driving problem.
link |
Tesla is drifting out of lane?
link |
The static driving problem is not easy for the world.
link |
The static driving problem is easy for one route.
link |
One route and one weather condition
link |
with one state of lane markings
link |
and like no deterioration, no cracks in the road.
link |
No, I'm assuming you have a perfect localizer.
link |
So that's solved for the weather condition
link |
and the lane marking condition.
link |
But that's the problem is,
link |
how do you have a perfect localizer?
link |
Perfect localizers are not that hard to build.
link |
Okay, come on now, with LIDAR?
link |
Oh, with LIDAR, okay.
link |
With LIDAR, yeah, but you use LIDAR, right?
link |
Like use LIDAR, build a perfect localizer.
link |
Building a perfect localizer without LIDAR,
link |
it's gonna be hard.
link |
You can get 10 centimeters without LIDAR,
link |
you can get one centimeter with LIDAR.
link |
I'm not even concerned about the one or 10 centimeters.
link |
I'm concerned if every once in a while,
link |
you're just way off.
link |
Yeah, so this is why you have to carefully make sure
link |
you're always tracking your position.
link |
You wanna use LIDAR camera fusion,
link |
but you can get the reliability of that system
link |
up to 100,000 miles,
link |
and then you write some fallback condition
link |
where it's not that bad if you're way off, right?
link |
I think that you can get it to the point,
link |
it's like ASLD that you're never in a case
link |
where you're way off and you don't know it.
link |
Yeah, okay, so this is brilliant.
link |
So that's the static. Static.
link |
We can, especially with LIDAR and good HG maps,
link |
you can solve that problem. Easy.
link |
No, I just disagree with your word easy.
link |
The static problem's so easy.
link |
It's very typical for you to say something is easy.
link |
It's not as challenging as the other ones, okay.
link |
Well, okay, maybe it's obvious how to solve it.
link |
The third one's the hardest.
link |
And a lot of people don't even think about the third one
link |
and even see it as different from the second one.
link |
So the second one is dynamic.
link |
The second one is like, say there's an obvious example
link |
is like a car stopped at a red light, right?
link |
You can't have that car in your map
link |
because you don't know whether that car
link |
is gonna be there or not.
link |
So you have to detect that car in real time
link |
and then you have to do the appropriate action, right?
link |
Also, that car is not a fixed object.
link |
That car may move and you have to predict
link |
what that car will do, right?
link |
So this is the dynamic problem.
link |
So you have to deal with this.
link |
This involves, again, like you're gonna need models
link |
of other people's behavior.
link |
Are you including in that,
link |
I don't wanna step on the third one.
link |
But are you including in that your influence on people?
link |
Ah, that's the third one.
link |
That's the third one.
link |
We call it the counterfactual.
link |
I just talked to Judea Pearl
link |
who's obsessed with counterfactuals.
link |
And the counterfactual.
link |
Oh yeah, yeah, I read his books.
link |
So the static and the dynamic
link |
Our approach right now for lateral
link |
will scale completely to the static and dynamic.
link |
The counterfactual, the only way I have to do it yet,
link |
the thing that I wanna do once we have all of these cars
link |
is I wanna do reinforcement learning on the world.
link |
I'm always gonna turn the exploiter up to max.
link |
I'm not gonna have them explore.
link |
But the only real way to get at the counterfactual
link |
is to do reinforcement learning
link |
because the other agents are humans.
link |
So that's fascinating that you break it down like that.
link |
I agree completely.
link |
I've spent my life thinking about this problem.
link |
And part of it, because you're slightly insane,
link |
Just the last four years.
link |
You have some nonzero percent of your brain
link |
has a madman in it, which is good.
link |
That's a really good feature.
link |
But there's a safety component to it
link |
that I think sort of with counterfactuals and so on
link |
that would just freak people out.
link |
How do you even start to think about just in general?
link |
I mean, you've had some friction with NHTSA and so on.
link |
I am frankly exhausted by safety engineers.
link |
The prioritization on safety over innovation
link |
to a degree where it kills, in my view,
link |
kills safety in the long term.
link |
So the counterfactual thing,
link |
they just actually exploring this world
link |
of how do you interact with dynamic objects and so on.
link |
How do you think about safety?
link |
You can do reinforcement learning without ever exploring.
link |
And I said that, so you can think about your,
link |
in reinforcement learning,
link |
it's usually called a temperature parameter.
link |
And your temperature parameter
link |
is how often you deviate from the argmax.
link |
I could always set that to zero and still learn.
link |
And I feel that you'd always want that set to zero
link |
on your actual system.
link |
But the problem is you first don't know very much.
link |
And so you're going to make mistakes.
link |
So the learning, the exploration happens through mistakes.
link |
So the consequences of a mistake.
link |
Open pilot and autopilot are making mistakes left and right.
link |
We have 700 daily active users,
link |
a thousand weekly active users.
link |
Open pilot makes tens of thousands of mistakes a week.
link |
These mistakes have zero consequences.
link |
These mistakes are,
link |
oh, I wanted to take this exit and it went straight.
link |
So I'm just going to carefully touch the wheel.
link |
The humans catch them.
link |
The humans catch them.
link |
And the human disengagement is labeling
link |
that reinforcement learning
link |
in a completely consequence free way.
link |
So driver monitoring is the way you ensure they keep.
link |
They keep paying attention.
link |
How is your messaging?
link |
Say I gave you a billion dollars,
link |
you would be scaling it now.
link |
Oh, I couldn't scale it with any amount of money.
link |
I'd raise money if I could, if I had a way to scale it.
link |
Yeah, you're now not focused on scale.
link |
I don't know how to do,
link |
oh, like I guess I could sell it to more people,
link |
but I want to make the system better.
link |
And I don't know how to, I mean.
link |
But what's the messaging here?
link |
I got a chance to talk to Elon and he basically said
link |
that the human factor doesn't matter.
link |
You know, the human doesn't matter
link |
because the system will perform,
link |
there'll be sort of a, sorry to use the term,
link |
but like a singular,
link |
like a point where it gets just much better.
link |
And so the human, it won't really matter.
link |
But it seems like that human catching the system
link |
when it gets into trouble is like the thing
link |
which will make something like reinforcement learning work.
link |
So how do you think messaging for Tesla,
link |
for you should change,
link |
for the industry in general should change?
link |
I think our messaging is pretty clear.
link |
At least like our messaging wasn't that clear
link |
in the beginning and I do kind of fault myself for that.
link |
We are proud right now to be a level two system.
link |
We are proud to be level two.
link |
If we talk about level four,
link |
it's not with the current hardware.
link |
It's not gonna be just a magical OTA upgrade.
link |
It's gonna be new hardware.
link |
It's gonna be very carefully thought out.
link |
Right now, we are proud to be level two
link |
and we have a rigorous safety model.
link |
I mean, not like, okay, rigorous, who knows what that means,
link |
but we at least have a safety model
link |
and we make it explicit as in safety.md in OpenPilot.
link |
And it says, seriously though, safety.md.
link |
This is brilliant, this is so Android.
link |
Well, this is the safety model
link |
and I like to have conversations like,
link |
sometimes people will come to you and they're like,
link |
your system's not safe.
link |
Okay, have you read my safety docs?
link |
Would you like to have an intelligent conversation
link |
And the answer is always no.
link |
They just like scream about, it runs Python.
link |
So you're saying that because Python's not real time,
link |
Python not being real time never causes disengagements.
link |
Disengagements are caused by, the model is QM.
link |
But safety.md says the following,
link |
first and foremost,
link |
the driver must be paying attention at all times.
link |
I still consider the software to be alpha software
link |
until we can actually enforce that statement,
link |
but I feel it's very well communicated to our users.
link |
One is the user must be able to easily take control
link |
of the vehicle at all times.
link |
So if you step on the gas or brake with OpenPilot,
link |
it gives full manual control back to the user
link |
or press the cancel button.
link |
Step two, the car will never react so quickly,
link |
we define so quickly to be about one second,
link |
that you can't react in time.
link |
And we do this by enforcing torque limits,
link |
braking limits and acceleration limits.
link |
So we have like our torque limits way lower than Tesla's.
link |
This is another potential.
link |
If I could tweak Autopilot,
link |
I would lower their torque limit
link |
and I would add driver monitoring.
link |
Because Autopilot can jerk the wheel hard.
link |
We limit, and all this code is open source, readable.
link |
And I believe now it's all Misra C compliant.
link |
Misra is like the automotive coding standard.
link |
At first, I've come to respect.
link |
I've been reading like the standards lately
link |
and I've come to respect them.
link |
They're actually written by very smart people.
link |
Yeah, they're brilliant people actually.
link |
They have a lot of experience.
link |
They're sometimes a little too cautious,
link |
but in this case, it pays off.
link |
Misra is written by like computer scientists.
link |
And you can tell by the language they use.
link |
You can tell by the language they use,
link |
they talk about like whether certain conditions in Misra
link |
are decidable or undecidable.
link |
And you mean like the halting problem?
link |
And yes, all right, you've earned my respect.
link |
I will read carefully what you have to say
link |
and we wanna make our code compliant with that.
link |
All right, so you're proud level two, beautiful.
link |
So you were the founder and I think CEO of Kama AI,
link |
then you were the head of research.
link |
What the heck are you now?
link |
What's your connection to Kama AI?
link |
I'm the president, but I'm one of those
link |
like unelected presidents of like a small dictatorship
link |
country, not one of those like elected presidents.
link |
Oh, so you're like Putin when he was like the,
link |
So there's a, what's the governance structure?
link |
What's the future of Kama AI?
link |
I mean, yeah, it's a business.
link |
Do you want, are you just focused on getting things
link |
right now, making some small amount of money in the meantime
link |
and then when it works, it works and you scale.
link |
Our burn rate is about 200K a month
link |
and our revenue is about 100K a month.
link |
So we need to forex our revenue,
link |
but we haven't like tried very hard at that yet.
link |
And the revenue is basically selling stuff online.
link |
Yeah, we sell stuff shop.kama.ai.
link |
Is there other, well, okay,
link |
so you'll have to figure out the revenue.
link |
That's our only, see, but to me,
link |
that's like respectable revenues.
link |
We make it by selling products to consumers
link |
who are honest and transparent about what they are.
link |
Most actually level four companies, right?
link |
Cause you could easily start blowing up like smoke,
link |
like overselling the hype and feeding into,
link |
getting some fundraisers.
link |
Oh, you're the guy, you're a genius
link |
because you hacked the iPhone.
link |
Oh, I hate that, I hate that.
link |
Yeah, well, I can trade my social capital for more money.
link |
I did it once, I almost regret it doing it the first time.
link |
Well, on a small tangent,
link |
what's your, you seem to not like fame
link |
and yet you're also drawn to fame.
link |
Where are you on that currently?
link |
Have you had some introspection, some soul searching?
link |
I've come to a pretty stable position on that.
link |
Like after the first time,
link |
I realized that I don't want attention from the masses.
link |
I want attention from people who I respect.
link |
Who do you respect?
link |
I can give a list of people.
link |
So are these like Elon Musk type characters?
link |
Yeah, well, actually, you know what?
link |
I'll make it more broad than that.
link |
I won't make it about a person, I respect skill.
link |
I respect people who have skills, right?
link |
And I would like to like be, I'm not gonna say famous,
link |
but be like known among more people who have like real skills.
link |
Who in cars do you think have skill, not do you respect?
link |
Oh, Kyle Vogt has skill.
link |
A lot of people at Waymo have skill and I respect them.
link |
I respect them as engineers.
link |
Like I can think, I mean,
link |
I think about all the times in my life
link |
where I've been like dead set on approaches
link |
and they turn out to be wrong.
link |
So, I mean, this might, I might be wrong.
link |
I accept that there's a decent chance that I'm wrong.
link |
And actually, I mean,
link |
having talked to Chris Hermsons, Sterling Anderson,
link |
those guys, I mean, I deeply respect Chris.
link |
I just admire the guy.
link |
When you drive a car through the desert
link |
when everybody thinks it's impossible, that's legit.
link |
And then I also really respect the people
link |
who are like writing the infrastructure of the world,
link |
like the Linus Torvalds and the Chris Lattiners.
link |
They were doing the real work.
link |
I know, they're doing the real work.
link |
This, having talked to Chris,
link |
like Chris Lattiners, you realize,
link |
especially when they're humble,
link |
it's like you realize, oh, you guys,
link |
we're just using your,
link |
All the hard work that you did.
link |
Yeah, that's incredible.
link |
What do you think, Mr. Anthony Lewandowski,
link |
what do you, he's another mad genius.
link |
Sharp guy, oh yeah.
link |
What, do you think he might long term become a competitor?
link |
Well, so I think that he has the other right approach.
link |
I think that right now there's two right approaches.
link |
One is what we're doing, and one is what he's doing.
link |
Can you describe, I think it's called Pronto AI.
link |
He started a new thing.
link |
Do you know what the approach is?
link |
I actually don't know.
link |
Embark is also doing the same sort of thing.
link |
The idea is almost that you want to,
link |
so if you're, I can't partner with Honda and Toyota.
link |
Honda and Toyota are like 400,000 person companies.
link |
It's not even a company at that point.
link |
I don't think of it like, I don't personify it.
link |
I think of it like an object,
link |
but a trucker drives for a fleet,
link |
maybe that has like, some truckers are independent.
link |
Some truckers drive for fleets with a hundred trucks.
link |
There are tons of independent trucking companies out there.
link |
Start a trucking company and drive your costs down
link |
or figure out how to drive down the cost of trucking.
link |
Another company that I really respect is Nato.
link |
Actually, I respect their business model.
link |
Nato sells a driver monitoring camera
link |
and they sell it to fleet owners.
link |
If I owned a fleet of cars
link |
and I could pay 40 bucks a month to monitor my employees,
link |
this is gonna, it like reduces accidents 18%.
link |
It's so like that, in the space,
link |
that is like the business model that I like most respect.
link |
Cause they're creating value today.
link |
Yeah, which is a, that's a huge one.
link |
How do we create value today with some of this?
link |
And the lane keeping thing is huge.
link |
And it sounds like you're creeping in
link |
or full steam ahead on the driver monitoring too,
link |
which I think actually where the short term value,
link |
if you can get it right.
link |
I still, I'm not a huge fan of the statement
link |
that everything has to have driver monitoring.
link |
I agree with that completely,
link |
but that statement usually misses the point
link |
that to get the experience of it right is not trivial.
link |
Oh no, not at all.
link |
In fact, like, so right now we have,
link |
I think the timeout depends on speed of the car,
link |
but we want to depend on like the scene state.
link |
If you're on like an empty highway,
link |
it's very different if you don't pay attention
link |
than if like you're like coming up to a traffic light.
link |
And longterm, it should probably learn from the driver
link |
because that's to do, I watched a lot of video.
link |
We've built a smartphone detector
link |
just to analyze how people are using smartphones
link |
and people are using it very differently.
link |
It's a texting styles.
link |
We haven't watched nearly enough of the videos.
link |
We haven't, I got millions of miles
link |
of people driving cars.
link |
In this moment, I spent a large fraction of my time
link |
just watching videos because it's never fails to learn.
link |
Like it never, I've never failed
link |
from a video watching session
link |
to learn something I didn't know before.
link |
In fact, I usually like when I eat lunch,
link |
I'll sit, especially when the weather is good
link |
and just watch pedestrians with an eye to understand
link |
like from a computer vision eye,
link |
just to see can this model, can you predict,
link |
what are the decisions made?
link |
And there's so many things that we don't understand.
link |
This is what I mean about the state vector.
link |
Yeah, it's, I'm trying to always think like,
link |
cause I'm understanding in my human brain,
link |
how do we convert that into,
link |
how hard is the learning problem here?
link |
I guess is the fundamental question.
link |
So something that's from a hacking perspective,
link |
this is always comes up, especially with folks.
link |
Well, first the most popular question
link |
is the trolley problem, right?
link |
So that's not a sort of a serious problem.
link |
There are some ethical questions I think that arise.
link |
Maybe you wanna, do you think there's any ethical,
link |
serious ethical questions?
link |
We have a solution to the trolley problem at Comm.ai.
link |
Well, so there is actually an alert in our code,
link |
ethical dilemma detected.
link |
It's not triggered yet.
link |
We don't know how yet to detect the ethical dilemmas,
link |
but we're a level two system.
link |
So we're going to disengage
link |
and leave that decision to the human.
link |
You're such a troll.
link |
No, but the trolley problem deserves to be trolled.
link |
Yeah, that's a beautiful answer actually.
link |
I know, I gave it to someone who was like,
link |
sometimes people will ask,
link |
like you asked about the trolley problem,
link |
like you can have a kind of discussion about it.
link |
Like you get someone who's like really like earnest about it
link |
because it's the kind of thing where,
link |
if you ask a bunch of people in an office,
link |
whether we should use a SQL stack or a no SQL stack,
link |
if they're not that technical, they have no opinion.
link |
But if you ask them what color they want to paint the office,
link |
everyone has an opinion on that.
link |
And that's why the trolley problem is...
link |
I mean, that's a beautiful answer.
link |
Yeah, we're able to detect the problem
link |
and we're able to pass it on to the human.
link |
Wow, I've never heard anyone say it.
link |
This is your nice escape route.
link |
I'm proud level two.
link |
So the other thing that people have some concern about
link |
with AI in general is hacking.
link |
So how hard is it, do you think,
link |
to hack an autonomous vehicle,
link |
either through physical access
link |
or through the more sort of popular now,
link |
these adversarial examples on the sensors?
link |
Okay, the adversarial examples one.
link |
You want to see some adversarial examples
link |
that affect humans, right?
link |
Oh, well, there used to be a stop sign here,
link |
but I put a black bag over the stop sign
link |
and then people ran it, adversarial, right?
link |
Like there's tons of human adversarial examples too.
link |
The question in general about like security,
link |
if you saw something just came out today
link |
and like there are always such hypey headlines
link |
about like how navigate on autopilot
link |
was fooled by a GPS spoof to take an exit.
link |
At least that's all they could do was take an exit.
link |
If your car is relying on GPS
link |
in order to have a safe driving policy,
link |
you're doing something wrong.
link |
If you're relying,
link |
and this is why V2V is such a terrible idea.
link |
V2V now relies on both parties getting communication right.
link |
This is not even, so I think of safety,
link |
security is like a special case of safety, right?
link |
Safety is like we put a little, you know,
link |
piece of caution tape around the hole
link |
so that people won't walk into it by accident.
link |
Security is like put a 10 foot fence around the hole
link |
so you actually physically cannot climb into it
link |
with barbed wire on the top and stuff, right?
link |
So like if you're designing systems that are like unreliable,
link |
they're definitely not secure.
link |
Your car should always do something safe
link |
using its local sensors.
link |
And then the local sensor should be hardwired.
link |
And then could somebody hack into your CAN bus
link |
and turn your steering wheel on your brakes?
link |
Yes, but they could do it before common AI too, so.
link |
Let's think out of the box on some things.
link |
So do you think teleoperation has a role in any of this?
link |
So remotely stepping in and controlling the cars?
link |
No, I think that if the safety operation by design
link |
requires a constant link to the cars,
link |
I think it doesn't work.
link |
So that's the same argument you're using for V2I, V2V?
link |
Well, there's a lot of non safety critical stuff
link |
you can do with V2I.
link |
I like V2I, I like V2I way more than V2V.
link |
Because V2I is already like,
link |
I already have internet in the car, right?
link |
There's a lot of great stuff you can do with V2I.
link |
Like for example, you can, well, I already have V2I,
link |
Waze is V2I, right?
link |
Waze can route me around traffic jams.
link |
That's a great example of V2I.
link |
And then, okay, the car automatically talks
link |
to that same service, like it works.
link |
So it's improving the experience,
link |
but it's not a fundamental fallback for safety.
link |
No, if any of your things that require wireless communication
link |
are more than QM, like have an ASL rating, it shouldn't be.
link |
You previously said that life is work
link |
and that you don't do anything to relax.
link |
So how do you think about hard work?
link |
What do you think it takes to accomplish great things?
link |
And there's a lot of people saying
link |
that there needs to be some balance.
link |
You need to, in order to accomplish great things,
link |
you need to take some time off,
link |
you need to reflect and so on.
link |
Now, and then some people are just insanely working,
link |
burning the candle on both ends.
link |
How do you think about that?
link |
I think I was trolling in the Siraj interview
link |
Off camera, right before I smoked a little bit of weed,
link |
like, you know, come on, this is a joke, right?
link |
Like I do nothing to relax.
link |
Look where I am, I'm at a party, right?
link |
Yeah, yeah, yeah, that's true.
link |
So no, no, of course I don't.
link |
When I say that life is work though,
link |
I mean that like, I think that what gives my life meaning is work.
link |
I don't mean that every minute of the day
link |
you should be working.
link |
I actually think this is not the best way to maximize results.
link |
I think that if you're working 12 hours a day,
link |
you should be working smarter and not harder.
link |
Well, so work gives you meaning.
link |
For some people, other sorts of meaning
link |
is personal relationships, like family and so on.
link |
You've also, in that interview with Siraj,
link |
or the trolling, mentioned that one of the things
link |
you look forward to in the future is AI girlfriends.
link |
So that's a topic that I'm very much fascinated by,
link |
not necessarily girlfriends,
link |
but just forming a deep connection with AI.
link |
What kind of system do you imagine
link |
when you say AI girlfriend,
link |
whether you were trolling or not?
link |
No, that one I'm very serious about.
link |
And I'm serious about that on both a shallow level
link |
I think that VR brothels are coming soon
link |
and are going to be really cool.
link |
It's not cheating if it's a robot.
link |
I see the slogan already.
link |
But there's, I don't know if you've watched,
link |
or just watched the Black Mirror episode.
link |
I watched the latest one, yeah.
link |
Oh, the Ashley 2 one?
link |
No, where there's two friends
link |
who are having sex with each other in...
link |
Oh, in the VR game.
link |
It's just two guys,
link |
but one of them was a female, yeah.
link |
Which is another mind blowing concept.
link |
That in VR, you don't have to be the form.
link |
You can be two animals having sex.
link |
I mean, I'll see how nice that the software
link |
maps the nerve endings, right?
link |
I mean, yeah, they sweep a lot of the fascinating,
link |
really difficult technical challenges under the rug,
link |
like assuming it's possible
link |
to do the mapping of the nerve endings, then...
link |
I wish, yeah, I saw that,
link |
the way they did it with the little like stim unit
link |
on the head, that'd be amazing.
link |
So, well, no, no, on a shallow level,
link |
like you could set up like almost a brothel
link |
with like real dolls and Oculus Quests,
link |
write some good software.
link |
I think it'd be a cool novelty experience.
link |
But no, on a deeper, like emotional level,
link |
I mean, yeah, I would really like to fall in love
link |
Do you see yourself having a long term relationship
link |
of the kind monogamous relationship that we have now
link |
with a robot, with a AI system even,
link |
not even just a robot?
link |
So I think about maybe my ideal future.
link |
When I was 15, I read Eliezer Yudkowsky's early writings
link |
on the singularity and like that AI
link |
is going to surpass human intelligence massively.
link |
He made some Moore's law based predictions
link |
that I mostly agree with.
link |
And then I really struggled
link |
for the next couple of years of my life.
link |
Like, why should I even bother to learn anything?
link |
It's all gonna be meaningless when the machines show up.
link |
Maybe when I was that young,
link |
I was still a little bit more pure
link |
and really like clung to that.
link |
And then I'm like, well,
link |
the machines ain't here yet, you know,
link |
and I seem to be pretty good at this stuff.
link |
Let's try my best, you know,
link |
like what's the worst that happens.
link |
But the best possible future I see
link |
is me sort of merging with the machine.
link |
And the way that I personify this
link |
is in a long term monogamous relationship with a machine.
link |
Oh, you don't think there's a room
link |
for another human in your life,
link |
if you really truly merge with another machine?
link |
I mean, I see merging.
link |
I see like the best interface to my brain
link |
is like the same relationship interface
link |
to merge with an AI, right?
link |
What does that merging feel like?
link |
I've seen couples who've been together for a long time.
link |
And like, I almost think of them as one person,
link |
like couples who spend all their time together and...
link |
That's fascinating.
link |
You're actually putting,
link |
what does that merging actually looks like?
link |
It's not just a nice channel.
link |
Like a lot of people imagine it's just an efficient link,
link |
search link to Wikipedia or something.
link |
I don't believe in that.
link |
you're saying that there's the same kind of relationship
link |
you have with another human,
link |
that's a deep relationship.
link |
That's what merging looks like.
link |
I don't believe that link is possible.
link |
I think that that link,
link |
so you're like, oh, I'm gonna download Wikipedia
link |
right to my brain.
link |
My reading speed is not limited by my eyes.
link |
My reading speed is limited by my inner processing loop.
link |
And to like bootstrap that sounds kind of unclear
link |
how to do it and horrifying.
link |
But if I am with somebody and I'll use a somebody
link |
who is making a super sophisticated model of me
link |
and then running simulations on that model,
link |
I'm not gonna get into the question
link |
whether the simulations are conscious or not.
link |
I don't really wanna know what it's doing.
link |
But using those simulations
link |
to play out hypothetical futures for me,
link |
deciding what things to say to me,
link |
to guide me along a path.
link |
And that's how I envision it.
link |
So on that path to AI of superhuman level intelligence,
link |
you've mentioned that you believe in the singularity,
link |
that singularity is coming.
link |
Again, could be trolling, could be not,
link |
could be part, all trolling has truth in it.
link |
I don't know what that means anymore.
link |
What is the singularity?
link |
Yeah, so that's really the question.
link |
How many years do you think before the singularity,
link |
what form do you think it will take?
link |
Does that mean fundamental shifts in capabilities of AI?
link |
Or does it mean some other kind of ideas?
link |
Maybe that's just my roots, but.
link |
So I can buy a human beings worth of compute
link |
for like a million bucks today.
link |
It's about one TPU pod V3.
link |
I want like, I think they claim a hundred pay to flops.
link |
That's being generous.
link |
I think humans are actually more like 20.
link |
So that's like five humans.
link |
That's pretty good.
link |
Google needs to sell their TPUs.
link |
But I could buy, I could buy, I could buy GPUs.
link |
I could buy a stack of like, I'd buy 1080 TIs,
link |
build data center full of them.
link |
And for a million bucks, I can get a human worth of compute.
link |
But when you look at the total number of flops in the world,
link |
when you look at human flops,
link |
which goes up very, very slowly with the population
link |
and machine flops, which goes up exponentially,
link |
but it's still nowhere near.
link |
I think that's the key thing to talk about
link |
when the singularity happened.
link |
When most flops in the world are Silicon and not biological,
link |
that's kind of the crossing point.
link |
Like they're now the dominant species on the planet.
link |
And just looking at how technology is progressing,
link |
when do you think that could possibly happen?
link |
You think it would happen in your lifetime?
link |
Oh yeah, definitely in my lifetime.
link |
I've done the math.
link |
I like 2038 because it's the Unix timestamp rollover.
link |
Yeah, beautifully put.
link |
So you've said that the meaning of life is to win.
link |
If you look five years into the future,
link |
what does winning look like?
link |
I can go into like technical depth
link |
to what I mean by that, to win.
link |
It may not mean, I was criticized for that in the comments.
link |
Like, doesn't this guy wanna like save the penguins
link |
in Antarctica or like,
link |
oh man, listen to what I'm saying.
link |
I'm not talking about like I have a yacht or something.
link |
But I am an agent.
link |
I am put into this world.
link |
And I don't really know what my purpose is.
link |
But if you're an intelligent agent
link |
and you're put into a world,
link |
what is the ideal thing to do?
link |
Well, the ideal thing mathematically,
link |
you can go back to like Schmidt Hoover theories about this,
link |
is to build a compressive model of the world.
link |
To build a maximally compressive,
link |
to explore the world such that your exploration function
link |
maximizes the derivative of compression of the past.
link |
Schmidt Hoover has a paper about this.
link |
And like, I took that kind of
link |
as like a personal goal function.
link |
So what I mean to win, I mean like,
link |
maybe this is religious,
link |
but like I think that in the future,
link |
I might be given a real purpose
link |
or I may decide this purpose myself.
link |
And then at that point,
link |
now I know what the game is and I know how to win.
link |
I think right now,
link |
I'm still just trying to figure out what the game is.
link |
so you have imperfect information,
link |
you have a lot of uncertainty about the reward function
link |
and you're discovering it.
link |
But the purpose is...
link |
That's a better way to put it.
link |
The purpose is to maximize it
link |
while you have a lot of uncertainty around it.
link |
And you're both reducing the uncertainty
link |
and maximizing at the same time.
link |
And so that's at the technical level.
link |
What is the, if you believe in the universal prior,
link |
what is the universal reward function?
link |
That's the better way to put it.
link |
So that win is interesting.
link |
I think I speak for everyone in saying that
link |
I wonder what that reward function is for you.
link |
And I look forward to seeing that in five years,
link |
I think a lot of people, including myself,
link |
are cheering you on, man.
link |
So I'm happy you exist and I wish you the best of luck.
link |
Thanks for talking to me, man.