The following is a conversation with Dimitri Dolgov, the CTO of Waymo, which is an autonomous

driving company that started as Google self driving car project in 2009 and became Waymo in 2016.

Dimitri was there all along. Waymo is currently leading in the fully autonomous vehicle space

in that they actually have an at skill deployment of publicly accessible autonomous vehicles

driving passengers around with no safety driver with nobody in the driver seat. This to me is an

incredible accomplishment of engineering on one of the most difficult and exciting artificial

intelligence challenges of the 21st century. Quick mention of a sponsor followed by some

thoughts related to the episode. Thank you to Trial Labs, a company that helps businesses

apply machine learning to solve real world problems. Blinkist, an app I use for reading

through summaries of books, BetterHelp, online therapy with a licensed professional,

and cash app, the app I use to send money to friends. Please check out the sponsors in the

description to get a discount at the support this podcast. As a side note, let me say that

autonomous and semi autonomous driving was the focus of my work at MIT and as a problem space

that I find fascinating and full of open questions from both a robotics and a human psychology

perspective. There's quite a bit that I could say here about my experiences in academia on this

topic that revealed to me, let's say the less admirable sides of human beings. But I choose

to focus on the positive, on solutions. I'm brilliant engineers like Dimitri and the team

at Waymo, who work tirelessly to innovate and to build amazing technology that will define our

future. Because of Dimitri and others like him, I'm excited for this future. And who knows,

perhaps I too will help contribute something of value to it. If you enjoy this thing,

subscribe on YouTube, review it with five stars and up a podcast, follow on Spotify,

support on Patreon, or connect with me on Twitter, Alex Friedman. And now here's my

conversation with Dimitri Dolgov. When did you first fall in love with robotics or even computer

science more in general? Computer science first at a fairly young age. Then robotics happened

much later. I think my first interesting introduction to computers was in the late 80s,

when we got our first computer, I think it was an IBM, I think IBM AT. Remember those things

that had like a turbo button in the front? You press it and make the thing goes faster.

Did they already have floppy disks? Yeah, yeah, yeah. Like the 5.4 inch ones.

I think there was a bigger inch. So when something, then five inches and three inches.

Yeah, I think that was the five. I don't know, maybe that was before that was the giant plates,

then it didn't get that. But it was definitely not the three inch ones.

Anyway, so we got that computer. I spent the first few months just playing video games,

as you would expect. I got bored of that. So I started messing around and trying to figure

out how to make the thing do other stuff. Got into exploring programming. And a couple of years

later, I got to a point where I actually wrote a game, a lot of games, and a game developer,

a Japanese game developer actually offered to buy it for me for a few hundred bucks,

but for a kid in Russia. That's a big deal. That's a big deal. Yeah. I did not take the deal.

Wow, integrity. Yeah, I instead used the bid. Yes, that was not the most acute financial move

that I made in my life. Looking back at it now, I instead put it, well, I had a reason. I put it

online. It was, what did you call it back in the days? It was a freeware thing. It was not open

source, but you could upload the binaries, you would put the game online. And the idea was that

people like it and then they contribute and they send you a little donations. So I did my quick

math of like, of course, thousands and millions of people are going to play my game, send me a

couple of bucks a piece. You should definitely do that. As I said, not the best find. You're

already playing business models at that age. Remember what language it was? What programming

it was? Pascal. Pascal. Pascal. Pascal. And they had a graphical component. So it's not text based.

Yeah, it was like, I think there are 320 by 200, whatever it was, I think kind of the earlier

version. That's the resolution. That's the resolution, right? And I actually think the reason

why this company wanted to buy it is not like the fancy graphics or the implementation. It was

maybe the idea of my actual game. The idea of the game. Well, one of the things, it's so funny,

I used to play this game called Golden Axe. And the simplicity of the graphics and something

about the simplicity of the music, like it still haunts me. I don't know if that's a childhood

thing. I don't know if that's the same thing for Call of Duty these days for young kids. But

I still think that the simple one of the games are simple, that simple purity makes for like

allows your imagination to take over and thereby creating a more magical experience. Like now

with better and better graphics, it feels like your imagination doesn't get to create worlds,

which is kind of interesting. It could be just an old man on a porch waving at kids these days

that have no respect. But I still think that graphics almost get in the way of the experience.

I don't know. Flippy Bird. Yeah. Why don't I don't know if the game is closed.

I don't yet. But that's more about games that like that's more like Tetris World where they

optimally, masterfully like create a fun short term dopamine experience versus I'm more referring to

like role playing games where there's like a story you can live in it for months or years.

Like there's an Elder Scrolls series, which is probably my favorite set of games.

That was a magical experience. And then the graphics are terrible. The characters were

all randomly generated. But they're, I don't know, it pulls you in. There's a story. It's like

an interactive version of an Elder Scrolls Tolkien world and you get to live in it. I don't know. I

miss it. It's one of the things that suck about being an adult is there's no, you have to live

in the real world as opposed to the Elder Scrolls world. You know, whatever brings you joy, right?

Minecraft, right? Minecraft is a great example. You create like it's not the fancy graphics,

but it's the creation of your own worlds. Yeah, that one is crazy. You know, one of the pitches

for being a parent that people tell me is that you can like use the excuse of parenting to

go back into the video game world. And like, like that's like, you know, father, son, father,

father, daughter time, but really you just get to play video games with your kids. So anyway,

at that time, did you have any ridiculous, ambitious dreams of where as a creator you might go

as an engineer? What did you think of yourself as an engineer, as a tinker, or did you want to be

like an astronaut or something like that? You know, I'm tempted to make something up about,

you know, robots, engineering or, you know, mysteries of the universe. But that's not the

actual memory that pops into my mind when you when you asked me about childhood dreams. So I

actually share the real thing. When I was maybe four or five years old, I, as we all do, I thought

about no one I wanted to do when I grow up. And I had this dream of being a traffic control cup.

You know, they don't have those todays, I think, but you know, back in the 80s and you know, in

Russia, you probably are familiar with athletics, they had these, you know, police officers that

would stand in the middle of an intersection all day, and they would have their like striped back

back and wide batons that they would use to, you know, control the flow of traffic. And you know,

for whatever reason, I was strangely infatuated with this whole process. And like that, that was

my dream. That's what I wanted to do when I grew up. And you know, my parents, both physics profs,

by the way, I think we're, you know, a little concerned with that level of ambition coming

from their child at that age. Well, it's an interesting, I don't know if you can relate,

but I very much love that idea. I have an OCD nature that I think lends itself

very close to the engineering mindset, which is you want to kind of optimize,

you know, solve a problem by creating an automated solution, like a set of rules,

that set of rules you can follow, and then thereby make it ultra efficient. I don't know if that's,

it was of that nature. I certainly have that. There's like factor like SimCity and factory

building games, all those kinds of things kind of speak to that engineering mindset. Or did you

just like the uniform? I think it was more of the latter. I think it was the uniform and, you know,

the, the striped baton that made cars go in the right directions. But I guess, you know, it is,

I did end up, I guess, you know, working on the transportation industry one way or another.

No uniform, no, but that's right. Maybe it was my, you know, deep inner infatuation with the,

you know, traffic control batons that led to this career. Okay, when did you,

when was the leap from programming to robotics? That happened later. That was after grad school.

After, and actually the, you know, cell driving cars was, I think, my first real hands on introduction

to robotics. But I never really had that much hands on experience in school and training. I,

you know, worked on applied math and physics. Then in college, I did more abstract computer science.

And it was after grad school that I really got involved in robotics, which was actually cell

driving cars. And, you know, that was a big, big flip. What, what grad school? So I went to

grad school in Michigan, and then I did a postdoc at Stanford, which is that was the postdoc where

I got to play with cell driving cars. Yeah, so we'll return there. Let's go back to, to Moscow.

So I, you know, for episode 100, I talked to my dad. And also I grew up with my dad, I guess.

So I had to put up with them for many years. And he, he went to the Fistyev or MIPT.

It's weird to say in English, because I've heard all this in Russian, Moscow Institute of Physics

and Technology. And to me, that was like, I met some super interesting, as a child, I met some

super interesting characters. It felt to me like the greatest university in the world, the most

elite university in the world. And just the people that I met that came out of there were like,

not only brilliant, but also special humans. It seems like that place really tested the soul,

both like in terms of technically and like spiritually. So that could be just the romanticization

of that place. I'm not sure. But so maybe you can speak to it. But is it correct to say that you

spent some time at Fistyev? Yeah, that's right. Six years. I got my bachelor's and master's and

physics and math there. And it was interesting because my dad, actually both my parents,

went there. And I think all the stories that I heard, like, just like you, Alex, growing up

about the place and, you know, how interesting and special and magical it was, I think that was

a significant, maybe the main reason I wanted to go there for college, enough so that I actually

went back to Russia from the U.S. I graduated high school in the U.S. And you went back there.

I went back there. Yeah. Wow. Exactly the reaction most of my peers in college had,

but, you know, perhaps a little bit stronger that would point me out as this crazy kid.

Were your parents supportive of that? Yeah. Yeah. My games, your previous question, they

supported me in letting me pursue my passions and the things that I was there.

That's a bold move. Wow. What was it like there?

It was interesting. You know, definitely fairly hardcore on the fundamentals of, you know,

math and physics and, you know, lots of good memories from, you know, from those times.

So, okay. So, Stanford, how'd you get into autonomous vehicles?

I had the great fortune and great honor to join Stanford's DARPA urban challenge team

in 2006. This was a third in the sequence of the DARPA challenges. There were two

grand challenges prior to that. And then in 2007, they held the DARPA urban challenge.

So, you know, I was doing my postdoc I had. I joined the team and worked on motion planning for,

you know, that competition. So, okay. So, for people who might not know,

I know from a certain perspective, autonomous vehicles is a funny world.

In a certain circle of people, everybody knows everything. And in a certain circle,

nobody knows anything in terms of general public. So, it's interesting. It's a good question of

what to talk about. But I do think that the urban challenge is worth revisiting. It's a fun little

challenge. One that, first of all, like sparked so many incredible minds to focus on one of the

hardest problems of our time in artificial intelligence. So, that's a success from a

perspective of a single little challenge. But can you talk about, like, what did the challenge

involve? So, were there pedestrians? Were there other cars? What was the goal? Who was on the

team? How long did it take? Any fun sort of specs? Sure, sure, sure. So, the way that the challenge

was constructed in just a little bit of background, as I mentioned, this was the third

competition in that series. The first two were the grand challenge called the grand challenge.

The goal there was to just drive in a completely static environment. You had to drive in a desert.

That was very successful. So, then DARPA followed with what they called the urban challenge, where

the goal was to build vehicles that could operate in more dynamic environments and share them with

other vehicles. There were no pedestrians there. But what DARPA did is they took over an abandoned

air force base. And it was kind of like a little fake city that they built out there. And they had

a bunch of robots, you know, cars that were autonomous in there all at the same time,

mixed in with other vehicles driven by professional drivers. And each car had a mission. And so,

there's a crude map that they received at the beginning. And they had a mission and go, you

know, here and then there and over here. And they kind of all were sharing this environment at the

same time. They had to interact with each other. They had to interact with the human drivers.

So, it's this very first, very rudimentary version of a self driving car that, you know,

could operate in an environment shared with other dynamic actors. That, as you said,

you know, really, in many ways, you know, kickstarted this whole industry.

Okay. So, who was on the team? And how'd you do? I forget.

I came in second. Perhaps that was my contribution to the team. I think the staffer team came in

first in the DARPA challenge. But then I joined the team and, you know, you were the one with the

bug in the code. I mean, do you have sort of memories of some particularly challenging things?

Or, you know, one of the cool things is not, you know, this isn't a product. This isn't the thing

that, you know, there's, you have a little bit more freedom to experiment. So, you can take risks.

And there's, so you can make mistakes. So, is there interesting mistakes? Is there interesting

challenges that stand out to you as something that taught you a good technical lesson or a

good philosophical lesson from that time? Yeah. You know, definitely, definitely a very memorable

time. Not really a challenge, but like one of the most vivid memories that I have from the time. And

I think that was actually one of the days that, you know, really got me hooked on this whole field

was the first time I got to run my software on the car. And I was working on a part of our planning

algorithm that had to navigate in parking lots. So it was, you know, something that, you know,

called free space motion planning. So the very first version of that was, you know, we tried on

the car. It was on Stanford's campus in the middle of the night. And, you know, I had this little,

you know, course constructed with cones in the middle of a parking lot. So we're there like

3am, you know, by the time we got the code to, you know, you know, compile and turn over. And,

you know, it drove. I could actually did something quite reasonable. And, you know, it was, of course,

very buggy at the time and had all kinds of problems. But it was pretty darn magical. I remember

going back and, you know, later at night and trying to fall asleep and just being unable to

fall asleep for the rest of the night. Just my mind was blown. And that's what I've been doing

ever since for more than a decade. In terms of challenges and, you know, interesting memories

like on the day of the competition, it was pretty nerve wrecking. I remember standing there with

Mike Montemarillo, who was the software lead and wrote most of the code. I think I did one little

part of the planner, Mike, you know, incredibly did pretty much the rest of it with a bunch of

other incredible people. But I remember standing on the day of the competition, you know, watching

the car, you know, with Mike and your cars are completely empty, right? They're all there,

lined up in the beginning of the race. And then, you know, DARPA sends them, you know, on their

mission, one by one, something leave. And Mike, you just, they had these sirens. They all had

their different silence, right? Each siren had its own personality, if you will. So, you know,

off they go and you don't see them. You just kind of, and then every once in a while, they, you know,

come a little bit closer to where the audience is. And you can kind of hear, you know, the sound of

your car. And, you know, it seems to be moving along. So that gives you hope. And then, you know,

it goes away and you can't hear it for too long. You start getting anxious, right? So it's a little

bit like, you know, sending your kids to college and like, you know, kind of you invested in them.

You hope you, you build it properly, but like, it's still anxiety inducing. So that was an

incredibly fun few days. In terms of, you know, bugs, as we mentioned, you know, one that was my bug

that caused us the loss of the first place is still a debate that, you know, occasionally have

with people on the CMU team, CMU came first, I should mention that. See, you haven't heard of

them. But yeah, it's something, you know, it's a small school, it's, it's, it's, you know,

it's really a glitch that, you know, they happen to succeed at something robotics related.

Very scenic though. So most people go there for the scenery. Yeah, it's a beautiful campus.

Unlike Stanford. So for people, yeah, that's true. Unlike Stanford. For people who don't

know, CMU is one of the great robotics and sort of artificial intelligence universities in the

world. CMU Carnegie Mellon University. Okay, sorry, go ahead. Good, good PSA. So in the part that

I contributed to, which was navigating parking lots, and the way, you know, that part of the

mission work is you in a parking lot, you would get from DARPA an outline of the map. You basically

get this giant polygon that defined the perimeter of the parking lot. And there would be an entrance

and maybe multiple entrances or exits to it. And then you would get a goal within that open space,

XY heading, where the car had to park. It had no information about the obstacles that the car

might encounter there. So it had to navigate kind of completely free space from the entrance to the

parking lot into that parking space. And then once you're parked there, it had to exit the

parking lot. And you know, while of course, encountering and reasoning about all the obstacles

that it encounters in real time. So our interpretation, or at least my interpretation

of the rules was that you had to reverse out of the parking spot. And that's what our cars did,

even if there's no obstacle in front. That's not what CMU's car did. And it just kind of

drove right through. So there's still a debate. And of course, you know, if you stop and then

reverse out and go out the different way, that costs you some time. And so there's still a debate

whether, you know, it was my poor implementation that cost us extra time, or whether it was,

you know, CMU violating an important rule of the competition. And, you know, I have my own

opinion here. In terms of other bugs, and like, I have to apologize to Mike Montemarillo for

sharing this on air. But it is actually one of the more memorable ones. And it's something that's

kind of become a bit of a metaphor in the industry since then, I think, at least in some

circles, it's called the victory circle or victory lap. And our cars did that. So in one of the

missions in the urban challenge, in one of the courses, there was this big oval right by the

start and finish of the race. So the ARPA head, a lot of the missions would finish in that same

location. And it was pretty cool because you could see the cars come by, you know, kind of finish

that part leg of the trip, that leg of the mission, and then go on and finish the rest of it. And

other vehicles would, you know, come hit their waypoint and exit the oval and off they would go.

Our car in the hand would hit the checkpoint, and then it would do an extra lap around the

oval and only then, you know, leave and go in its merry way. So over the course of the full day,

it accumulated some extra time. And the problem was that we had a bug where it wouldn't, you know,

start reasoning about the next waypoint and plan a route to get to that next point until it

hit a previous one. And in that particular case, by the time you hit that one, it was too late

for us to consider the next one and kind of make a lane change so that every time we would do like

an extra lap. So, you know, that's the Stanford victory lap. The victory lap. Oh, that's, there's,

I feel like there's something philosophically profound in there somehow. But I mean, ultimately,

everybody is a winner in that kind of competition. And it led to sort of famously

to the creation of Google self driving car project and now Waymo. So can we give an overview of

how is Waymo born? How is the Google self driving car project born? What is the mission? What is

the hope? What is it is the engineering kind of set of milestones that it seeks to accomplish?

There's a lot of questions in there. Yeah. I don't know.

But you're right. Kind of the DARPA urban challenge and the previous DARPA grand challenges

kind of led, I think, to a very large, you know, degree to that next step. And then, you know,

Larry and Sergey, Larry Page and Sergey Brin, Google hunter scores, I saw that competition

and believed in the technology that the Google self driving car project was born, you know, at

that time, and we started in 2009, it was a pretty small group of us, about a dozen people

who came together to work on this project at Google. At that time, we saw an incredible early

result in the DARPA urban challenge. I think we're all incredibly excited about

where we got to. And we believed in the future of the technology, but we still had a very

rudimentary understanding of the problem space. So the first goal of this project in 2009 was to

really better understand what we're up against. And, you know, with that goal in mind, when we

started the project, we created a few milestones for ourselves that maximized learning. Well,

the two milestones were, you know, one was to drive 100,000 miles in autonomous mode, which was,

at that time, orders of magnitude that more than anybody has ever done. And the second milestone

was to drive 10 routes. Each one was 100 miles long. They were specifically chosen to be kind of

extra spicy, extra complicated and sampled the full complexity of that domain. And you had to

drive each one from beginning to end with no intervention, no human intervention. So you'd

get to the beginning of the course, you would press the button that would engage in autonomy,

and you had to go for 100 miles beginning to end with no interventions. And it sampled,

again, the full complexity of driving conditions. Some were on freeways. We had one route that

went all through all the freeways and all the bridges in the Bay Area. You know, we had some

that went around Lake Tahoe and kind of mountains roads. We had some that drove through dense urban

environments like downtown Palo Alto and through San Francisco. So it was incredibly

interesting to work on. And it took us just under two years, about a year and a half, a little bit

more to finish both of these milestones. And in that process, it was an incredible amount of fun,

probably the most fun I had in my professional career. And you're just learning so much. You are,

you know, the goal here is to learn a prototype. You're not yet starting to build a production

system, right? So you just, you were, you know, this is when you're kind of, you know, working 24

seven and you're hacking things together. And you also don't know how hard this is. I mean,

it's the point, like, so, I mean, that's an ambitious, if I put myself in that mindset, even

still, that's a really ambitious set of goals, like just those two to picking, picking 10 different,

different, difficult, spicy challenges, and then having zero interventions. So like not saying,

gradually, we're going to like, you know, over a period of 10 years, we're going to have a bunch

of routes and gradually reduce the number of interventions, you know, that literally says,

like, by as soon as possible, we want to have zero and on hard roads. So like to me, if I was

facing that, it's unclear that whether that takes two years or whether that takes 20 years.

I mean, under two, I guess that speaks to a really big difference between doing something

once and having a prototype where you're going after, you know, learning about the problem versus

how you go about engineering a product that where you look at, you know, you properly do

evaluation, you look at metrics, you know, drive dog, and you're confident that you can do that.

And I guess that's why it took a dozen people, you know, 16 months or a little bit more than that,

back in 2009 and 2010, with the technology of, you know, the more than a decade ago,

that amount of time to achieve that milestone of 10 routes, 100 miles each in no interventions.

And, you know, it took us a little bit longer to get to, you know, a full driverless product

that customers use. That's another really important moment. Is there some

memories of technical lessons? Or just one, like, what did you learn about the problem

of driving from that experience? I mean, we can now talk about, like, what you learned from

modern day Waymo. But I feel like you may have learned some profound things in those early days,

even more so, because it feels like what Waymo is now is to trying to, you know, how to do scale,

how to make sure you create a product, how to make sure it's like safe, you know, those things,

which is all fascinating challenges. But, like, you were facing the more fundamental

philosophical problem of driving in those early days, like, what the hell is driving?

As an autonomous, or maybe I'm again romanticizing it, but is there a, is there some

valuable lessons you picked up over there at those two years?

A ton. The most important one is probably that we believe that it's doable. And we've gotten

far enough into the problem that, you know, we had a, I think, only glimpse of the true complexity

of that domain. And it's a little bit like, you know, climbing a mountain where you kind of,

you know, see the next peak and you think that's kind of the summit, but then you get to that and

you kind of see that this is just the start of the journey. But we've tried, we've sampled enough

of the problem space, and we've made enough rapid success, even, you know, with technology

of 2009, 2010, that it gave us confidence to then pursue this as a real product.

So, okay, so the next step, you mentioned the milestones that you had in those two years.

What are the next milestones that then led to the creation of Waymo and beyond?

Yeah, it was a really interesting journey. And, you know, Waymo came a little bit later.

Then, you know, we completed those milestones in 2010. That was the pivot when we decided to

focus on actually building a product using this technology. The initial couple years after that,

we were focused on a freeway, you know, what you would call a driver assist, maybe, you know,

an L3 driver assist program. Then around 2013, we've learned enough about the space and if

thought more deeply about, you know, the product that we wanted to build that we pivoted. We pivoted

towards this vision of, you know, building a driver and deploying it fully driverless vehicles

without a person. And that's the path that we've been on since then. And it was exactly the right

decision for us. So there was a moment where you're also considered like, what is the right

trajectory here? What is the right role of automation in the task of driving? It wasn't

from the early days, obviously, you want to go fully autonomous. From the early days,

it was not. I think it was around 2013, maybe, that we've, that became very clear and we made

that pivot and it also became very clear. And that it's, even the way you go building a driver

assist system is fundamentally different from how you go building a fully driverless vehicle. So,

you know, we've pivoted towards the ladder and that's what we've been working on ever since.

And so that was around 2013. Then there's a sequence of really meaningful for us,

really important defining milestones since then. In 2015, we had our first, actually,

the world's first fully driverless ride on public roads. It was in a custom build vehicle that we

had. We must have seen those. We called them the Firefly, that, you know, funny looking marshmallow

looking thing. And we put a passenger, his name was Steve Mann, a great friend of our project

from the early days. The man happens to be blind. So we put him in that vehicle. The car had no

steering wheel, no pedals. It was an uncontrolled environment. You know, no, you know, lead or

chase cars, no police escorts. And, you know, we did that trip a few times in Austin, Texas.

So that was a really big milestone. But that was in Austin. Yeah. Okay. And, you know, we only,

but at that time, we're only, it took a tremendous amount of engineering. It took a tremendous amount

of validation to get to that point. But, you know, we only did it a few times. We only did that. It

was a fixed route. It was not kind of a controlled environment, but it was a fixed route. And we

only did a few times. Then in 2016, end of 2016, beginning of 2017 is when we founded Waymo,

the company. That's when we kind of, that was the next phase of the project where I wanted,

we believed in kind of the commercial vision of this technology. And it made sense to create an

independent empty, you know, within that alphabet umbrella to pursue this product at scale.

Beyond that, in 2017, later in 2017, was another really huge step for us, really big milestone

where we started, it was October of 2017, where when we started regular driverless operations

on public roads, that first day of operations, we drove in one day, in that first day, 100 miles

in driverless fashion. And then the most important thing about that milestone was not that 100

miles in one day, but that it was the start of kind of regular, ongoing driverless operations.

And when you say driverless, it means no driver.

That's exactly right. So on that first day, we actually hit a mix. And in some,

we didn't want to like, you know, be on YouTube and Twitter that same day. So in many of the rides,

we had somebody in the driver's seat, but they could not disengage, like the car could not

disengage. But actually, on that first day, some of the miles were driven and just completely

empty driver's seat. And this is the key distinction that I think people don't realize,

it's, you know, that oftentimes when you talk about autonomous vehicles, there's often a driver

in the seat that's ready to take over what's called a safety driver. And then Waymo is really one of

the only companies that I'm aware of, or at least as like boldly and carefully and all that is

actually has cases. And now we'll talk about more and more, where there's literally no driver.

So that's another interesting case of where the driver is not supposed to disengage. That's like

a nice middle ground. They're still there, but they're not supposed to disengage. But really,

there's the case when there's no, okay, there's something magical about there being nobody in

the driver's seat. Like, just like to me, you mentioned the first time you wrote some code

for free space navigation of the parking lot, that was like a magical moment. To me, just sort of

as an observer of robots, the first magical moment is seeing an autonomous vehicle turn,

like make a left turn, like, apply sufficient torque to the steering wheel to where like,

there's a lot of rotation. And for some reason, and there's nobody in the driver's seat, for some

reason that, that communicates that here's a being with power that makes a decision. There's

something about like the steering wheel, because we perhaps romanticize the notion of the steering

wheel, it's so essential to our conception, our 20th century conception of a car. And it turning

the steering wheel with nobody in driver's seat, that to me, I think maybe to others, it's really

powerful, like this thing is in control. And then there's this leap of trust that you give,

like, I'm going to put my life in the hands of this thing that's in control. So in that sense,

when there's no driver in the driver's seat, that's a magical moment for robots. So I got

the chance to last year to take a ride in a Williamsville vehicle. And that was the magical

moment. There's like, nobody in the driver's seat. It's like the little details, you would think it

doesn't matter whether there's a driver or not. But like, if there's no driver, and the steering

wheel is turning on its own, I don't know, that's magical. It's absolutely magical. I've taken many

of these rides in a completely empty car. No human in the car pulls up. You call it on your

cell phone, it pulls up, you get in, it takes you on its way. There's nobody in the car but you,

right? That's something called fully driverless, our rider only mode of operation. Yeah, it is

magical. It is transformative. This is what we hear from our riders. It really changes your

experience. And that really is what unlocks the real potential of this technology. But coming

back to our journey, that was 2017 when we started truly driverless operations. Then in 2018, we've

launched our public commercial service that we called Waymo One in Phoenix. In 2019, we started

offering truly driverless rider only rides to our early rider population of users. And then in

2020, it's also been a pretty interesting year, one of the first months. I less about technology,

but more about the maturing and the growth of Waymo as a company. We raised our first

round of external financing this year. We were part of Alphabet. So obviously, we have access to

significant resources. But on the journey of Waymo maturing as a company, it made sense for us to

partially go externally in this round. So we raised about $3.2 billion from that round.

We've also started putting our fifth generation of our driver, our hardware that is on the new

vehicle. But it's also a qualitatively different set of self driving hardware that is now on the

JLR pace. So that was a very important step for us. The hardware specs, fifth generation,

I think it would be fun to maybe, I apologize if I'm interrupting, but maybe talk about maybe the

generations with the focus on what we're talking about on the fifth generation in terms of hardware

specs, like what's on this car. Sure. So we separated on the actual car that we are driving

from the self driving hardware we put on it. Right now, we have, so this is, as I mentioned,

the fifth generation, we've gone through, we started building our own hardware many,

many years ago. And that Firefly vehicle also had the hardware suite that was mostly

designed and engineered and built in house. Lighters are one of the more important components

that we design and build from the ground up. So on the fifth generation,

after drivers of our software and hardware that we're switching to right now, we have,

as with previous generations, in terms of sensing, we have lighters, cameras, and radars. And we

have a pretty beefy computer that processes all that information and makes decisions and real

time on board the car. So in all of the, and it's really a qualitative jump forward in terms of

the capabilities and the various parameters and specs of the hardware compared to what we had

before and compared to what you can get off the shelf in the market today. Meaning from fifth

to fourth or from fifth to first? Definitely from first to fifth, but also from the fourth.

World's dumbest question. Definitely from fourth to fifth, as well as the last step is a big step

forward. So everything's in house. So like lighters built in house and cameras are built in house?

It's different. We work with partners. There's some components that we get from our manufacturing

and supply chain partners. What exactly is in house is a bit different. We do a lot of custom

design on all of our sensing material. There's lighters, radars, cameras. Exactly. There's

lighters are almost exclusively in house. And some of the technologies that we have, some of the

fundamental technologies there are completely unique to Waymo. That is also largely true about

radars and cameras. It's a little bit more of a mix in terms of what we do ourselves versus what

we get from partners. Is there something super sexy about the computer that you can use?

The computer that you can mention? That's not top secret. Like for people who enjoy computers,

I mean, there's a lot of machine learning involved, but there's a lot of just basic

computer. You have to probably do a lot of signal processing on all the different sensors.

You have to integrate everything. It has to be in real time. There's probably some kind of redundancy

type of situation. Is there something interesting you can say about the computer for the people who

love hardware? It does have all of the characteristics, all the properties that you just mentioned.

Redundancy, very beefy compute for general processing as well as inference and ML models.

It is some of the more sensitive stuff that I don't want to get into for IP reasons, but

we've shared a little bit in terms of the specs of the sensors that we have on the car. We've

actually shared some videos of what our lighters see in the world. We have 20 line cameras. We have

five lighters. We have six radars on these vehicles. You can get a feel for the amount of data that

they're producing that all has to be processed in real time to do perception, to do complex

reasoning. It gives you some idea of how beefy those computers are, but I don't want to get

into specifics of exactly how we build them. Okay. Let me try some more questions that you

can get into the specifics of like GPU wise. Is that something you can get into? I know that

Google works with GPUs and so on. I mean, for machine learning folks, it's kind of interesting.

How do I ask it? I've been talking to people in the government about UFOs and they don't

answer any questions. This is how I feel right now asking about GPUs.

But is there something interesting that you could reveal or would leave it up to our imagination

some of the compute? Is there any fun trickery? Like I talked to Chris Latner for a second time

and he was a key person about TPUs and there's a lot of fun stuff going on in Google in terms of

hardware that optimizes for machine learning. Is there something you can reveal in terms of how

much you mentioned customization, how much customization there is for hardware for machine

learning purposes? I'm going to be like that government, you know, you've got a person who

bought UFOs, but I guess I will say that it's really compute. It's really important. We have

very data hungry and compute hungry ML models all over our stack and this is where, you know,

both being part of Alphabet as well as designing our own sensors and the entire hardware suite

together where on one hand you get access to really rich raw sensor data that you can pipe from your

sensors into your compute platform and build the whole pipe from sensor raw sensor data

to the big compute as then have the massive compute to process all that data. This is where

we're finding that having a lot of control of that hardware part of the stack is really

advantageous. One of the fascinating magical places to me, again, might not be able to speak to

the details, but it is the other compute, which is like, we're just talking about a single car,

but the driving experience is a source of a lot of fascinating data and you have a huge

amount of data coming in on the car and the infrastructure of storing some of that data

to then train or to analyze or so on, that's a fascinating piece of it that I understand

a single car. I don't understand how you pull it all together in a nice way. Is that something

that you could speak to in terms of the challenges of seeing the network of cars and then bringing

the data back and analyzing things that went like edge cases of driving, be able to learn on them,

to improve the system, to see where things went wrong, where things went right and analyze all

that kind of stuff? Is there something interesting there from an engineering perspective?

Oh, there's an incredible amount of really interesting work that's happening there,

both in the real time operation of the fleet of cars and the information that they exchange

with each other in real time to make better decisions, as well as the off board component

where you have to deal with massive amounts of data for training your ML models, evaluating

the ML models for simulating the entire system and for evaluating your entire system. And this is

where being part of Alphabet has, once again, been tremendously advantageous. We consume an

incredible amount of compute for ML infrastructure. We build a lot of custom frameworks to get good

on data mining, finding the interesting edge cases for training and for evaluation of the system,

for both training and evaluating some components and your sub parts of the system in various ML

models, as well as the evaluating the entire system and simulation.

Okay, that first piece that you mentioned that cars communicating to each other, essentially,

I mean, through perhaps through a centralized point. But what, that's fascinating too.

How much does that help you? Like, if you imagine, you know, right now the number of

way more vehicles is whatever x, I don't know if you can talk to what that number, but it's not in

the hundreds of millions yet. And imagine if the whole world is way more vehicles,

like that changes potentially the power of connectivity, like the more cars you have,

I guess actually, if you look at Phoenix, because there's enough vehicles, there's enough,

when there's like some level of density, you can start to probably do some really interesting

stuff with the fact that cars can negotiate, can be, can communicate with each other and thereby

make decisions. Is there something interesting there that you can talk to about like, how does

that help with the driving problem from as compared to just a single car solving the driving problem

by itself? Yeah, it's a spectrum. I first to say that, you know, it's it helps. And it helps in

various ways, but it's not required. Right now, the way we build our system, like each cars can

operate independently, they can operate with no connectivity. So I think it is important that,

you know, you have a fully autonomous, you know, fully capable driver that, you know, computerized

driver that each car has. Then, you know, they do share information. And they share information

in real time, and it really helps. So the way we do this today is, you know, whenever one car

encounters something interesting in the world, whether it might be an accident or a new construction

zone, that information immediately gets, you know, uploaded over the air and is propagated to the rest

of the fleet. So and that's kind of how we think about maps as priors in terms of the knowledge

of our drivers, of our fleet of drivers that is distributed across the fleet. And it's updated

in real time. So that's one use case. You know, you can imagine as the, you know, the density

of these vehicles go up that they can exchange more information in terms of what they're planning

to do and start influencing how they interact with each other, as well as, you know, potentially

sharing some observations, right, to help with, you know, if you have enough density of these

vehicles where, you know, one car might be seeing something that another is relevant to another

car that is very dynamic, you know, it's not part of kind of your updating your static prior

of the map of the world, but it's more of a dynamic information that could be relevant to the

decisions that another car is making real time. So you can see them exchanging that information

and you can build on that. But again, I see that as an advantage, but it's, you know, not a requirement.

So what about the human in the loop? So when I got a chance to drive with a

ride in a Waymo, you know, there's customer service.

So like there is somebody that's able to dynamically like tune in and help you out.

What, what role does the human play in that picture? That's a fascinating like, you know,

the idea of teleoperation, be able to remotely control a vehicle. So here what we're talking about

is like, like frictionless, like a human being able to in a, in a frictionless way, sort of

help you out. I don't know if they're able to actually control the vehicle. Is that something

you could talk to? Yes. Okay. To be clear, we don't do teleoperation. I gotta believe in

teleoperation for various reasons. That's not what we have in our cars. We do, as you mentioned,

have, you know, version of, you know, customer support, you know, we call it live health. In

fact, we find it that it's very important for our rider experience, especially if it's your

first trip, you've never been in a fully driverless, right, or only Waymo vehicle, you get in,

there's nobody there. And so you can imagine having all kinds of, you know, questions in your head,

like how this thing works. So we've put a lot of thought into kind of guiding our,

our riders, our customers through that experience, especially for the first time,

they get some information on the phone. If the fully driverless vehicle is used to service their

trip, when you get into the car, we have an in car, you know, screen and audio that kind of guides

them and explains what to expect. They also have a button that they can push that will connect them

to, you know, a real life human being that they can talk to, right, about this whole process.

So that's one aspect of it. There is, you know, I should mention that there is another function

that humans provide to our cars, but it's not teleoperation. You can think of it a little bit

more like, you know, fleet assistance, kind of like, you know, traffic control that you have,

where our cars, again, they're responsible on their own for making all of the decisions,

all of the driving decisions that don't require connectivity. They, you know, anything that is

safety or latency critical is done, you know, purely autonomously by onboard system. But there

are situations where, you know, if connectivity is available, kind of car encounters a particularly

challenging situation, you can imagine like a super hairy scene of an accident, the cars will

do their best, they will recognize that it's an off nominal situation, they will, you know, do their

best to come up, you know, with the right interpretation, the best course of action in

that scenario. But if the connectivity is available, they can ask for confirmation

from, you know, a human assistant to kind of confirm those actions and, you know,

perhaps provide a little bit of kind of contextual information and guidance.

So October 8th was when you're talking about the, was Waymo launched the fully self,

the public version of its fully driverless, that's the right term, I think,

service in Phoenix. Is that October 8th? That's right. It was the introduction of fully driverless

rider only vehicles into our public Waymo One service. Okay, so that's, that's amazing. So

it's like anybody can get into Waymo in Phoenix? That's right. So we previously had early people

in our early rider program taking fully driverless rides in Phoenix. And just this a little while

ago, we opened on October 8th, we opened that mode of operation to the public. So I can,

you know, download the app and, you know, go on a ride. There is a lot more demand right now

for that service. And then we have capacity. So we're kind of managing that. But that's

exactly the way you describe it. Yeah, well, this is interesting. So there's more demand than you can,

you can handle. Like what, what has been the reception so far? Like what, I mean, okay, so,

you know, that's, this is a product, right? That's a whole nother discussion of like how

compelling of a product it is. Great. But it's also like one of the most kind of transformational

technologies of the 21st century. So it's also like a tourist attraction. Like it's fun to,

you know, to be a part of it. So it'd be interesting to see like, what do people say? What do people,

what, what have been the feedback so far? You know, still early days, but so far the feedback has

been incredible, incredibly positive. They, you know, we asked them for feedback during the ride,

we asked them for feedback after the ride as part of their trip. We asked them some questions,

we asked them to rate the performance of our driver. Most by far, you know, most of our

drivers give us five stars in our app, which is absolutely great to see. And, you know, that's,

and we're, they're also giving us feedback on, you know, things we can improve. And, you know,

that's one of the main reasons we're doing this is Phoenix. And, you know, over the last couple

years, and every day today, we are just learning a tremendous amount of new stuff from our users.

There's, there's no substitute for actually doing the real thing, actually having a fully

driverless product out there in the field with, you know, users that are actually, you know,

paying us money to get from point A to point B. So this is a legitimate, like, there's a paid

service. That's right. And the idea is you use the app to go from point A to point B. And then

what, what are the A's? What are the, what's the freedom of the, of the starting and ending places?

It's an area of geography where that service is enabled. It's a, you know, decent size of

geography of territory. It's actually larger than, you know, than size of San Francisco.

And, you know, within that, you have, you know, full freedom of, you know, selecting where you

want to go. You know, of course, there's some and you, on your app, you get a map, you tell the car

where you want to be picked up, you know, and where you want, you know, the car to pull over

and pick you up. And then you tell it where you want to be dropped off. Right. And of course,

there are some exclusions, right? You want to be, you know, you wear it in terms of where the car

is allowed to pull over, right? So, you know, that you can do, but, you know, besides that,

it's amazing. It's not like a fixed, just would be very, I guess, I don't know, maybe that's

what's the question behind your question, but it's not a, you know, preset set of.

Yes, I guess. So within the geographic constraints with that, within that area,

anywhere out, it can be, you can be picked up and dropped off anywhere.

That's right. And, you know, people use them on like all kinds of trips they,

we have, and we have an incredible spectrum of riders. I think the youngest,

actually have car seats in them. And we have, you know, people taking their kids and rides. I

think the youngest riders we had on the cars are, you know, one or two years old, you know,

and the full spectrum of use cases. People, you can take them to, you know, schools,

to, you know, go grocery shopping, to restaurants, to bars, you know, run errands, you know, go

shopping, et cetera, et cetera, you go to your office, right? Like the full spectrum of use cases.

And people, you know, use them in their daily lives to get around. And we see all kinds of,

you know, really interesting use cases and that, that, that's providing us incredibly valuable

experience that we then, you know, used to improve our product.

So as somebody who's been on, done a few long rants with Joe Rogan and others about the toxicity

of the internet and the comments and the negativity in the comments, I'm fascinated by feedback. I,

I believe that most people are good and kind and intelligent and can provide, like,

even in disagreement, really fascinating ideas. So on a product side, it's fascinating to me,

like, how do you get the richest possible user feedback, like, to improve? What's,

what are the channels that you use to measure? Because, like, you're, you're no longer,

you're, it's one of the magical things about autonomous vehicles. It's not,

like, it's frictionless interaction with the human. So, like, you don't get to,

you know, it's just giving a ride. So, like, how do you get feedback from people to, in order to

improve? Yeah, great question. Various mechanisms. So as part of the normal flow, we ask people for

feedback. They, as the car is driving around, you know, we have on the phone and in the car,

and we have a touch screen in the car, you can actually click some buttons and provide

real time feedback on how the car is doing and how the car is handling a particular situation,

you know, both positive and negative. So that's one channel. We have, as we discussed, customer

support or life help, where, you know, if a customer wants to, has a question, or he has

some sort of concern, they can talk to a person in real time. So that, that is another mechanism

that gives us feedback. At the end of a trip, you know, we also ask them how things went. They

give us comments and, you know, a star rating. And, you know, if it's, we also, you know, ask them to

explain what, you know, what went well and, you know, what could be improved. And we, we have

our writers providing, you know, very rich feedback there. A lot, a large fraction is

very passionate and very excited about this technology. So we get really good feedback.

We also run UXR studies, right? You know, specific and that are kind of more, you know,

go more in depth and we will run both kind of lateral and longitudinal studies,

where we have, you know, deeper engagement with our customers. You know, we have our

user experience research team tracking over time. That's things about longitudinal is cool.

That's, that's exactly right. And, you know, that's another really valuable feedback, source of feedback.

And we're just covering a tremendous amount, right? People go grocery shopping and they like

want to load, you know, 20 bags of groceries in our cars. And like that's one workflow that you

maybe don't, you know, think about, you know, getting just right when you're building the

driverless product. I have people like, you know, who bike as part of their trip. So they,

you know, bike somewhere, then they get in our cars, they take a path or their bike,

they load into our vehicle, then they go, and that's, you know, how they, you know,

where we want to pull over and how that, you know, get in and get out process works,

provides very useful feedback in terms of, you know, what makes a good pickup and drop off location?

We get really valuable feedback. And in fact, we had to do some really interesting work with

high definition maps and thinking about walking directions. And if you imagine you're in a store,

in some giant space, and then, you know, you want to be picked up somewhere, like if you just drop

pin at the current location, which is maybe in the middle of a shopping mall, like what's the best

location for the car to come pick you up. And you can, you know, have simple heuristics where

you just kind of take your, you know, you clean in distance and find the nearest spot where the

car can pull over as closest to you. But oftentimes, that's not the most convenient one,

you know, I have many anecdotes where that heuristic breaks in horrible ways. I want example

that, you know, I often mentioned is somebody wanted to be, you know,

dropped off and Phoenix and, you know, weak car picked location that was close and the closest to

there, you know, where the pin was dropped on the map in terms of, you know, latitude and

longitude. But it happened to be on the other side of a parking lot that had this row of cacti

and the poor person had to like walk all around the parking lot to get to where they wanted to be

in 110 degree heat. So that, you know, that was about. So then, you know, we took all take all of

these, all that feedback from our users and incorporated into our system and improve it.

Yeah, I feel like that's like requires AGI to solve the problem of like, when you're, which is a

very common case when you're in a big space of some kind, like apartment building, it doesn't

matter. It's not some large space. And then you call the like the Waymo from there, right?

Like, whatever that doesn't matter, right? Chair vehicle. And like, where is the pin supposed to

drop? I feel like that's, you don't think I think that requires AGI. I'm going to

okay, the alternative, which I think the Google search engine is taught is like,

there's something really valuable about the perhaps slightly dumb answer, but a really powerful one,

which is like, what was done in the past by others? Like, what was the choice made by others?

That seems to be like in terms of Google search, when you have like billions of searches,

that you could, you could see which, like when they recommend what you might possibly mean,

they suggest based on not some machine learning thing, which they also do, but like on what

was successful for others in the past and finding a thing that they were happy with.

Is that integrated at all? Waymo, like what, what pickups work for others?

It is. I think you're exactly right. So there's a real, it's an interesting problem.

Naive solutions have, you know, interesting failure modes. So there's definitely lots of

things that can be done to improve. And both learning from, you know, what works, what doesn't

work in actual hail from, you know, getting richer data and getting more information about the

environment and, you know, richer maps. But you're absolutely right that there's something,

I think there's some properties of solutions that in terms of the effect that they have on

users so much, much, much, much better than others, right? And predictability and

understandability is important. So you can have maybe something that is not quite as optimal,

but is very natural and predictable to the user and kind of works the same way all the time.

And that matters. That matters a lot for the user experience. And, but, you know, to get to the basics,

the pretty fundamental property is that the car actually arrives where you told it, right? Like,

you can always, you know, change it, see it on the map and you can move it around if you don't

like it. And, but like that property that the car actually shows up on pin is critical, which,

you know, where compared to some of the human driven analogs, I think, you know, you can have

more predictability. It's actually the fact, if I did a little bit of a detail here, I think the

fact that it's, you know, your phone and the cars two computers talking to each other can lead to

some really interesting things we can do. And in terms of the user interfaces, you know, both in

terms of function, like the car actually shows up exactly where you told it, you want it to be.

But also some, you know, really interesting things on the user interface, like as the car is driving,

as you, you know, call it, and it's on the way to come pick you up. And of course, you get the

position of the car and the route on the map. But, and they actually follow that route, of course.

But it can also share some really interesting information about what is doing. So, you know,

our cars, as they are coming to pick you up, if it's come, if a car is coming up to a stop sign,

it will actually show you that, like, it's there sitting, because it's at a stop sign,

or a traffic light will show you that it's sitting at a red light. So, you know, they look

little things, right? But I find those little touch touches really interesting, really magical.

And it's just, you know, little things like that that you can do to kind of delight your users.

You know, this makes me think of there's some products that I just love,

like, there's a, there's a company called Rev, rev.com, where I like for this podcast,

for example, I can drag and drop a video. And then they do all the captioning,

it's humans doing the captioning, but they connect you, they automate everything of connecting you

to the humans, and they do the captioning and transcription, it's all effortless. And it like,

I remember when I first started using them, it was like, life's good. Like, because it was so

painful to figure that out earlier. The same thing with something called Isotope RX, this

company I use for cleaning up audio, like the sound cleanup they do, it's like drag and drop,

and it just cleans everything up very nicely. Another experience like that had with Amazon,

one click purchase, first time, I mean, other places do that now, but just the effortlessness

of purchasing, making it frictionless. It kind of communicates to me, like, I'm a fan of design,

I'm a fan of products, that you can just create a really pleasant experience, the simplicity

of it, the elegance just makes you fall in love with it. So I know, do you think about this kind

of stuff? I mean, it's exactly what we've been talking about. It's like, the little details

that somehow make you fall in love with the product, is that we went from like urban challenge days,

where love was not part of the conversation, probably. And to this point where there's human

beings, and you want them to fall in love with the experience, is that something you're trying to

optimize for, trying to think about like how do you create experience that people love? Absolutely.

That's the vision is removing any friction or complexity from getting our users, our writers,

to where they want to go. Making that as simple as possible. And then beyond that,

just transportation, making things and goods get to their destination as seamlessly as

possible. I talked about a drag and drop experience where I kind of express your intent. And then,

it just magically happens. And for our writers, that's what we're trying to get to is you download

an app, and you click and car shows up. It's the same car. It's very predictable. It's a safe and

high quality experience. And then it gets you in a very reliable, very convenient,

uh, frictionless way to where you want to be. And along the journey,

I think we also want to do little things to delight our users.

Like the ride sharing companies, because they don't control the experience, I think,

they can't make people fall in love necessarily with the experience,

or maybe they haven't put in the effort. But I think if I would just speak to the ride

sharing experience that currently have, it's just very, it's just very convenient.

But there's a lot of room for like falling in love with it. Like we can speak to sort of car

companies. Car companies do this well. You can fall in love with a car, right? And be like a loyal

car person, like whatever. Like I like badass hot rods, I guess 69 Corvette. And at this point,

you know, you can't really, cars are so owning a car. It's so 20th century, man. But is there

something about the way more experience where you hope that people will fall in love with it?

Because that, is that part of it? Or is it part of, is it just about making a convenient ride,

not ride sharing? I don't know what the right term is, but just a convenient A to B autonomous

transport. Or like, do you want them to fall in love with Waymo? Maybe elaborate a little bit.

I mean, almost like from a business perspective, I'm curious, like,

how do you want to be in the background invisible? Or do you want to be

like a source of joy that's in the very much in the foreground?

I want to provide the best, most enjoyable transportation solution. And that means building

it, building our product and building our service in a way that people do, kind of use in a very

seamless, frictionless way in their day to day lives. And I think that does mean,

you know, in some way falling in love in that product, right? It just kind of becomes part

of your routine. It comes down, in my mind, to safety, predictability of the experience,

and privacy, I think, aspects of it, right? Our cars, you get the same car, you get very

predictable behavior. And that is important. And if you're going to use it in your daily life.

Privacy. And when you're in a car, you can do other things. You're spending a bunch of just

another space where you're spending a significant part of your life. So not having to share it

with other people who you don't want to share it with, I think is a very nice property. Maybe

you want to take a phone call or do something else in the vehicle. And safety on the quality

of the driving, as well as the physical safety of not having to share that ride is important

to a lot of people. What about the idea that when there's somebody, like a human driving,

and they do a rolling stop on a stop sign, like sometimes you get an Uber or a Lyft or whatever,

like human driver, and they can be a little bit aggressive as drivers.

It feels like there is not all aggression is bad. Now, that may be a wrong, again, 20th century

conception of driving. Maybe it's possible to create a driving experience. Like, if you're in

the back, busy doing something, maybe aggression is not a good thing. It's a very different kind

of experience, perhaps. But it feels like in order to navigate this world, you need to,

how do I phrase this? You need to kind of bend the rules a little bit, or at least I can test

the rules. I don't know what language politicians use to discuss this, but whatever language they

use, you like flirt with the rules. I don't know. But like you sort of have a bit of an aggressive

way of driving that asserts your presence in this world, thereby making other vehicles and people

respect your presence, and thereby allowing you to navigate through intersections in a timely

fashion. I don't know if any of that made sense, but how does that fit into the experience of

driving autonomously? This is your hitting on a very important point of a number of behavioral

components and parameters that make your driving feel assertive and natural, comfortable,

predictable. Our cars will follow rules. They will do the safest thing possible in all situations,

let me be clear on that. But if you think of really, really good drivers, think about professional

limo drivers. They will follow the rules. They're very, very smooth, and yet they're very efficient,

but they're assertive. They're comfortable for the people in the vehicle. They're predictable for the

other people outside the vehicle that they share the environment with, and that's the kind of driver

that we want to build. Maybe there's a sport analogy there. You can do in many sports the

true professionals are very efficient in their movements. They don't do hectic flailing. They're

smooth and precise, and they get the best results. That's the kind of driver that we want to build.

In terms of aggressiveness, you can roll through the stop signs. You can do crazy lane changes.

Typically, it doesn't get you to your destination faster, typically not the safest or most predictable,

the most comfortable thing to do. But there is a way to do both. That's what we're doing,

we're trying to build the driver that is safe, comfortable, smooth, and predictable.

Yeah, that's a really interesting distinction. I think in the early days of autonomous vehicles,

the vehicles felt cautious as opposed to efficient. But when I rode in the Waymo,

it was quite assertive. It moved pretty quickly. One of the surprising feelings was that it

actually went fast and it didn't feel awkwardly cautious than autonomous vehicles. I've also

programmed autonomous vehicles, and everything I've ever built felt awkwardly either overly

aggressive, especially when it was my code, or awkwardly cautious as the way I would put it.

Waymo's vehicle felt assertive, and I think efficient is the right terminology here.

I also like the professional limo driver. We often think an Uber driver or a bus driver or

taxi. People think taxi drivers are professionals. That's like saying I'm a professional walker

just because I've been walking all my life. I think there's an art to it. If you take it

seriously as an art form, then there's a certain way that mastery looks like. It's interesting

to think about what does mastery look like in driving? Perhaps what we associate with aggressiveness

is unnecessary. It's not part of the experience of driving. It's like unnecessary fluff,

that efficiency. You can create a good driving experience within the rules.

You're the first person to tell me this, so it's kind of interesting. I need to think about this,

but that's exactly what it felt like with Waymo. I kind of had this intuition. Maybe it's the

Russian thing. I don't know that you have to break the rules in life to get anywhere,

but maybe it's possible that that's not the case in driving. I have to think about that,

but it certainly felt that way on the streets of Phoenix when I was there in Waymo,

that that was a very pleasant experience. It wasn't frustrating in that come on,

move already kind of feeling. That wasn't there. That's what we're going after.

I don't think you have to pick one. I think truly good driving gives you both efficiency,

assertiveness, but also comfort and predictability and safety. That's what fundamental

improvements in the core capabilities truly unlock. You can kind of think of it as a precision and

recoil tradeoff. You have certain capabilities of your model, and then it's very easy when you

have some curve of precision and recoil. You can move things around. You can choose your operating

point and you're trading off precision versus recoil, false positives versus false negatives.

You can tune things on that curve and be kind of more cautious or more aggressive,

but then aggressive is bad or cautious is bad. True capabilities come from actually

moving the whole curve up, and then you are kind of on a very different plane of those tradeoffs.

That's what we're trying to do here is to move the whole curve up.

Before I forget, let's talk about trucks a little bit. I also got a chance to check out some of

the Waymo trucks. I'm not sure if we want to go too much into that space, but it's a fascinating one,

so maybe you can mention it at least briefly. Waymo is also now doing autonomous trucking,

and how different philosophically and technically is that whole space of problems?

It's one of our two big products and commercial applications of our driver, right hailing and

deliveries. We have Waymo 1 and Waymo Via, moving people and moving goods. Trucking is

an example of moving goods. We've been working on trucking since 2017. It is a very interesting

space. Another question, how different is it? It has this really nice property that the first order

challenges, like the science, the hard engineering, whether it's hardware or onboard software or

offboard software, all of the systems that you build for training your ML models for evaluating

your entire system. Those fundamentals carry over. The true challenges of driving, perception,

semantic understanding, prediction, decision making, planning, evaluation, the simulator,

ML infrastructure, those carry over. The data and the application and the domains might be

different, but the most difficult problems all of that carries over between the domains. That's

very nice. That's how we approach it. We build investing in the core, the technical core,

and then there's specialization of that core technology to different product lines, to different

commercial applications. Just to tease it apart a little bit, on trucks, starting with the hardware.

The configuration of the sensors is different. They're different physically,

geometrically, different vehicles. For example, we have two of our main lasers on the trucks

on both sides so that we don't have the blind spots, whereas on the JLR iPace, we have one of it

sitting at the very top. The actual sensors are almost the same, or largely the same.

So all of the investment that over the years we've put into building our custom lighters,

custom radars, putting the whole system together, that carries over very nicely.

Then on the perception side, the fundamental challenges of seeing, understanding the world,

whether it's object detection, classification, tracking, semantic understanding, all that

carries over. Yes, there's some specialization when you're driving on freeways, range becomes

more important. The domain is a little bit different. But again, the fundamentals carry over

very, very nicely. Same, you get into prediction or decision making. The fundamentals of what it

takes to predict what other people are going to do, to find the long tail, to improve your system

and that long tail of behavior prediction and response, that carries over and so on and so on.

So, I mean, that's pretty exciting. By the way, does Waymovia include using the smaller vehicles

for transportation of goods? That's an interesting distinction. So I would say

there's three interesting modes of operation. So one is moving humans, one is moving goods,

and one is like moving nothing, zero occupancy, meaning like you're going to

the destination, your empty vehicle. I mean, it's... The third is the last way,

if that's the entirety of it, it's the last exciting from the commercial perspective.

Well, I mean, in terms of like, if you think about what's inside a vehicle as it's moving,

because it does some significant fraction of the vehicle's movement has to be empty.

I mean, it's kind of fascinating. Maybe just on that small point, is there different

control and like policies that are applied for a zero occupancy vehicle? So a vehicle with nothing

in it, or is it just move as if there is a person inside? What was with some subtle differences?

As a first order approximation, there are no differences. And if you think about safety and

comfort and quality of driving, only part of it has to do with the people or the goods inside

of the vehicle. But you don't want to be... You want to drive smoothly as we discussed,

not purely for the benefit of whatever you have inside the car. It's also for the benefit of

the people outside kind of feeding, feeding naturally and predictably into that whole

environment. So yes, there's some second order things you can do. You can change

your route and optimize maybe kind of your fleet things at the fleet scale. And you would take

into account whether some of your cars are actually serving a useful trip, whether with people or

with goods, whereas other cars are driving completely empty to that next valuable trip

that they're going to provide. But those are mostly second order effects.

Okay, cool. So Phoenix is an incredible place. And what you've announced in Phoenix

is kind of amazing. But that's just like one city. How do you take over the world?

I mean, I'm asking for a friend.

One step at a time.

Is that the cartoon pinky in the brain? Yeah. Okay.

But gradually is a true answer. So I think the heart of your question is...

Can you ask a better question than I ask?

You're asking a great question. Answer that one.

I'm just going to phrase it in the terms that I want to answer.

That's perfect. That's exactly right. Brilliant. Please.

Please. Where are we today? And what happens next? And what does it take to go beyond Phoenix?

And what does it take to get this technology to more places and more people around the world?

All right. So our next big area of focus is exactly that larger scale commercialization and just scaling up.

If I think about the main... And Phoenix gives us that platform.

It gives us that foundation of upon which we can build. And it's...

There are a few really challenging aspects of this whole problem that you have to pull together

in order to build the technology, in order to deploy it into the field, to go from a driverless car

to a fleet of cars that are providing a service and then all the way to commercialization.

This is what we have in Phoenix. We've taken the technology from a proof point to an actual

deployment and have taken our driver from one car to a fleet that can provide a service.

Beyond that, if I think about what it will take to scale up and deploy in more places with more

customers, I tend to think about three main dimensions, three main axes of scale. One

is the core technology, the hardware and software, core capabilities of our driver.

The second dimension is evaluation and deployment. And the third one is the product, commercial,

and operational excellence. So you can talk a bit about where we are along each one of those

three dimensions about where we are today and what will happen next. On the core technology,

on the hardware and software, together comprised of driver, we obviously have that foundation

that is providing fully driverless trips to our customers as we speak, in fact.

And we've learned a tremendous amount from that. So now what we're doing is we are incorporating all

those lessons into some pretty fundamental improvements in our core technology, both on

the hardware side and on the software side, to build a more general, more robust solution

that then will enable us to massively scale and be young Phoenix. So on the hardware side,

all of those lessons are now incorporated into this fifth generation hardware platform

that is being deployed right now. And that's the platform, the fourth generation, the thing that

we have right now driving in Phoenix, it's good enough to operate fully driverlessly,

night and day, in various speeds and various conditions. But the fifth generation is the

platform upon which we want to go to massive scale. We've really made qualitative improvements in

terms of the capability of the system, the simplicity of the architecture, the reliability

of the redundancy. It is designed to be manufacturable at very large scale and provides the right

unit economics. So that's the next big step for us on the hardware side. That's already there for

scale, the version five. That's right. Is that a coincidence or should we look into a conspiracy

theory that it's the same version as the Pixel phone? Is that what's the hardware thing?

I can neither confirm nor deluxe. All right, cool. So sorry. So that's the, okay, that's

that axis. What else? So similarly, hardware is a very discreet jump. But you know,

similar to how we're making that change from the fourth generation hardware to the fifth,

we're making similar improvements on the software side to make it more robust and

more general and allow us to quickly scale beyond Phoenix. So that's the first dimension

of core technology. The second dimension is evaluation and deployment. How do you

measure your system? How do you evaluate it? How do you build the release and deployment process

where with confidence, you can regularly release new versions of your driver into a fleet?

How do you get good at it so that it is not a huge tax on your researchers and engineers?

So how do you build all these processes, the frameworks, the simulation, the evaluation,

the data science, the validation so that people can focus on improving the system

and kind of the releases just go out the door and get deployed across the fleet.

So we've gotten really good at that in Phoenix. That's been a tremendously difficult problem.

But that's what we have in Phoenix right now that gives us that foundation. And now we're

working on kind of incorporating all the lessons that we've learned to make it more efficient to

go to new places and scale up and just kind of stamp things out. So that's that second dimension

of evaluation and deployment. And the third dimension is product commercial and operational

excellence. And again, Phoenix there is providing an incredibly valuable platform.

That's why we're doing things end to end in Phoenix. We're learning as we discussed a little

earlier today, tremendous amount of really valuable lessons from our users getting really

incredible feedback. And we'll continue to iterate on that and incorporate all those lessons into

making our product even better and more convenient for our users.

So you're converting this whole process of Phoenix in Phoenix into something that could

be copy and pasted elsewhere. So like, perhaps you didn't think of it that way when you were doing

the experimentation in Phoenix. But so how long did you basically, you can correct me, but you've,

I mean, it's still early days, but you've taken a full journey in Phoenix, right?

As you were saying, of like what it takes to basically automate, I mean, it's not the entirety of

Phoenix, right? But I imagine it can encompass the entirety of Phoenix at some, some near term

date, but that's not even perhaps important. Like as long as it's a large enough geographic area.

So what, how copy pasteable is that process currently? And how like, you know, like when

you copy and paste in, in, in Google docs, I think now in, or in Word, you can like apply source

formatting or apply destination formatting. So when you copy and paste the Phoenix into like,

say, Boston, how do you apply the destination formatting? Like how much of the core of the

entire process of bringing an actual public transportation, autonomous transportation

service to a city is there in Phoenix that you understand enough to copy and paste into Boston

or wherever. So we're not quite there yet. We're not at a point where we're kind of massively

copy and pasting all over the place. But Phoenix, what we did in Phoenix, and we very intentionally

have chosen Phoenix as our first full deployment area, you know, exactly for that reason to kind

of tease the problem apart, look at each dimension, you know, focus on the fundamentals of complexity

and de risking those dimensions, and then bringing the entire thing together to get all the way and

force ourselves to learn all those hard lessons on this technology hardware and software on the

evaluation deployment on operating a service operating a business using actually serving

our customers all the way so that we're fully informed about the most difficult, most important

challenges to get us to that next step of massive copy and pasting, as, as you said. And

that's what we're doing right now. We're incorporating all those things that we learned

into that next system that then will allow us to kind of copy and paste all over the place and to

massively scale to, you know, more users and more locations. I mean, you know, just talked a little

bit about, you know, what does that mean along those different dimensions. So on the hardware

side, for example, again, it's that switch from the fourth to the fifth generation. And the fifth

generation is designed to kind of have that property. Can you say what other cities you're

thinking about? Like, I'm thinking about, sorry, when San Francisco now, I thought I want to move

to San Francisco. But I'm thinking about moving to Austin. I don't know why people are not being

very nice about San Francisco currently. For maybe it's a small, it's maybe it's in vogue right now.

But Austin seems I visited there and it was, I was in a Walmart. It's funny, these moments,

like turn your life, there's this very nice woman with kind eyes, just like stopped and said,

you look so handsome in that tie, honey, to me, this is never happening to me in my life, but

just the sweetness of this woman is something I've never experienced, certainly in the streets of

Boston. But even in San Francisco, where people wouldn't, that's just not how they speak or think.

I don't know, there's a warmth to Austin that love. And since Waymo does have a little bit of a

history there, is that a possibility? Is this your version of asking the question of like,

you know, Dimitri, I know you can't share your commercial and deployment roadmap,

but I'm thinking about moving to San Francisco of Austin, like in a blink twice, if you think I

should move to it. Yeah, that's true. That's true. You got me. Well, you know, we've been testing

in all over the place. I think we've been testing in more than 25 cities. We drive in San Francisco,

we drive in, you know, Michigan for snow. We are doing significant amount of testing

in the Bay Area, including San Francisco. Which is not like, because we're talking about the

very different thing, which is like a full on large geographic area, public service.

You can't share. Okay.

What about Moscow? When is that happening? Take on Yandex. I'm not paying attention to those folks.

They're doing, you know, there's a lot of fun. I mean, maybe as a way of a question,

you didn't speak to sort of like policy or like, is there tricky things with government and so on?

Like, is there other friction that you've encountered, except sort of technological friction

of solving this very difficult problem? Is there other stuff that you have to overcome

when deploying a public service in a city? That's interesting.

It's very important. So we put significant effort in creating those partnerships and,

you know, those relationships with governments at all levels, you know, local governments,

municipalities, you know, state level, federal level. We've been engaged in very deep conversations

from the earliest days of our projects. Whenever at all of these levels, you know, whenever we go

to test or, you know, operate in a new area, you know, we always lead with the conversation

with the local officials. But the result of that investment is that, no, it's not challenges we

have to overcome, but it is a very important that we continue to have this conversation.

Oh, yeah. I love politicians too. Okay. So Mr. Elon Musk said that LiDAR is a crutch.

What are your thoughts?

I wouldn't characterize it exactly that way. I know I think LiDAR is very important. It is a key

sensor that, you know, we use just like other modalities, right? As we discussed, our cars use

cameras, LiDARs and radars. They are all very important. They are at kind of the physical level.

They are very different. They have very different, you know, physical characteristics.

Cameras are passive. LiDARs and radars are active. You use different wavelengths.

So that means they complement each other very nicely. And they, you know, together combine.

They can be used to build a much safer and much more capable system. So, you know,

to me, it's more of a question, you know, why the heck would you handicap yourself and not use one

or more of those sensing modalities when they, you know, undoubtedly just make your system

more capable and safer. Now, it, you know, what might make sense for one product or one business

might not make sense for another one. So if you're talking about driver assist technologies,

you make certain design decisions and you make certain tradeoffs. And you make different ones

if you are, you know, building a driver that you deploy in fully driverless vehicles.

And, you know, and LiDAR specifically, when this question comes up, I, you know, typically the

criticisms that I hear are, you know, the counterpoints that cost and aesthetics. And I don't

find either of those honestly very compelling. So on the cost side, there's nothing fundamentally

prohibitive about, you know, the cost of LiDARs. You know, radars used to be very expensive before

people started, you know, before people made certain advances in technology and you started to

manufacture them massive scale and deploy them in vehicles, right? You know, similar with LiDARs.

And this is where the LiDARs that we have on our car, especially the fifth generation,

you know, we've been able to make some pretty qualitative discontinuous jumps in terms of the

fundamental technology that allow us to, you know, manufacture those things at very significant scale

and add a fraction of the cost of your both our previous generation, as well as a fraction of

the cost of, you know, what might be available on the market, you know, off the shelf right now.

And, you know, that improvement will continue. So I think, you know, cost is not a real issue.

Second one is, you know, aesthetics. You know, I don't think that's, you know, a real issue either.

Beauty is an eye of the beholder. You can make LiDAR sexy again.

I think you're exactly right. I think it is sexy. Like, honestly, I think form is a function.

Well, okay. You know, I was actually somebody brought this up to me. I mean, all forms of

LiDAR, even, even like the ones that are like big, you can make look, I mean, you can make look

beautiful. Like there's no sense in which you can't integrate into design. Like there's all

kinds of awesome designs. I don't think small and humble is beautiful. It could be like,

you know, brutalism or like it could be like harsh corners. I mean, like I said, like hot rods,

like I don't like, I don't necessarily like, like, oh man, I'm going to start so much controversy

with this. I don't like Porsches. Okay. The Porsche 911, like everyone says the most beautiful.

No, it, no, it's like, it's like a baby car. It doesn't make any sense. But everyone, it's

beauties in the eye of the beholder. You're already looking at me like, what's this kid talking about?

I'm happy to talk about you're digging your own hole. The form and function and my take on the beauty

of the hardware that we put on our vehicles. You know, I will not comment on the Porsche.

You know, Porsche monologues. Okay. All right. So, but aesthetics, fine. But there's an underlying

like philosophical question behind the kind of LiDAR question is like, how much of the problem

can be solved with computer vision, with machine learning? So I think without sort of disagreements

and so on, it's nice to put it on the spectrum because Waymo is doing a lot of machine learning as

well. It's interesting to think how much of driving if we look at five years, 10 years, 50 years down

the road would can be learned in almost more and more and more end to end way. If you look at what

Tesla is doing with as a machine learning problem, they're doing a multitask learning

thing where it's just they break up driving into a bunch of learning tasks and they have

one single neural network and they're just collecting huge amounts of data that's training that.

I've recently hung out with George Hots. I don't know if you know George.

I love him so much. He's just an entertaining human being. We were off mic talking about

Hunter S. Thompson. He's the Hunter S. Thompson of the time I was driving. Okay. So he I didn't

realize this with comma AI, but they're like really trying to end to end. They're the machine

like looking at the machine learning problem. They're really not doing multitask learning,

but it's it's it's computing the drivable area as a machine learning task and hoping that like

down the line, this level two system as driver assistance will eventually lead to allowing

you to have a fully autonomous vehicle. Okay. There's an underlying deep philosophical question

there, technical question of how much of driving can be learned. So LiDAR is an effective tool today

for actually deploying a successful service in Phoenix, right? That's safe, that's reliable,

etc, etc. But the the question and I'm not saying you can't do machine learning on LiDAR,

but the question is that like how much of driving can be learned eventually? Can we do fully

autonomous that's learned? Yeah. You know, learning is all over the place and play is the

key role in every part of our system. As you said, I would, you know, decouple the sensing

modalities from the, you know, ML and the software parts of it. LiDAR, radar, cameras,

it's all machine learning. All of the object detection classification, of course, I go that

that's what, you know, these modern deep nuts and con nuts are very good at. You feed them raw data,

massive amounts of raw data. And that's actually what our custom build LiDARs and

radars are really good at. And radars, they don't just give you point estimates of, you know,

objects in space, they give you raw, like physical observations. And then you take all

of that raw information, you know, there's colors of the pixels, whether it's, you know,

LiDAR's returns and some auxiliary information, it's not just distance, right? And, you know,

angle and distance is much richer information that you get from those returns, plus really rich

information from the radars, you fuse it all together and you feed it into those massive ML

models that then, you know, lead to the best results in terms of, you know, object detection,

classification, you know, state estimation. So there's a side to interrupt, but there is a fusion.

I mean, that's something that people didn't do for a very long time, which is like at the

sensor fusion level, I guess, like early on fusing the information together, whether

so that the the sensory information that the vehicle receives from the different modalities,

or even from different cameras is combined before it is fed into the machine learning models.

Yeah. So I think this is one of the trends. You're seeing more of that. You mentioned N10.

There's different interpretations of N10. There is kind of the purest interpretation. I'm going

to like have one model that goes from raw sensor data to like, you know, steering torque and,

you know, gas brakes. But, you know, that's too much. I don't think that's the right way to do it.

There's more, you know, smaller versions of N10, where you're kind of doing more end to end learning

or core training or depropagation of kind of signals back and forth across the different stages

of your system. There's, you know, really good ways. It gets into some fairly complex design

choices, where on one hand you want modularity and the composability, the composability of your

system. But on the other hand, you don't want to create interfaces that are too narrow or too

brittle to engineered, where you're giving up on the generality of a solution, or you're unable

to properly propagate signal, you know, reach signal forward and losses and, you know, back

so you can optimize the whole system jointly. So I would decouple, and I guess what you're seeing

in terms of the fusion of the sensing data from different modalities, as well as kind of fusion

at in the temporal level, going more from, you know, frame by frame, where, you know, you would

have one net that would do frame by frame detection and camera and then, you know, something that does

frame by frame and lighter and then radar. And then you fuse it, you know, in a weaker engineered

way later, like the field over the last decade has been evolving in more kind of joint fusion,

more end to end models that are solving some of these tasks, you know, jointly. And there's

tremendous power in that. And, you know, that's the progression that you kind of are, you know,

our stack has been on as well. Now, you know, so I would decouple the kind of sensing and how

that information is used from the role of ML and the entire stack. And, you know, I guess it's,

there's tradeoffs and modularity and how do you inject inductive bias into your system?

All right, this is, there's tremendous power in being able to do that. So, you know, we have,

there's no part of our system that is not heavily, that does not heavily, you know, leverage,

you know, data driven development or, you know, state of the art of ML. But there's mapping,

there's simulator, or there's perception, you know, object level, you know, perception,

whether it's semantic understanding, prediction, decision making, you know, so forth and so on.

It's, and of course, object detection and classification, like finding pedestrians and

cars and cyclists and, you know, cones and signs and vegetation and being very good at

estimating kind of detection classification and state estimation, there's just stable stakes,

like, like that's step zero of this whole stack. You can be incredibly good at that,

whether you use cameras or light as a radar, but that's just, you know, that's stable stakes,

that's just step zero. Beyond that, you get into the really interesting challenges of

semantic understanding of the perception level, you get into scene level reasoning,

you get into very deep problems that have to do with prediction and joint prediction and

interaction, so the interaction between all the actors in the environment, pedestrians, cyclists,

other cars, and you get into decision making, right? So how do you build a lot of systems?

So we leverage ML very heavily in all of these components. I do believe that the best results

you achieve by kind of using a hybrid approach and having different types of ML,

having different models with different degrees of inductive bias that you can have,

and combining kind of model, you know, free approaches with some, you know, model based

approaches and some rule based, physics based systems. So, you know, one example I can give you

is traffic lights. There's a problem of the detection of traffic light state and obviously

that's a great problem for, you know, computer vision, confinates are, you know, that's their

bread and butter, right? That's how you build that. But then the interpretation of, you know,

of a traffic light that you're gonna need to learn that, right? You read, you don't need to

build some, you know, complex ML model that, you know, infers with some, you know, precision

and recall that read means stop. Like, it was a, it's a very clear engineered signal with very

clear semantics, right? So you want to induce that bias, like how you induce that bias and that,

whether, you know, it's a constraint or a cost, you know, function in your stack. But like,

it is important to be able to inject that like clear semantic signal into your stack. And,

you know, that's what we do. And, but then the question of like, and that's when you apply it

to yourself, when you are making decisions, whether you want to stop for a red light,

you know, or not. But if you think about how other people treat traffic lights, we're back to the

ML version of that. Because, you know, they're supposed to stop for a red light, but that doesn't

mean they will. So then you're back in the like very heavy ML domain where you're picking up on

like very subtle keys about, you know, they have to do with the behavior of objects and pedestrians,

cyclists, cars, and the whole thing, you know, the entire configuration of the scene that allow

you to make accurate predictions on whether they will in fact stop or run a red light.

So it sounds like already for Waymo, like machine learning is a huge part of the stack. So it's a

huge part of like, not just, so obviously, the first level zero or whatever you said, which is

like just the object detection of things that, you know, know that machine learning can do,

but also starting to do prediction behavior and so on to model the what other or the other parties

in the scene, entities in the scene are going to do. So machine learning is more and more

playing a role in that as well. Of course. Oh, absolutely. I think we've been going back to

the earliest days, like DARPA, or even the DARPA grand challenge, and team was leveraging, you

know, machine learning was like pre, you know, image nut and I was very different type of ML,

but and I think actually was it was before my time, but the Stanford team on during the grand

challenge had a very interesting machine learned system that would, you know, use lighter and camera

when driving in the desert. And it we had build the model where it would kind of extend the range

of free space reasoning, we get a clear signal from lighter. And then it had a model said,

hey, like this stuff and camera kind of sort of looks like this stuff and lighter. And I know

this stuff and that I've seen in lighter, I'm very confident of this free space. So let me extend

that free space zone into the camera range that would allow the vehicle to drive faster.

And then we've been building on top of that and kind of staying and pushing the state of the art

in ML in all kinds of different ML over the years. And in fact, from the earliest days, I think,

you know, 2010, probably the year where Google, maybe 2011, probably got pretty heavily involved

in machine learning, kind of deep nuts. And at that time, it's probably the only company that

was very heavily investing in kind of state of the art ML and self driving cars, right? And they

go hand in hand. And we've been on that journey ever since we're doing pushing a lot of these

areas in terms of research, you know, at Waymo and we collaborate very heavily with the researchers

in alphabet. And I call kinds of ML, supervised ML, unsupervised ML, you know, publish some

interesting research papers in the space, especially recently, it's just a super super

active learning as well. Yeah, super super active. Of course, there's kind of the more

mature stuff, like, you know, convenants for, you know, object detection. But there's some

really interesting, really active work that's happening in more and bigger models and models

that have more structure to them, you know, not just large bitmaps and reason about temporal

sequences. And some of the interesting breakthroughs that you've, you know, we've seen in language

models, right, you know, transformers, you know, GPT three and friends. There's some really

interesting applications of some of the core breakthroughs to those problems of, you know,

behavior prediction, as well as, you know, decision making and planning, right? You can

think about it kind of the behavior, how, you know, the path of trajectories, the how people drive,

they have kind of a share a lot of the fundamental structure, you know, this problem. There's,

you know, sequential, you know, nature, there's a lot of structure. In this representation,

there is a strong locality, kind of like in sentences, you know, words that follow each other,

they're strongly connected. But there's also kind of larger context that doesn't have that

locality. And you also see that in driving, right? What's happening in the scene as a whole

has very strong implications on, you know, the kind of the next step in that sequence where

whether you're predicting what other people are going to do, whether you're making your own decisions,

or whether in the simulator, you're building generative models of, you know,

humans walking, cyclists riding and other cars driving.

Oh, that's, that's all really fascinating. Like how it's fascinating to think that, uh,

transforming models and all this, all the breakthroughs in language and NLP that might

be applicable to like driving at the higher level at the behavioral level. That's kind of fascinating.

Let me ask about pesky little creatures called pedestrians and cyclists.

They seem so humans are a problem if we can get rid of them, I would.

But unfortunately, they're also a source of joy and love and beauty. So let's keep them around.

They're also our customers. Oh, for your perspective, yes, yes, for sure.

There's also some money. Very good. Um, but I don't even know where I was going. Oh,

yes, pedestrians and cyclists. Uh, I, you know, they're a fascinating injection into the system of

uncertainty of, um, of like a game theoretic dance of what to do. And also,

uh, they have perceptions of their own and they can tweet about your product. So you

don't want to run them over from that perspective. Uh, I mean, I don't know, I'm, I'm joking a lot,

but I think in seriousness, like, you know, pedestrians are a complicated, um, uh, computer

vision problem, a complicated behavioral problem. Is there something interesting you could say about

we, what you've learned from a machine learning perspective, from also an autonomous vehicle

and a product perspective about just interacting with the humans in this world?

Yeah. Just, you know, to state on record, we care deeply about the safety of pedestrians,

you know, even the ones that don't have Twitter accounts. Um, thank you. All right, cool. Not me.

But yes, I, I'm glad, I'm glad somebody does. Okay.

Uh, but you know, in all seriousness, safety of, uh, vulnerable road users, uh, pedestrians

or cyclists is one of our highest priorities. Uh, we do a tremendous amount of testing, uh,

and validation and put a very significant emphasis on, you know, the capabilities of our systems that

have to do with safety around those unprotected vulnerable road users. Um, you know, cars just,

you know, we discussed earlier in Phoenix, we have completely empty cars, completely driverless

cars, you know, driving in this very large area. Uh, and you know, some people use them to, you

know, go to school. So they'll drive through school zones, right? So, uh, kids are kind of the very

special class of those vulnerable user road users, right? You want to be super, super safe, uh, and

super, super cautious around those. So we take it very, very, very seriously. Um, and you know,

what does it take, uh, to, uh, be good at it? Uh, you know, an incredible amount of, uh, performance

across your whole stack, you know, starts with hardware. Uh, and again, you want to use all

something of modalities available to you. Imagine driving on a residential road at night and kind

of making a turn and you don't have, you know, headlights covering some part of the space and

like, you know, a kid might run out and you know, lighters are amazing at that. Uh, they see just

as well in complete darkness as they do during the day, right? So just again, it gives you that

extra, uh, uh, you know, margin in terms of, you know, capability and performance and safety and

quality. Uh, and in fact, we oftentimes, uh, in these kinds of situations, we have our system

detect something in some cases even earlier than our trained operators in the car might do,

especially, you know, in conditions like, you know, very dark nights. Um, so starts with sensing,

then, you know, perception has to be incredibly good. And you have to be very, very good at kind

of detecting, uh, pedestrians, uh, in all kinds of situations and all kinds of environments,

including, you know, people in weird poses, uh, people kind of running, uh, around and, you know,

being partially occluded. Um, uh, so, you know, that, that's stuff number one, right? Then you

have to have in very high accuracy and very low latency in terms of your reactions, uh, to, you

know, what, you know, these, uh, actors might do, right? And we've put a tremendous amount of

engineering and tremendous amount of validation in to make sure our system performs, uh, properly.

And, you know, oftentimes it does require very strong reaction to do the safe thing. And, you

know, we actually see a lot of cases like that. That's the long tail of really rare, you know,

really, uh, you know, crazy events, uh, that, um, contribute to the safety around pedestrians. Like

one, one example that comes to mind that we actually got happened, uh, in Phoenix where we

were, uh, driving, uh, along and I think it was a 45 mile per hour road. So, you know, pretty high

speed traffic and there was a sidewalk next to it and there was a cyclist on the sidewalk. And

as, uh, we were in the right lane and right next to the site, uh, so it was a multi lane road.

Uh, so as we got close, uh, to the cyclist on the sidewalk, uh, it was a woman, you know, she

tripped and fell, just, you know, fell right into the path of our vehicle. Right. Um, and our, you

know, car, uh, uh, you know, this was actually with a test driver, our test drivers, uh, uh, did

exactly the right thing. Uh, they kind of reacted and came to stop and requires both very strong

steering and, uh, you know, strong application of the brake. Uh, and then we simulated what our

system would have done in that situation and it did, you know, exactly the same thing. It, uh,

and that, that speaks to all of those components of really good, uh, state estimation and tracking

and like imagine, you know, a person on a bike and they're falling over and they're doing that

right in front of you. Right. So you have to be really like, things are changing. The appearance

of that whole, uh, thing is changing. Right. And a person goes one way. They're falling on the road.

They're, you know, being flat on the ground in front of you, you know, the bike goes flying the

other direction. Like the two objects that used to be one, they're now, uh, are splitting apart

and the car has to like detect all of that. Uh, like milliseconds matter and it doesn't, you know,

it's not good enough to just break. You have to like steer and break and there's traffic around

you. So like it all has to come together and it was really great, uh, to see in this case in other

cases like that, uh, that we're actually seeing in the wild that our system is, you know, performing

exactly the way, uh, that we would have liked and is able to, you know, avoid, uh, collisions like

this. It's such an exciting space for robotics. Like in the, in that split second to make decisions

of life and death, I don't know if the stakes are high in a sense, but it's also beautiful that,

um, um, for somebody who loves artificial intelligence, the possibility that an AI system

might be able to save a human life. Uh, that's kind of exciting as a, as a problem. Like to wake up,

you get, it's terrifying probably from an, for an engineer to wake up and to think about,

but it's also exciting because it's like, it's, it's in your hands. Let me try to ask a question

that's often brought up about autonomous vehicles and, uh, it might be fun to see if you have

anything, anything interesting to say, which is about the trolley problem. So, uh, the trolley

problem is a interesting philosophical construct of, uh, that highlights and there's many others

like it of the difficult ethical decisions that, uh, we humans have before us in this complicated

world. Uh, so the specifically is the choice between if you were forced to choose, uh, to kill

a group X of people versus a group Y of people, like one person, if you didn't, if you did nothing,

you would kill one person. But if you, you would kill five people. And if you decide to

swerve out of the way, you would only kill one person. Do you do nothing or you choose to do

something and you can construct all kinds of sort of ethical experiments of this kind that,

uh, I think at least on a positive note, inspire you to think about like introspect

what are the, the physics of our morality. And there's usually not good answers there.

I think people love it because it's just an exciting thing to think about. I think people

who build autonomous vehicles usually roll their eyes because, uh, this is not,

this one as constructed, this like literally never comes up in reality. You never have to

choose between killing one or like one of two groups of people. But I wonder if you can speak to,

is there some, something interesting to use an engineer of autonomous vehicles that's within

the trolley problem? Or maybe more generally, are there difficult ethical decisions that you find

that, uh, algorithm must make on the specific version of the trolley problem? Which one would

you do if you're driving? The question itself is a profound question because we humans ourselves

cannot answer it. And that's the very point. Uh, I will kill both. Um, you know, humans,

I think you're exactly right. And that, you know, humans are not particularly good. I think

they kind of phrased as a, like, what would a computer do? But like humans are not very good.

And actually oftentimes think that, you know, freezing and kind of not doing anything because

like you've taken a few extra milliseconds to just process and then you end up like doing the worst

of the possible outcomes, right? So I do think that as you've pointed out, it can be a bit of

a distraction and it can be a bit of a kind of red herring. I think it's an interesting

philosophy, you know, discussion in the realm of philosophy, right? But in terms of what,

you know, how that affects the actual engineering and deployment of self driving vehicles, I,

um, it's not how you go about building a system, right? We have talked about how you engineer a

system, how you go about evaluating the different components and the, you know, the safety of the

entire thing. How do you kind of inject the, you know, various model based safety based

arguments and like, yes, your reason at parts of the system, you know, your reason about the

probability of a collision, the severity of that collision, right? And that is incorporated. And

there's, you know, you have to properly reason about the uncertainty that flows through the system,

right? So, you know, those, you know, factors definitely play a role in how the cars don't

behave, but they tend to be more of like the immersion behavior. And what you see, like,

you're absolutely right, that these, you know, clear theoretical problems that they, you know,

you don't occur to that in the system and really kind of being back to our previous discussion

of like, what, you know, what, you know, which one do you choose? Well, you know, oftentimes,

like, you made a mistake earlier, like, you shouldn't be in that situation in the first place,

right? And in reality, the system comes up. If you build a very good safe and capable driver,

you have enough, you know, clues in the environment that you drive defensively,

so you don't put yourself in that situation, right? And again, you know, it has, you know,

this, if you go back to that analogy of, you know, precision and recoil, like, okay, you can make a,

you know, very hard tradeoff off, but like neither answer is really good. But what instead you focus

on is kind of moving the whole curve up, and then you focus on building the right capability and

the right defensive driving so that, you know, you don't put yourself in a situation like this.

I don't know if you have a good answer for this, but people love it when I ask this question

about books. Are there books in your life that you've enjoyed philosophical fiction,

technical, that had a big impact on you as an engineer or as a human being, you know,

everything from science fiction to a favorite textbook? Is there three books that stand out

that you can think of? Three books. So I would, you know, that impacted me. I would say,

and this one is, you probably know it well, but not generally well known. I think in the U.S.

or kind of internationally, The Master and Margarita. It's one of actually my favorite books. It is,

you know, by Russian, it's a novel by Russian author Mikhail Bulgakov. And it's just, it's a

great book. It's one of those books that you can like reread your entire life. And it's very

accessible. You can read it as a kid. And like, it's, you know, the plot is interesting. It's,

you know, the devil, you know, visiting the Soviet Union. But it, like, you read it, reread it at

different stages of your life. And you enjoy it for different, very different reasons. And you

keep finding like deeper and deeper meaning. And, you know, it kind of affected, you know,

had a, definitely had an imprint on me, you know, mostly from the, probably kind of the cultural,

stylistic aspect, like it makes you one of those books that, you know, is good and makes you

think, but also has like this really, you know, silly, quirky, dark sense of, you know, humor.

Okay. It captures the Russian soul. That's more than many, perhaps many other books. On that,

like, slight note, just out of curiosity, one of the saddest things is I've read that book

in English. Did you, by chance, read it in English or in Russian?

In Russian, only in Russian. And I actually, that is a question I had.

Post myself every once in a while. I wonder how well it translates, if it translates at all.

And there's the language aspect of it. And then there's the cultural aspect. So I,

and actually, I'm not sure if, you know, either of those would work well in English.

Now, I forget their names, but so when the COVID lifts a little bit, I'm traveling to Paris,

for, for several reasons. One is just, I've never been to Paris. I want to go to Paris,

but there's the most famous translators of the Stiyovsky Tolstoy of most of Russian literature

live there. There's a couple, they're famous, a man and a woman. And I'm going to sort of have

a series of conversations with them. And in preparation for that, I'm starting to read

Stiyovsky in Russian. So I'm really embarrassed to say that I've read this, everything I've

read in Russian literature of like serious depth has been in English, even though I can also read,

I mean, obviously in Russian, but for some reason, it seemed in the optimization of life,

it seemed the improper decision to do, to read in Russian, like, you know,

like I don't need to, I need to think in English, not in Russian, but now I'm changing my mind

on that. And so the question of how well it translates is a really fundamental one, like

even with Dostoyevsky. So for what I understand, Dostoyevsky translates easier. Others don't

as much. Obviously the poetry doesn't translate as well. I'm also the music of big fan of Vladimir

Vosotsky. He doesn't obviously translate well. People have tried. But mastermind, I don't know,

I don't know about that one. I just know it in English, you know, as fun as hell in English.

So, so, but it's a curious question. And I want to study it rigorously from both the machine

learning aspect. And also because I want to do a couple of interviews in Russia, that I'm still

unsure of how to properly conduct an interview across a language barrier. It's a fascinating

question that ultimately communicates to an American audience. There's a few Russian people that

I think are truly special human beings. And I feel like I sometimes encounter this with

some incredible scientists and maybe you encounter this as well at some point in your life that

it feels like because of language barrier, their ideas are lost to history. It's a sad thing. I

think about like Chinese scientists or even authors that like, that we don't in English

speaking world don't get to appreciate some like the depth of the culture because it's

lost in translation. And I feel like I would love to show that to the world. Like I'm just

some idiot. But because I have this, like at least some semblance of skill in speaking Russian,

I feel like and I know how to record stuff on a video camera. I feel like I want to catch like

Gregorio Perlman who's a mathematician. I'm not sure if you're familiar with him. I want to talk

to him like he's a fascinating mind and to bring him to a wider audience in English speaking,

it'll be fascinating. But that requires to be rigorous about this question of how well

Bogakov translates. I mean, I know it's a silly concept, but it's a fundamental one

because how do you translate? And that's the thing that Google Translate is also facing

as a more machine learning problem. But I wonder as a more bigger problem for AI,

how do we capture the magic that's there in the language?

I think that's a really interesting, really challenging problem. If you do read it,

master in Russian and be curious to get your opinion. And I think part of it is language,

but part of it is just centuries of culture. The cultures are different, so it's hard to

connect that. Okay, so that was my first one, right? You had two more. The second one I would

probably pick the science fiction by the Strogosky brothers. It's up there with Isaac

Asimov and Ray Bradbury and Company. The Strogosky brothers kind of appealed more to me. I think

more, it made more of an impression on me growing up. I apologize if I'm showing my complete ignorance.

I'm so weak on sci fi. What are they, right? Oh, roadside picnic. Hard to be a god. Beedle

in an anthill. Monday starts on Saturday. It's not just science fiction. It also has very

interesting interpersonal and societal questions. And some of the language is just completely

hilarious. That's right. That's right. Oh, interesting. Monday starts on Saturday.

So I need to read. Oh, boy. You put that in the category of science fiction.

That one is, I mean, this was more of a silly, humorous work. I mean, there is kind of

profound too, right? Science fiction, right, is about this research institute. It has

deep parallels to serious research, but the setting, of course, is that they're working on

magic, right? And that's their style, right? And other books are very different, right? Hard

to be a god, right? It's about this higher society being injected into this primitive world and how

they operate there. Some of the very deep ethical questions there, right? And they've got this full

spectrum. Some is more about a kind of more adventurous style. But I enjoy all of their

books. There's probably a couple. Actually, one, I think that they consider their most important

work. I think it's the snail on a hill. I don't know exactly how it translates. I tried reading

a couple of times. I still don't get it. But everything else I fully enjoyed. And like for

one of my birthdays as a kid, I got their entire collection occupied a giant shelf in my room.

And then over the holidays, my parents couldn't drag me out of the room. And I read the whole

thing cover to cover. And I really enjoyed it. And that's one more. For the third one,

maybe a little bit darker. But it comes to mind is Orwell's 1984.

And I asked what made an impression on me and the books that people should read. That one,

I think, falls in the category of both. Now, definitely, it's one of those books that you

read and you just kind of put it down and you stare in space for a while.

Yeah, that kind of work. I think there's lessons there people should not ignore.

Nowadays, with everything that's happening in the world, I can't help it. But I have my mind jump

to some parallels with what Orwell described. And there's this whole concept of double think

and ignoring logic and holding completely contradictory opinions in your mind and not

have that not bother you and stick into the party line at all costs. There's something there.

If anything, 2020 has taught me. And I'm a huge fan of Animal Farm, which is a kind of friendly

as a friend of 1984 by Orwell. It's kind of another thought experiment of how our society

may go in directions that we wouldn't like it to go. But if anything, that's been kind of

heartbreaking to an optimist about 2020, is that that society is kind of fragile. Like, we have

this. This is a special little experiment we have going on. And not it's not unbreakable.

Like, we should be careful to preserve whatever the special thing we have going on. I mean,

I think 1984 and in these books, Brave New World, they're helpful in thinking stuff can go wrong

in non obvious ways. And it's up to us to preserve it. And it's a responsibility. It's been weighing

heavy on me. Because like, for some reason, like, more than my mom follows me on Twitter. And I feel

like I have I have like now somehow responsibility to do this world. And it dawned on me that like

me and millions of others are like the little ants that maintain this little colony. Right.

So we have a responsibility not to be, I don't know what the right analogy is, but put a flamethrower

to the place. We want to not do that. And there's interesting, complicated ways of doing that as

1984 shows. It could be through bureaucracy, it could be through incompetence, it could be through

misinformation, it could be through division and toxicity. I'm a huge believer in like that love

will be the somehow the solution. So love and robots. Love and robots. Yeah. I think you're

exactly right. Unfortunately, I think it's less of a flamethrower type of an extra. I think it's

more of a in many cases, can be more of a slow boil. And that that's the danger.

Let me ask, it's a fun thing to make a world class roboticist, engineer, and leader uncomfortable

with a ridiculous question about life. What is the meaning of life? It's a main tree from a robotics

and a human perspective. You only have a couple of minutes or one minute to answer. So

I don't know if that makes it more difficult or easier.

You know, they're very tempted to quote one of the stories by Isaac Asimov, actually titled

appropriately titled The Last Question, short story, where the plot is that humans build this

supercomputer, this AI intelligence, and once it gets powerful enough, they pose this question to

it. How can the entropy in the universe be reduced? So the computer replies, hang on,

as of yet, insufficient information to give a meaningful answer. And then thousands of years

go by and they keep posing the same question. The computer gets more and more powerful and

keeps giving the same answer. As of yet, insufficient information to give a meaningful

answer or something along those lines. And then it keeps happening and happening fast forward

like millions of years into the future and billions of years. And at some point, it's just the only

entity in the universe. It's absorbed all humanity and all knowledge in the universe. And it keeps

posing the same question to itself. And finally, it gets to the point where it is able to answer

that question. But of course, at that point, there's the heat death of the universe has occurred,

and that's the only entity. And there's nobody else to provide that answer to. So the only thing

it can do is to answer it by demonstration. So it recreates the Big Bang and resets the clock.

Right? But I can try to give a different version of the answer. Maybe not on the

behalf of all humanity. I think that might be a little presumptuous for me to speak about the

meaning of life on the behalf of all humans. But at least, personally, it changes. I think if

you think about what gives you and your life meaning and purpose and what drives you,

it seems to change over time in the lifespan of your existence. When you just enter this world,

it's all about new experiences. You get new smells, new sounds, new emotions. And that's what's

driving you. You're experiencing new amazing things. And that's magical. That's pretty awesome.

That gives you kind of a meaning. Then you get a little bit older. You start more intentionally

learning about things. I guess actually, before you start intentionally learning,

probably fun. Fun is a thing that gives you kind of meaning and purpose and the thing you optimize

for. And fun is good. Then you start learning. And I guess that this joy of

comprehension and discovery is another thing that gives you meaning and purpose and drives you.

You learn enough stuff and you want to give some of it back. So impact and contributions back to

technology or society, people, local or more globally becomes a new thing that drives a lot

of your behavior and something that gives you purpose and that you derive positive feedback

from. Then you go on and so forth. You go through various stages of life. If you have kids,

that definitely changes your perspective on things. I have three that definitely flips

some bits in your head in terms of what you care about and what you optimize for and what

matters, what doesn't matter. And so on and so forth. It seems to me that it's all of those

things. As you go through life, you want these to be additive. New experiences, fun, learning

impact. You want to be accumulating. I don't want to stop having fun or experiencing new things.

And I think it's important that it just becomes additive as opposed to a replacement or subtraction.

Those fewest problems as far as I got, but ask me in a few years, I might have one or two more

to get to the list. And before you know it, time is up just like it is for this conversation.

But hopefully it was a fun ride. It was a huge honor to meet you. As you know, I've been a

fan of yours and a fan of Google self driving car and Waymo for a long time. I can't wait.

I mean, it's one of the most exciting. If we look back in the 21st century, I truly believe

it'll be one of the most exciting things we descendants of apes have created on this earth.

So I'm a huge fan and I can't wait to see what you do next. Thanks so much for talking to me.

Thanks. Thanks for having me. And it's also a huge fan.

Thanks for listening to this conversation. For reading through summaries of books, better help,

online therapy with a licensed professional and cash app, the app I use to send money to friends.

Please check out these sponsors in the description to get a discount and to support this podcast.

If you enjoy this thing, subscribe on YouTube, review it with five stars and upper podcast,

follow on Spotify, support on Patreon or connect with me on Twitter at Lex Friedman.

And now let me leave you with some words from Isaac Asimov. Science can amuse and

fascinate us all, but it is engineering that changes the world. Thank you for listening

and hope to see you next time.