back to indexVijay Kumar: Flying Robots | Lex Fridman Podcast #37
link |
The following is a conversation with Vijay Kumar.
link |
He's one of the top roboticists in the world,
link |
a professor at the University of Pennsylvania,
link |
a dean of pen engineering, former director of Grasp Lab,
link |
or the General Robotics Automation Sensing
link |
and Perception Laboratory at Penn,
link |
that was established back in 1979, that's 40 years ago.
link |
Vijay is perhaps best known for his work
link |
in multi robot systems, robot swarms,
link |
and micro aerial vehicles,
link |
robots that elegantly cooperate in flight
link |
under all the uncertainty and challenges
link |
that the real world conditions present.
link |
This is the Artificial Intelligence Podcast.
link |
If you enjoy it, subscribe on YouTube,
link |
give it five stars on iTunes, support on Patreon,
link |
or simply connect with me on Twitter
link |
at Lex Friedman, spelled F R I D M A N.
link |
And now, here's my conversation with Vijay Kumar.
link |
What is the first robot you've ever built
link |
or were a part of building?
link |
Way back when I was in graduate school,
link |
I was part of a fairly big project
link |
that involved building a very large hexapod.
link |
It's weighed close to 7,000 pounds,
link |
and it was powered by hydraulic actuation,
link |
or it was actuated by hydraulics with 18 motors,
link |
hydraulic motors, each controlled by an Intel 8085 processor
link |
and an 8086 co processor.
link |
And so imagine this huge monster that had 18 joints,
link |
each controlled by an independent computer,
link |
and there was a 19th computer that actually did
link |
the coordination between these 18 joints.
link |
So I was part of this project,
link |
and my thesis work was how do you coordinate the 18 legs?
link |
And in particular, the pressures in the hydraulic cylinders
link |
to get efficient locomotion.
link |
It sounds like a giant mess.
link |
So how difficult is it to make all the motors communicate?
link |
Presumably, you have to send signals hundreds of times
link |
a second, or at least.
link |
So this was not my work,
link |
but the folks who worked on this wrote what I believe
link |
to be the first multiprocessor operating system.
link |
This was in the 80s, and you had to make sure
link |
that obviously messages got across
link |
from one joint to another.
link |
You have to remember the clock speeds on those computers
link |
were about half a megahertz.
link |
So not to romanticize the notion,
link |
but how did it make you feel to see that robot move?
link |
In hindsight, it looks like, well, we built this thing
link |
which really should have been much smaller.
link |
And of course, today's robots are much smaller.
link |
You look at Boston Dynamics or Ghost Robotics,
link |
a spinoff from Penn.
link |
But back then, you were stuck with the substrate you had,
link |
the compute you had, so things were unnecessarily big.
link |
But at the same time, and this is just human psychology,
link |
somehow bigger means grander.
link |
People never had the same appreciation
link |
for nanotechnology or nanodevices
link |
as they do for the Space Shuttle or the Boeing 747.
link |
Yeah, you've actually done quite a good job
link |
at illustrating that small is beautiful
link |
in terms of robotics.
link |
So what is on that topic is the most beautiful
link |
or elegant robot in motion that you've ever seen?
link |
Not to pick favorites or whatever,
link |
but something that just inspires you that you remember.
link |
Well, I think the thing that I'm most proud of
link |
that my students have done is really think about
link |
small UAVs that can maneuver in constrained spaces
link |
and in particular, their ability to coordinate
link |
with each other and form three dimensional patterns.
link |
So once you can do that,
link |
you can essentially create 3D objects in the sky
link |
and you can deform these objects on the fly.
link |
So in some sense, your toolbox of what you can create
link |
has suddenly got enhanced.
link |
And before that, we did the two dimensional version of this.
link |
So we had ground robots forming patterns and so on.
link |
So that was not as impressive, that was not as beautiful.
link |
But if you do it in 3D,
link |
suspended in midair, and you've got to go back to 2011
link |
when we did this, now it's actually pretty standard
link |
to do these things eight years later.
link |
But back then it was a big accomplishment.
link |
So the distributed cooperation
link |
is where beauty emerges in your eyes?
link |
Well, I think beauty to an engineer is very different
link |
from beauty to someone who's looking at robots
link |
from the outside, if you will.
link |
But what I meant there, so before we said that grand,
link |
so before we said that grand is associated with size.
link |
And another way of thinking about this
link |
is just the physical shape
link |
and the idea that you can get physical shapes in midair
link |
and have them deform, that's beautiful.
link |
But the individual components,
link |
the agility is beautiful too, right?
link |
So then how quickly can you actually manipulate
link |
these three dimensional shapes
link |
and the individual components?
link |
Yes, you're right.
link |
But by the way, you said UAV, unmanned aerial vehicle.
link |
What's a good term for drones, UAVs, quad copters?
link |
Is there a term that's being standardized?
link |
I don't know if there is.
link |
Everybody wants to use the word drones.
link |
And I've often said this, drones to me is a pejorative word.
link |
It signifies something that's dumb,
link |
that's pre programmed, that does one little thing
link |
and robots are anything but drones.
link |
So I actually don't like that word,
link |
but that's what everybody uses.
link |
You could call it unpiloted.
link |
But even unpiloted could be radio controlled,
link |
could be remotely controlled in many different ways.
link |
And I think the right word is,
link |
thinking about it as an aerial robot.
link |
You also say agile, autonomous, aerial robot, right?
link |
Yeah, so agility is an attribute, but they don't have to be.
link |
So what biological system,
link |
because you've also drawn a lot of inspiration with those.
link |
I've seen bees and ants that you've talked about.
link |
What living creatures have you found to be most inspiring
link |
as an engineer, instructive in your work in robotics?
link |
To me, so ants are really quite incredible creatures, right?
link |
So you, I mean, the individuals arguably are very simple
link |
in how they're built and yet they're incredibly resilient
link |
And as individuals, they're incredibly robust.
link |
So, if you take an ant, it's six legs,
link |
you remove one leg, it still works just fine.
link |
And it moves along.
link |
And I don't know that he even realizes it's lost a leg.
link |
So that's the robustness at the individual ant level.
link |
But then you look about this instinct
link |
for self preservation of the colonies
link |
and they adapt in so many amazing ways.
link |
You know, transcending gaps by just chaining themselves
link |
together when you have a flood,
link |
being able to recruit other teammates
link |
to carry big morsels of food,
link |
and then going out in different directions looking for food,
link |
and then being able to demonstrate consensus,
link |
even though they don't communicate directly with each other
link |
the way we communicate with each other.
link |
In some sense, they also know how to do democracy,
link |
probably better than what we do.
link |
Yeah, somehow it's even democracy is emergent.
link |
It seems like all of the phenomena that we see
link |
It seems like there's no centralized communicator.
link |
There is, so I think a lot is made about that word,
link |
emergent, and it means lots of things to different people.
link |
But you're absolutely right.
link |
I think as an engineer, you think about
link |
what element, elemental behaviors
link |
were primitives you could synthesize
link |
so that the whole looks incredibly powerful,
link |
incredibly synergistic,
link |
the whole definitely being greater than some of the parts,
link |
and ants are living proof of that.
link |
So when you see these beautiful swarms
link |
where there's biological systems of robots,
link |
do you sometimes think of them
link |
as a single individual living intelligent organism?
link |
So it's the same as thinking of our human beings
link |
are human civilization as one organism,
link |
or do you still, as an engineer,
link |
think about the individual components
link |
and all the engineering
link |
that went into the individual components?
link |
Well, that's very interesting.
link |
So again, philosophically as engineers,
link |
what we wanna do is to go beyond
link |
the individual components, the individual units,
link |
and think about it as a unit, as a cohesive unit,
link |
without worrying about the individual components.
link |
If you start obsessing about
link |
the individual building blocks and what they do,
link |
you inevitably will find it hard to scale up.
link |
Just mathematically,
link |
just think about individual things you wanna model,
link |
and if you want to have 10 of those,
link |
then you essentially are taking Cartesian products
link |
of 10 things, and that makes it really complicated.
link |
Then to do any kind of synthesis or design
link |
in that high dimension space is really hard.
link |
So the right way to do this
link |
is to think about the individuals in a clever way
link |
so that at the higher level,
link |
when you look at lots and lots of them,
link |
abstractly, you can think of them
link |
in some low dimensional space.
link |
So what does that involve?
link |
For the individual, do you have to try to make
link |
the way they see the world as local as possible?
link |
And the other thing,
link |
do you just have to make them robust to collisions?
link |
Like you said with the ants,
link |
if something fails, the whole swarm doesn't fail.
link |
Right, I think as engineers, we do this.
link |
I mean, you think about, we build planes,
link |
or we build iPhones,
link |
and we know that by taking individual components,
link |
well engineered components with well specified interfaces
link |
that behave in a predictable way,
link |
you can build complex systems.
link |
So that's ingrained, I would claim,
link |
in most engineers thinking,
link |
and it's true for computer scientists as well.
link |
I think what's different here is that you want
link |
the individuals to be robust in some sense,
link |
as we do in these other settings,
link |
but you also want some degree of resiliency
link |
for the population.
link |
And so you really want them to be able to reestablish
link |
communication with their neighbors.
link |
You want them to rethink their strategy for group behavior.
link |
You want them to reorganize.
link |
And that's where I think a lot of the challenges lie.
link |
So just at a high level,
link |
what does it take for a bunch of,
link |
what should we call them, flying robots,
link |
to create a formation?
link |
Just for people who are not familiar
link |
with robotics in general, how much information is needed?
link |
How do you even make it happen
link |
without a centralized controller?
link |
So, I mean, there are a couple of different ways
link |
of looking at this.
link |
If you are a purist,
link |
you think of it as a way of recreating what nature does.
link |
So nature forms groups for several reasons,
link |
but mostly it's because of this instinct
link |
that organisms have of preserving their colonies,
link |
their population, which means what?
link |
You need shelter, you need food, you need to procreate,
link |
and that's basically it.
link |
So the kinds of interactions you see are all organic.
link |
They're all local.
link |
And the only information that they share,
link |
and mostly it's indirectly, is to, again,
link |
preserve the herd or the flock,
link |
or the swarm, and either by looking for new sources of food
link |
or looking for new shelters, right?
link |
As engineers, when we build swarms, we have a mission.
link |
And when you think of a mission, and it involves mobility,
link |
most often it's described in some kind
link |
of a global coordinate system.
link |
As a human, as an operator, as a commander,
link |
or as a collaborator, I have my coordinate system,
link |
and I want the robots to be consistent with that.
link |
So I might think of it slightly differently.
link |
I might want the robots to recognize that coordinate system,
link |
which means not only do they have to think locally
link |
in terms of who their immediate neighbors are,
link |
but they have to be cognizant
link |
of what the global environment is.
link |
They have to be cognizant of what the global environment
link |
So if I say, surround this building
link |
and protect this from intruders,
link |
well, they're immediately in a building centered
link |
coordinate system, and I have to tell them
link |
where the building is.
link |
And they're globally collaborating
link |
on the map of that building.
link |
They're maintaining some kind of global,
link |
not just in the frame of the building,
link |
but there's information that's ultimately being built up
link |
explicitly as opposed to kind of implicitly,
link |
like nature might.
link |
So in some sense, nature is very, very sophisticated,
link |
but the tasks that nature solves or needs to solve
link |
are very different from the kind of engineered tasks,
link |
artificial tasks that we are forced to address.
link |
And again, there's nothing preventing us
link |
from solving these other problems,
link |
but ultimately it's about impact.
link |
You want these swarms to do something useful.
link |
And so you're kind of driven into this very unnatural,
link |
Unnatural, meaning not like how nature does, setting.
link |
And it's probably a little bit more expensive
link |
to do it the way nature does,
link |
because nature is less sensitive
link |
to the loss of the individual.
link |
And cost wise in robotics,
link |
I think you're more sensitive to losing individuals.
link |
I think that's true, although if you look at the price
link |
to performance ratio of robotic components,
link |
it's coming down dramatically, right?
link |
It continues to come down.
link |
So I think we're asymptotically approaching the point
link |
where we would get, yeah,
link |
the cost of individuals would really become insignificant.
link |
So let's step back at a high level view,
link |
the impossible question of what kind of, as an overview,
link |
what kind of autonomous flying vehicles
link |
are there in general?
link |
I think the ones that receive a lot of notoriety
link |
are obviously the military vehicles.
link |
Military vehicles are controlled by a base station,
link |
but have a lot of human supervision.
link |
But they have limited autonomy,
link |
which is the ability to go from point A to point B.
link |
And even the more sophisticated now,
link |
sophisticated vehicles can do autonomous takeoff
link |
And those usually have wings and they're heavy.
link |
Usually they're wings,
link |
but then there's nothing preventing us from doing this
link |
for helicopters as well.
link |
There are many military organizations
link |
that have autonomous helicopters in the same vein.
link |
And by the way, you look at autopilots and airplanes
link |
and it's actually very similar.
link |
In fact, one interesting question we can ask is,
link |
if you look at all the air safety violations,
link |
all the crashes that occurred,
link |
would they have happened if the plane were truly autonomous?
link |
And I think you'll find that in many of the cases,
link |
because of pilot error, we made silly decisions.
link |
And so in some sense, even in air traffic,
link |
commercial air traffic, there's a lot of applications,
link |
although we only see autonomy being enabled
link |
at very high altitudes when the plane is an autopilot.
link |
The plane is an autopilot.
link |
There's still a role for the human
link |
and that kind of autonomy is, you're kind of implying,
link |
I don't know what the right word is,
link |
but it's a little dumber than it could be.
link |
Right, so in the lab, of course,
link |
we can afford to be a lot more aggressive.
link |
And the question we try to ask is,
link |
can we make robots that will be able to make decisions
link |
without any kind of external infrastructure?
link |
So what does that mean?
link |
So the most common piece of infrastructure
link |
that airplanes use today is GPS.
link |
GPS is also the most brittle form of information.
link |
If you have driven in a city, try to use GPS navigation,
link |
in tall buildings, you immediately lose GPS.
link |
And so that's not a very sophisticated way
link |
of building autonomy.
link |
I think the second piece of infrastructure
link |
they rely on is communications.
link |
Again, it's very easy to jam communications.
link |
In fact, if you use wifi, you know that wifi signals
link |
drop out, cell signals drop out.
link |
So to rely on something like that is not good.
link |
The third form of infrastructure we use,
link |
and I hate to call it infrastructure,
link |
but it is that, in the sense of robots, is people.
link |
So you could rely on somebody to pilot you.
link |
And so the question you wanna ask is,
link |
if there are no pilots, there's no communications
link |
with any base station, if there's no knowledge of position,
link |
and if there's no a priori map,
link |
a priori knowledge of what the environment looks like,
link |
a priori model of what might happen in the future,
link |
can robots navigate?
link |
So that is true autonomy.
link |
So that's true autonomy, and we're talking about,
link |
you mentioned like military application of drones.
link |
Okay, so what else is there?
link |
You talk about agile, autonomous flying robots,
link |
aerial robots, so that's a different kind of,
link |
it's not winged, it's not big, at least it's small.
link |
So I use the word agility mostly,
link |
or at least we're motivated to do agile robots,
link |
mostly because robots can operate
link |
and should be operating in constrained environments.
link |
And if you want to operate the way a global hawk operates,
link |
I mean, the kinds of conditions in which you operate
link |
are very, very restrictive.
link |
If you wanna go inside a building,
link |
for example, for search and rescue,
link |
or to locate an active shooter,
link |
or you wanna navigate under the canopy in an orchard
link |
to look at health of plants,
link |
or to look for, to count fruits,
link |
to measure the tree trunks.
link |
These are things we do, by the way.
link |
There's some cool agriculture stuff you've shown
link |
in the past, it's really awesome.
link |
So in those kinds of settings, you do need that agility.
link |
Agility does not necessarily mean
link |
you break records for the 100 meters dash.
link |
What it really means is you see the unexpected
link |
and you're able to maneuver in a safe way,
link |
and in a way that gets you the most information
link |
about the thing you're trying to do.
link |
By the way, you may be the only person
link |
who, in a TED Talk, has used a math equation,
link |
which is amazing, people should go see one of your TED Talks.
link |
Actually, it's very interesting,
link |
because the TED curator, Chris Anderson,
link |
told me, you can't show math.
link |
And I thought about it, but that's who I am.
link |
I mean, that's our work.
link |
And so I felt compelled to give the audience a taste
link |
for at least some math.
link |
So on that point, simply, what does it take
link |
to make a thing with four motors fly, a quadcopter,
link |
one of these little flying robots?
link |
How hard is it to make it fly?
link |
How do you coordinate the four motors?
link |
How do you convert those motors into actual movement?
link |
So this is an interesting question.
link |
We've been trying to do this since 2000.
link |
It is a commentary on the sensors
link |
that were available back then,
link |
the computers that were available back then.
link |
And a number of things happened between 2000 and 2007.
link |
One is the advances in computing,
link |
which is, so we all know about Moore's Law,
link |
but I think 2007 was a tipping point,
link |
the year of the iPhone, the year of the cloud.
link |
Lots of things happened in 2007.
link |
But going back even further,
link |
inertial measurement units as a sensor really matured.
link |
Again, lots of reasons for that.
link |
Certainly, there's a lot of federal funding,
link |
particularly DARPA in the US,
link |
but they didn't anticipate this boom in IMUs.
link |
But if you look, subsequently what happened
link |
is that every car manufacturer had to put an airbag in,
link |
which meant you had to have an accelerometer on board.
link |
And so that drove down the price to performance ratio.
link |
Wow, I should know this.
link |
That's very interesting.
link |
That's very interesting, the connection there.
link |
And that's why research is very,
link |
it's very hard to predict the outcomes.
link |
And again, the federal government spent a ton of money
link |
on things that they thought were useful for resonators,
link |
but it ended up enabling these small UAVs, which is great,
link |
because I could have never raised that much money
link |
and sold this project,
link |
hey, we want to build these small UAVs.
link |
Can you actually fund the development of low cost IMUs?
link |
So why do you need an IMU on an IMU?
link |
So I'll come back to that.
link |
So in 2007, 2008, we were able to build these.
link |
And then the question you're asking was a good one.
link |
How do you coordinate the motors to develop this?
link |
But over the last 10 years, everything is commoditized.
link |
A high school kid today can pick up
link |
a Raspberry Pi kit and build this.
link |
All the low levels functionality is all automated.
link |
But basically at some level,
link |
you have to drive the motors at the right RPMs,
link |
the right velocity,
link |
in order to generate the right amount of thrust,
link |
in order to position it and orient it in a way
link |
that you need to in order to fly.
link |
The feedback that you get is from onboard sensors,
link |
and the IMU is an important part of it.
link |
The IMU tells you what the acceleration is,
link |
as well as what the angular velocity is.
link |
And those are important pieces of information.
link |
In addition to that, you need some kind of local position
link |
or velocity information.
link |
For example, when we walk,
link |
we implicitly have this information
link |
because we kind of know what our stride length is.
link |
We also are looking at images fly past our retina,
link |
if you will, and so we can estimate velocity.
link |
We also have accelerometers in our head,
link |
and we're able to integrate all these pieces of information
link |
to determine where we are as we walk.
link |
And so robots have to do something very similar.
link |
You need an IMU, you need some kind of a camera
link |
or other sensor that's measuring velocity,
link |
and then you need some kind of a global reference frame
link |
if you really want to think about doing something
link |
in a world coordinate system.
link |
And so how do you estimate your position
link |
with respect to that global reference frame?
link |
That's important as well.
link |
So coordinating the RPMs of the four motors
link |
is what allows you to, first of all, fly and hover,
link |
and then you can change the orientation
link |
and the velocity and so on.
link |
So it's a bunch of degrees of freedom
link |
that you're complaining about.
link |
There's six degrees of freedom,
link |
but you only have four inputs, the four motors.
link |
And it turns out to be a remarkably versatile configuration.
link |
You think at first, well, I only have four motors,
link |
how do I go sideways?
link |
But it's not too hard to say, well, if I tilt myself,
link |
I can go sideways, and then you have four motors
link |
pointing up, how do I rotate in place
link |
about a vertical axis?
link |
Well, you rotate them at different speeds
link |
and that generates reaction moments
link |
and that allows you to turn.
link |
So it's actually a pretty, it's an optimal configuration
link |
from an engineer standpoint.
link |
It's very simple, very cleverly done, and very versatile.
link |
So if you could step back to a time,
link |
so I've always known flying robots as,
link |
to me, it was natural that a quadcopter should fly.
link |
But when you first started working with it,
link |
how surprised are you that you can make,
link |
do so much with the four motors?
link |
How surprising is it that you can make this thing fly,
link |
first of all, that you can make it hover,
link |
that you can add control to it?
link |
Firstly, this is not, the four motor configuration
link |
You can, it has at least a hundred year history.
link |
And various people, various people try to get quadrotors
link |
to fly without much success.
link |
As I said, we've been working on this since 2000.
link |
Our first designs were, well, this is way too complicated.
link |
Why not we try to get an omnidirectional flying robot?
link |
So our early designs, we had eight rotors.
link |
And so these eight rotors were arranged uniformly
link |
on a sphere, if you will.
link |
So you can imagine a symmetric configuration.
link |
And so you should be able to fly anywhere.
link |
But the real challenge we had is the strength to weight ratio
link |
And of course, we didn't have the sensors and so on.
link |
So everybody knew, or at least the people
link |
who worked with rotorcrafts knew,
link |
four rotors will get it done.
link |
So that was not our idea.
link |
But it took a while before we could actually do
link |
the onboard sensing and the computation that was needed
link |
for the kinds of agile maneuvering that we wanted to do
link |
in our little aerial robots.
link |
And that only happened between 2007 and 2009 in our lab.
link |
Yeah, and you have to send the signal
link |
maybe a hundred times a second.
link |
So the compute there, everything has to come down in price.
link |
And what are the steps of getting from point A to point B?
link |
So we just talked about like local control.
link |
But if all the kind of cool dancing in the air
link |
that I've seen you show, how do you make it happen?
link |
How do you make a trajectory?
link |
First of all, okay, figure out a trajectory.
link |
So plan a trajectory.
link |
And then how do you make that trajectory happen?
link |
Yeah, I think planning is a very fundamental problem
link |
I think 10 years ago it was an esoteric thing,
link |
but today with self driving cars,
link |
everybody can understand this basic idea
link |
that a car sees a whole bunch of things
link |
and it has to keep a lane or maybe make a right turn
link |
It has to plan a trajectory.
link |
It has to be safe.
link |
It has to be efficient.
link |
So everybody's familiar with that.
link |
That's kind of the first step that you have to think about
link |
when you say autonomy.
link |
And so for us, it's about finding smooth motions,
link |
motions that are safe.
link |
So we think about these two things.
link |
One is optimality, one is safety.
link |
Clearly you cannot compromise safety.
link |
So you're looking for safe, optimal motions.
link |
The other thing you have to think about is
link |
can you actually compute a reasonable trajectory
link |
in a small amount of time?
link |
Cause you have a time budget.
link |
So the optimal becomes suboptimal,
link |
but in our lab we focus on synthesizing smooth trajectory
link |
that satisfy all the constraints.
link |
In other words, don't violate any safety constraints
link |
and is as efficient as possible.
link |
And when I say efficient,
link |
it could mean I want to get from point A to point B
link |
as quickly as possible,
link |
or I want to get to it as gracefully as possible,
link |
or I want to consume as little energy as possible.
link |
But always staying within the safety constraints.
link |
But yes, always finding a safe trajectory.
link |
So there's a lot of excitement and progress
link |
in the field of machine learning
link |
and reinforcement learning
link |
and the neural network variant of that
link |
with deep reinforcement learning.
link |
Do you see a role of machine learning
link |
in, so a lot of the success of flying robots
link |
did not rely on machine learning,
link |
except for maybe a little bit of the perception
link |
on the computer vision side.
link |
On the control side and the planning,
link |
do you see there's a role in the future
link |
for machine learning?
link |
So let me disagree a little bit with you.
link |
I think we never perhaps called out in my work,
link |
called out learning,
link |
but even this very simple idea of being able to fly
link |
through a constrained space.
link |
The first time you try it, you'll invariably,
link |
you might get it wrong if the task is challenging.
link |
And the reason is to get it perfectly right,
link |
you have to model everything in the environment.
link |
And flying is notoriously hard to model.
link |
There are aerodynamic effects that we constantly discover.
link |
Even just before I was talking to you,
link |
I was talking to a student about how blades flap
link |
And that ends up changing how a rotorcraft
link |
is accelerated in the angular direction.
link |
Does he use like micro flaps or something?
link |
It's not micro flaps.
link |
So we assume that each blade is rigid,
link |
but actually it flaps a little bit.
link |
Interesting, yeah.
link |
And so the models rely on the fact,
link |
on the assumption that they're not rigid.
link |
On the assumption that they're actually rigid,
link |
but that's not true.
link |
If you're flying really quickly,
link |
these effects become significant.
link |
If you're flying close to the ground,
link |
you get pushed off by the ground, right?
link |
Something which every pilot knows when he tries to land
link |
or she tries to land, this is called a ground effect.
link |
Something very few pilots think about
link |
is what happens when you go close to a ceiling
link |
or you get sucked into a ceiling.
link |
There are very few aircrafts
link |
that fly close to any kind of ceiling.
link |
Likewise, when you go close to a wall,
link |
there are these wall effects.
link |
And if you've gone on a train
link |
and you pass another train that's traveling
link |
in the opposite direction, you feel the buffeting.
link |
And so these kinds of microclimates
link |
affect our UAV significantly.
link |
And they're impossible to model, essentially.
link |
I wouldn't say they're impossible to model,
link |
but the level of sophistication you would need
link |
in the model and the software would be tremendous.
link |
Plus, to get everything right would be awfully tedious.
link |
So the way we do this is over time,
link |
we figure out how to adapt to these conditions.
link |
So early on, we use the form of learning
link |
that we call iterative learning.
link |
So this idea, if you want to perform a task,
link |
there are a few things that you need to change
link |
and iterate over a few parameters
link |
that over time you can figure out.
link |
So I could call it policy gradient reinforcement learning,
link |
but actually it was just iterative learning.
link |
Iterative learning.
link |
And so this was there way back.
link |
I think what's interesting is,
link |
if you look at autonomous vehicles today,
link |
learning occurs, could occur in two pieces.
link |
One is perception, understanding the world.
link |
Second is action, taking actions.
link |
Everything that I've seen that is successful
link |
is on the perception side of things.
link |
So in computer vision,
link |
we've made amazing strides in the last 10 years.
link |
So recognizing objects, actually detecting objects,
link |
classifying them and tagging them in some sense,
link |
This is all done through machine learning.
link |
On the action side, on the other hand,
link |
I don't know of any examples
link |
where there are fielded systems
link |
where we actually learn
link |
the right behavior.
link |
Outside of single demonstration is successful.
link |
In the laboratory, this is the holy grail.
link |
Can you do end to end learning?
link |
Can you go from pixels to motor currents?
link |
This is really, really hard.
link |
And I think if you go forward,
link |
the right way to think about these things
link |
is data driven approaches,
link |
learning based approaches,
link |
in concert with model based approaches,
link |
which is the traditional way of doing things.
link |
So I think there's a piece,
link |
there's a role for each of these methodologies.
link |
So what do you think,
link |
just jumping out on topic
link |
since you mentioned autonomous vehicles,
link |
what do you think are the limits on the perception side?
link |
So I've talked to Elon Musk
link |
and there on the perception side,
link |
they're using primarily computer vision
link |
to perceive the environment.
link |
In your work with,
link |
because you work with the real world a lot
link |
and the physical world,
link |
what are the limits of computer vision?
link |
Do you think we can solve autonomous vehicles
link |
on the perception side,
link |
focusing on vision alone and machine learning?
link |
So, we also have a spinoff company,
link |
Exxon Technologies that works underground in mines.
link |
So you go into mines, they're dark, they're dirty.
link |
You fly in a dirty area,
link |
there's stuff you kick up from by the propellers,
link |
the downwash kicks up dust.
link |
I challenge you to get a computer vision algorithm
link |
So we use LIDARs in that setting.
link |
Indoors and even outdoors when we fly through fields,
link |
I think there's a lot of potential
link |
for just solving the problem using computer vision alone.
link |
But I think the bigger question is,
link |
can you actually solve
link |
or can you actually identify all the corner cases
link |
using a single sensing modality and using learning alone?
link |
So what's your intuition there?
link |
So look, if you have a corner case
link |
and your algorithm doesn't work,
link |
your instinct is to go get data about the corner case
link |
and patch it up, learn how to deal with that corner case.
link |
But at some point, this is gonna saturate,
link |
this approach is not viable.
link |
So today, computer vision algorithms can detect
link |
90% of the objects or can detect objects 90% of the time,
link |
classify them 90% of the time.
link |
Cats on the internet probably can do 95%, I don't know.
link |
But to get from 90% to 99%, you need a lot more data.
link |
And then I tell you, well, that's not enough
link |
because I have a safety critical application,
link |
I wanna go from 99% to 99.9%.
link |
That's even more data.
link |
So I think if you look at wanting accuracy on the X axis
link |
and look at the amount of data on the Y axis,
link |
I believe that curve is an exponential curve.
link |
Wow, okay, it's even hard if it's linear.
link |
It's hard if it's linear, totally,
link |
but I think it's exponential.
link |
And the other thing you have to think about
link |
is that this process is a very, very power hungry process
link |
to run data farms or servers.
link |
Power, do you mean literally power?
link |
Literally power, literally power.
link |
So in 2014, five years ago, and I don't have more recent data,
link |
2% of US electricity consumption was from data farms.
link |
So we think about this as an information science
link |
and information processing problem.
link |
Actually, it is an energy processing problem.
link |
And so unless we figured out better ways of doing this,
link |
I don't think this is viable.
link |
So talking about driving, which is a safety critical application
link |
and some aspect of flight is safety critical,
link |
maybe philosophical question, maybe an engineering one,
link |
what problem do you think is harder to solve,
link |
autonomous driving or autonomous flight?
link |
That's a really interesting question.
link |
I think autonomous flight has several advantages
link |
that autonomous driving doesn't have.
link |
So look, if I want to go from point A to point B,
link |
I have a very, very safe trajectory.
link |
Go vertically up to a maximum altitude,
link |
fly horizontally to just about the destination,
link |
and then come down vertically.
link |
This is preprogrammed.
link |
The equivalent of that is very hard to find
link |
in the self driving car world because you're on the ground,
link |
you're in a two dimensional surface,
link |
and the trajectories on the two dimensional surface
link |
are more likely to encounter obstacles.
link |
I mean this in an intuitive sense, but mathematically true.
link |
That's mathematically as well, that's true.
link |
There's other option on the 2G space of platooning,
link |
or because there's so many obstacles,
link |
you can connect with those obstacles
link |
and all these kind of options.
link |
Sure, but those exist in the three dimensional space as well.
link |
So the question also implies how difficult are obstacles
link |
in the three dimensional space in flight?
link |
So that's the downside.
link |
I think in three dimensional space,
link |
you're modeling three dimensional world,
link |
not just because you want to avoid it,
link |
but you want to reason about it,
link |
and you want to work in the three dimensional environment,
link |
and that's significantly harder.
link |
So that's one disadvantage.
link |
I think the second disadvantage is of course,
link |
anytime you fly, you have to put up
link |
with the peculiarities of aerodynamics
link |
and their complicated environments.
link |
How do you negotiate that?
link |
So that's always a problem.
link |
Do you see a time in the future where there is,
link |
you mentioned there's agriculture applications.
link |
So there's a lot of applications of flying robots,
link |
but do you see a time in the future
link |
where there's tens of thousands,
link |
or maybe hundreds of thousands of delivery drones
link |
that fill the sky, delivery flying robots?
link |
I think there's a lot of potential
link |
for the last mile delivery.
link |
And so in crowded cities, I don't know,
link |
if you go to a place like Hong Kong,
link |
just crossing the river can take half an hour,
link |
and while a drone can just do it in five minutes at most.
link |
I think you look at delivery of supplies to remote villages.
link |
I work with a nonprofit called Weave Robotics.
link |
So they work in the Peruvian Amazon,
link |
where the only highways that are available
link |
are the only highways or rivers.
link |
And to get from point A to point B may take five hours,
link |
while with a drone, you can get there in 30 minutes.
link |
So just delivering drugs,
link |
retrieving samples for testing vaccines,
link |
I think there's huge potential here.
link |
So I think the challenges are not technological,
link |
but the challenge is economical.
link |
The one thing I'll tell you that nobody thinks about
link |
is the fact that we've not made huge strides
link |
in battery technology.
link |
Yes, it's true, batteries are becoming less expensive
link |
because we have these mega factories that are coming up,
link |
but they're all based on lithium based technologies.
link |
And if you look at the energy density
link |
and the power density,
link |
those are two fundamentally limiting numbers.
link |
So power density is important
link |
because for a UAV to take off vertically into the air,
link |
which most drones do, they don't have a runway,
link |
you consume roughly 200 watts per kilo at the small size.
link |
That's a lot, right?
link |
In contrast, the human brain consumes less than 80 watts,
link |
the whole of the human brain.
link |
So just imagine just lifting yourself into the air
link |
is like two or three light bulbs,
link |
which makes no sense to me.
link |
Yeah, so you're going to have to at scale
link |
solve the energy problem then,
link |
charging the batteries, storing the energy and so on.
link |
And then the storage is the second problem,
link |
but storage limits the range.
link |
But you have to remember that you have to burn
link |
a lot of it per given time.
link |
So the burning is another problem.
link |
Which is a power question.
link |
Yes, and do you think just your intuition,
link |
there are breakthroughs in batteries on the horizon?
link |
How hard is that problem?
link |
Look, there are a lot of companies
link |
that are promising flying cars that are autonomous
link |
and that are clean.
link |
I think they're over promising.
link |
The autonomy piece is doable.
link |
The clean piece, I don't think so.
link |
There's another company that I work with called JetOptra.
link |
They make small jet engines.
link |
And they can get up to 50 miles an hour very easily
link |
and lift 50 kilos.
link |
But they're jet engines, they're efficient,
link |
they're a little louder than electric vehicles,
link |
but they can build flying cars.
link |
So your sense is that there's a lot of pieces
link |
that have come together.
link |
So on this crazy question,
link |
if you look at companies like Kitty Hawk,
link |
working on electric, so the clean,
link |
talking to Sebastian Thrun, right?
link |
It's a crazy dream, you know?
link |
But you work with flight a lot.
link |
You've mentioned before that manned flights
link |
or carrying a human body is very difficult to do.
link |
So how crazy is flying cars?
link |
Do you think there'll be a day
link |
when we have vertical takeoff and landing vehicles
link |
that are sufficiently affordable
link |
that we're going to see a huge amount of them?
link |
And they would look like something like we dream of
link |
when we think about flying cars.
link |
Yeah, like the Jetsons.
link |
The Jetsons, yeah.
link |
So look, there are a lot of smart people working on this
link |
and you never say something is not possible
link |
when you have people like Sebastian Thrun working on it.
link |
So I totally think it's viable.
link |
I question, again, the electric piece.
link |
The electric piece, yeah.
link |
And again, for short distances, you can do it.
link |
And there's no reason to suggest
link |
that these all just have to be rotorcrafts.
link |
You take off vertically,
link |
but then you morph into a forward flight.
link |
I think there are a lot of interesting designs.
link |
The question to me is, are these economically viable?
link |
And if you agree to do this with fossil fuels,
link |
it instantly immediately becomes viable.
link |
That's a real challenge.
link |
Do you think it's possible for robots and humans
link |
to collaborate successfully on tasks?
link |
So a lot of robotics folks that I talk to and work with,
link |
I mean, humans just add a giant mess to the picture.
link |
So it's best to remove them from consideration
link |
when solving specific tasks.
link |
It's very difficult to model.
link |
There's just a source of uncertainty.
link |
In your work with these agile flying robots,
link |
do you think there's a role for collaboration with humans?
link |
Or is it best to model tasks in a way
link |
that doesn't have a human in the picture?
link |
Well, I don't think we should ever think about robots
link |
without human in the picture.
link |
Ultimately, robots are there because we want them
link |
to solve problems for humans.
link |
But there's no general solution to this problem.
link |
I think if you look at human interaction
link |
and how humans interact with robots,
link |
you know, we think of these in sort of three different ways.
link |
One is the human commanding the robot.
link |
The second is the human collaborating with the robot.
link |
So for example, we work on how a robot
link |
can actually pick up things with a human and carry things.
link |
That's like true collaboration.
link |
And third, we think about humans as bystanders,
link |
self driving cars, what's the human's role
link |
and how do self driving cars
link |
acknowledge the presence of humans?
link |
So I think all of these things are different scenarios.
link |
It depends on what kind of humans, what kind of task.
link |
And I think it's very difficult to say
link |
that there's a general theory that we all have for this.
link |
But at the same time, it's also silly to say
link |
that we should think about robots independent of humans.
link |
So to me, human robot interaction
link |
is almost a mandatory aspect of everything we do.
link |
Yes, but to which degree, so your thoughts,
link |
if we jump to autonomous vehicles, for example,
link |
there's a big debate between what's called
link |
level two and level four.
link |
So semi autonomous and autonomous vehicles.
link |
And so the Tesla approach currently at least
link |
has a lot of collaboration between human and machine.
link |
So the human is supposed to actively supervise
link |
the operation of the robot.
link |
Part of the safety definition of how safe a robot is
link |
in that case is how effective is the human in monitoring it.
link |
Do you think that's ultimately not a good approach
link |
in sort of having a human in the picture,
link |
not as a bystander or part of the infrastructure,
link |
but really as part of what's required
link |
to make the system safe?
link |
This is harder than it sounds.
link |
I think, you know, if you, I mean,
link |
I'm sure you've driven before in highways and so on.
link |
It's really very hard to have to relinquish control
link |
to a machine and then take over when needed.
link |
So I think Tesla's approach is interesting
link |
because it allows you to periodically establish
link |
some kind of contact with the car.
link |
Toyota, on the other hand, is thinking about
link |
shared autonomy or collaborative autonomy as a paradigm.
link |
If I may argue, these are very, very simple ways
link |
of human robot collaboration,
link |
because the task is pretty boring.
link |
You sit in a vehicle, you go from point A to point B.
link |
I think the more interesting thing to me is,
link |
for example, search and rescue.
link |
I've got a human first responder, robot first responders.
link |
I gotta do something.
link |
I have to do it in two minutes.
link |
The building is burning.
link |
There's been an explosion.
link |
I think to me, those are the interesting things
link |
where it's very, very unstructured.
link |
And what's the role of the human?
link |
What's the role of the robot?
link |
Clearly, there's lots of interesting challenges
link |
and there's a field.
link |
I think we're gonna make a lot of progress in this area.
link |
Yeah, it's an exciting form of collaboration.
link |
In autonomous driving, the main enemy
link |
is just boredom of the human.
link |
As opposed to in rescue operations,
link |
it's literally life and death.
link |
And the collaboration enables
link |
the effective completion of the mission.
link |
In some sense, we're also doing this.
link |
You think about the human driving a car
link |
and almost invariably, the human's trying
link |
to estimate the state of the car,
link |
they estimate the state of the environment and so on.
link |
But what if the car were to estimate the state of the human?
link |
So for example, I'm sure you have a smartphone
link |
and the smartphone tries to figure out what you're doing
link |
and send you reminders and oftentimes telling you
link |
to drive to a certain place,
link |
although you have no intention of going there
link |
because it thinks that that's where you should be
link |
because of some Gmail calendar entry
link |
or something like that.
link |
And it's trying to constantly figure out who you are,
link |
what you're doing.
link |
If a car were to do that,
link |
maybe that would make the driver safer
link |
because the car is trying to figure out
link |
is the driver paying attention,
link |
looking at his or her eyes,
link |
looking at circadian movements.
link |
So I think the potential is there,
link |
but from the reverse side,
link |
it's not robot modeling, but it's human modeling.
link |
It's more on the human, right.
link |
And I think the robots can do a very good job
link |
of modeling humans if you really think about the framework
link |
that you have a human sitting in a cockpit,
link |
surrounded by sensors, all staring at him,
link |
in addition to be staring outside,
link |
but also staring at him.
link |
I think there's a real synergy there.
link |
Yeah, I love that problem
link |
because it's the new 21st century form of psychology,
link |
actually AI enabled psychology.
link |
A lot of people have sci fi inspired fears
link |
of walking robots like those from Boston Dynamics.
link |
If you just look at shows on Netflix and so on,
link |
or flying robots like those you work with,
link |
how would you, how do you think about those fears?
link |
How would you alleviate those fears?
link |
Do you have inklings, echoes of those same concerns?
link |
You know, anytime we develop a technology
link |
meaning to have positive impact in the world,
link |
there's always the worry that,
link |
you know, somebody could subvert those technologies
link |
and use it in an adversarial setting.
link |
And robotics is no exception, right?
link |
So I think it's very easy to weaponize robots.
link |
I think we talk about swarms.
link |
One thing I worry a lot about is,
link |
so, you know, for us to get swarms to work
link |
and do something reliably, it's really hard.
link |
But suppose I have this challenge
link |
of trying to destroy something,
link |
and I have a swarm of robots,
link |
where only one out of the swarm
link |
needs to get to its destination.
link |
So that suddenly becomes a lot more doable.
link |
And so I worry about, you know,
link |
this general idea of using autonomy
link |
with lots and lots of agents.
link |
I mean, having said that, look,
link |
a lot of this technology is not very mature.
link |
My favorite saying is that
link |
if somebody had to develop this technology,
link |
wouldn't you rather the good guys do it?
link |
So the good guys have a good understanding
link |
of the technology, so they can figure out
link |
how this technology is being used in a bad way,
link |
or could be used in a bad way and try to defend against it.
link |
So we think a lot about that.
link |
So we have, we're doing research
link |
on how to defend against swarms, for example.
link |
That's interesting.
link |
There's in fact a report by the National Academies
link |
on counter UAS technologies.
link |
This is a real threat,
link |
but we're also thinking about how to defend against this
link |
and knowing how swarms work.
link |
Knowing how autonomy works is, I think, very important.
link |
So it's not just politicians?
link |
Do you think engineers have a role in this discussion?
link |
I think the days where politicians
link |
can be agnostic to technology are gone.
link |
I think every politician needs to be
link |
literate in technology.
link |
And I often say technology is the new liberal art.
link |
Understanding how technology will change your life,
link |
I think is important.
link |
And every human being needs to understand that.
link |
And maybe we can elect some engineers
link |
to office as well on the other side.
link |
What are the biggest open problems in robotics?
link |
And you said we're in the early days in some sense.
link |
What are the problems we would like to solve in robotics?
link |
I think there are lots of problems, right?
link |
But I would phrase it in the following way.
link |
If you look at the robots we're building,
link |
they're still very much tailored towards
link |
doing specific tasks and specific settings.
link |
I think the question of how do you get them to operate
link |
in much broader settings
link |
where things can change in unstructured environments
link |
So think of self driving cars.
link |
Today, we can build a self driving car in a parking lot.
link |
We can do level five autonomy in a parking lot.
link |
But can you do a level five autonomy
link |
in the streets of Napoli in Italy or Mumbai in India?
link |
So in some sense, when we think about robotics,
link |
we have to think about where they're functioning,
link |
what kind of environment, what kind of a task.
link |
We have no understanding
link |
of how to put both those things together.
link |
So we're in the very early days
link |
of applying it to the physical world.
link |
And I was just in Naples actually.
link |
And there's levels of difficulty and complexity
link |
depending on which area you're applying it to.
link |
And we don't have a systematic way of understanding that.
link |
Everybody says, just because a computer
link |
can now beat a human at any board game,
link |
we certainly know something about intelligence.
link |
A computer board game is very, very structured.
link |
It is the equivalent of working in a Henry Ford factory
link |
where things, parts come, you assemble, move on.
link |
It's a very, very, very structured setting.
link |
That's the easiest thing.
link |
And we know how to do that.
link |
So you've done a lot of incredible work
link |
at the UPenn, University of Pennsylvania, GraspLab.
link |
You're now Dean of Engineering at UPenn.
link |
What advice do you have for a new bright eyed undergrad
link |
interested in robotics or AI or engineering?
link |
Well, I think there's really three things.
link |
One is you have to get used to the idea
link |
that the world will not be the same in five years
link |
or four years whenever you graduate, right?
link |
Which is really hard to do.
link |
So this thing about predicting the future,
link |
every one of us needs to be trying
link |
to predict the future always.
link |
Not because you'll be any good at it,
link |
but by thinking about it,
link |
I think you sharpen your senses and you become smarter.
link |
So that's number one.
link |
Number two, it's a corollary of the first piece,
link |
which is you really don't know what's gonna be important.
link |
So this idea that I'm gonna specialize in something
link |
which will allow me to go in a particular direction,
link |
it may be interesting,
link |
but it's important also to have this breadth
link |
so you have this jumping off point.
link |
I think the third thing,
link |
and this is where I think Penn excels.
link |
I mean, we teach engineering,
link |
but it's always in the context of the liberal arts.
link |
It's always in the context of society.
link |
As engineers, we cannot afford to lose sight of that.
link |
So I think that's important.
link |
But I think one thing that people underestimate
link |
when they do robotics
link |
is the importance of mathematical foundations,
link |
the importance of representations.
link |
Not everything can just be solved
link |
by looking for Ross packages on the internet
link |
or to find a deep neural network that works.
link |
I think the representation question is key,
link |
even to machine learning,
link |
where if you ever hope to achieve or get to explainable AI,
link |
somehow there need to be representations
link |
that you can understand.
link |
So if you wanna do robotics,
link |
you should also do mathematics.
link |
And you said liberal arts, a little literature.
link |
If you wanna build a robot,
link |
it should be reading Dostoyevsky.
link |
I agree with that.
link |
So Vijay, thank you so much for talking today.
link |
It was just a very exciting conversation.