back to indexVijay Kumar: Flying Robots | Lex Fridman Podcast #37
link |
The following is a conversation with Vijay Kumar.
link |
He's one of the top roboticists in the world,
link |
a professor at the University of Pennsylvania,
link |
a Dean of Penn Engineering,
link |
former director of Grasp Lab,
link |
or the General Robotics Automation Sensing
link |
and Perception Laboratory at Penn,
link |
that was established back in 1979, that's 40 years ago.
link |
Vijay is perhaps best known
link |
for his work in multi robot systems, robot swarms,
link |
and micro aerial vehicles.
link |
Robots that elegantly cooperate in flight
link |
under all the uncertainty and challenges
link |
that the real world conditions present.
link |
This is the Artificial Intelligence Podcast.
link |
If you enjoy it, subscribe on YouTube,
link |
give it five stars on iTunes,
link |
support it on Patreon,
link |
or simply connect with me on Twitter
link |
at Lex Freedman spelled FRID MAN.
link |
And now here's my conversation with Vijay Kumar.
link |
What is the first robot you've ever built
link |
or were a part of building?
link |
Way back when I was in graduate school,
link |
I was part of a fairly big project
link |
that involved building a very large hexapod.
link |
This weighed close to 7,000 pounds.
link |
And it was powered by hydraulic actuation,
link |
or was actuated by hydraulics with 18 motors,
link |
hydraulic motors, each controlled by an Intel 8085 processor
link |
and an 8086 co processor.
link |
And so imagine this huge monster that had 18 joints,
link |
each controlled by an independent computer.
link |
And there was a 19th computer
link |
that actually did the coordination
link |
between these 18 joints.
link |
So as part of this project,
link |
and my thesis work was,
link |
how do you coordinate the 18 legs?
link |
And in particular, the pressures in the hydraulic cylinders
link |
to get efficient locomotion.
link |
It sounds like a giant mess.
link |
So how difficult is it to make all the motors communicate?
link |
Presumably you have to send signals
link |
hundreds of times a second, or at least...
link |
This was not my work, but the folks who worked on this
link |
wrote what I believe to be
link |
the first multiprocessor operating system.
link |
This was in the 80s.
link |
And you had to make sure that obviously messages
link |
got across from one joint to another.
link |
You have to remember the clock speeds on those computers
link |
were about half a megahertz.
link |
So not to romanticize the notion,
link |
but how did it make you feel to make,
link |
to see that robot move?
link |
In hindsight, it looks like, well,
link |
we built the thing which really should have been much smaller.
link |
And of course, today's robots are much smaller.
link |
You look at, you know, Boston Dynamics,
link |
our ghost robotics has been off from pen.
link |
But back then, you were stuck
link |
with the substrate you had, the compute you had,
link |
so things were unnecessarily big.
link |
But at the same time, and this is just human psychology,
link |
somehow bigger means grander.
link |
You know, people never have the same appreciation
link |
for nanotechnology or nano devices
link |
as they do for the space shuttle or the Boeing 747.
link |
Yeah, you've actually done quite a good job
link |
at illustrating that small is beautiful
link |
in terms of robotics.
link |
So what is on that topic is the most beautiful
link |
or elegant robot emotion that you've ever seen.
link |
Not to pick favorites or whatever,
link |
but something that just inspires you that you remember.
link |
Well, I think the thing that I'm most proud of
link |
that my students have done is really think about
link |
small UAVs that can maneuver and constrain spaces
link |
and in particular, their ability to coordinate
link |
with each other and form three dimensional patterns.
link |
So once you can do that,
link |
you can essentially create 3D objects in the sky
link |
and you can deform these objects on the fly.
link |
So in some sense, your toolbox of what you can create
link |
has suddenly got enhanced.
link |
And before that, we did the two dimensional version of this.
link |
So we had ground robots forming patterns and so on.
link |
So that was not as impressive, that was not as beautiful.
link |
But if you do it in 3D, suspend it in midair
link |
and you've got to go back to 2011 when we did this.
link |
Now it's actually pretty standard to do these things
link |
eight years later, but back then it was a big accomplishment.
link |
So the distributed cooperation
link |
is where beauty emerges in your eyes?
link |
Well, I think beauty to an engineer is very different
link |
from beauty to someone who's looking at robots
link |
from the outside, if you will.
link |
But what I meant there, so before we said that grand
link |
is associated with size.
link |
And another way of thinking about this
link |
is just the physical shape and the idea
link |
that you can create physical shapes in midair
link |
and have them deform, that's beautiful.
link |
But the individual components,
link |
the agility is beautiful too, right?
link |
So then how quickly can you actually manipulate
link |
these three dimensional shapes
link |
and the individual components?
link |
Yes, you're right.
link |
Oh, by the way, said UAV, unmanned aerial vehicle.
link |
What's a good term for drones, UAVs, quadcopters?
link |
Is there a term that's being standardized?
link |
I don't know if there is.
link |
Everybody wants to use the word drones.
link |
And I've often said there's drones to me
link |
is a pejorative word.
link |
It signifies something that's dumb,
link |
a pre program that does one little thing
link |
and robots are anything but drones.
link |
So I actually don't like that word,
link |
but that's what everybody uses.
link |
You could call it unpiloted.
link |
But even unpiloted could be radio controlled,
link |
could be remotely controlled in many different ways.
link |
And I think the right word
link |
is thinking about it as an aerial robot.
link |
You also say agile, autonomous aerial robot, right?
link |
Yeah, so agility is an attribute,
link |
but they don't have to be.
link |
So what biological system,
link |
because you've also drawn a lot of inspiration
link |
with those I've seen bees and ants
link |
that you've talked about, what living creatures
link |
have you found to be most inspiring
link |
as an engineer, instructive in your work in robotics?
link |
To me, so ants are really quite incredible creatures, right?
link |
So you, I mean, the individuals arguably are very simple
link |
in how they're built, and yet they're incredibly resilient
link |
And as individuals, they're incredibly robust.
link |
So, if you take an ant with six legs, you remove one leg,
link |
it still works just fine.
link |
And it moves along,
link |
and I don't know that he even realizes it's lost a leg.
link |
So that's the robustness at the individual ant level.
link |
But then you look about this instinct
link |
for self preservation of the colonies,
link |
and they adapt in so many amazing ways,
link |
you know, transcending gaps by just chaining themselves
link |
together when you have a flood,
link |
being able to recruit other teammates
link |
to carry big morsels of food,
link |
and then going out in different directions,
link |
and then being able to demonstrate consensus,
link |
even though they don't communicate directly
link |
with each other the way we communicate with each other,
link |
in some sense, they also know how to do democracy,
link |
probably better than what we do.
link |
Yeah, somehow, even democracy is emergent.
link |
It seems like all of the phenomena
link |
that we see is all emergent.
link |
It seems like there's no centralized communicator.
link |
There is, so I think a lot is made about that word, emergent,
link |
and it means lots of things to different people,
link |
but you're absolutely right.
link |
I think as an engineer, you think about what element,
link |
elemental behaviors, what primitives you could synthesize
link |
so that the whole looks incredibly powerful,
link |
incredibly synergistic,
link |
the whole definitely being greater than some of the parts,
link |
and ants are living proof of that.
link |
So when you see these beautiful swarms
link |
where there's biological systems of robots,
link |
do you sometimes think of them
link |
as a single individual living intelligent organism?
link |
So it's the same as thinking of our human civilization
link |
as one organism, or do you still, as an engineer,
link |
think about the individual components
link |
and all the engineering that went into
link |
the individual components?
link |
Well, that's very interesting.
link |
So again, philosophically, as engineers,
link |
what we want to do is to go beyond the individual components,
link |
the individual units, and think about it as a unit,
link |
as a cohesive unit,
link |
without worrying about the individual components.
link |
If you start obsessing about the individual building blocks
link |
and what they do, you inevitably will find it hard
link |
Just mathematically,
link |
just think about individual things you want to model,
link |
and if you want to have 10 of those,
link |
then you essentially are taking Cartesian products
link |
of 10 things, and that makes it really complicated
link |
than to do any kind of synthesis or design
link |
in that high dimension space is really hard.
link |
So the right way to do this
link |
is to think about the individuals in a clever way
link |
so that at the higher level,
link |
when you look at lots and lots of them,
link |
abstractly, you can think of them
link |
in some low dimensional space.
link |
So what does that involve?
link |
For the individual, you have to try to make
link |
the way they see the world as local as possible,
link |
and the other thing,
link |
do you just have to make them robust to collisions?
link |
Like you said, with the ants,
link |
if something fails, the whole swarm doesn't fail.
link |
Right, I think as engineers, we do this.
link |
I mean, you know, think about,
link |
we build planes or we build iPhones,
link |
and we know that by taking individual components,
link |
well engineered components,
link |
with well specified interfaces
link |
that behave in a predictable way,
link |
you can build complex systems.
link |
So that's ingrained I would claim
link |
in most engineers thinking,
link |
and it's true for computer scientists as well.
link |
I think what's different here is that you want
link |
the individuals to be robust in some sense,
link |
as we do in these other settings,
link |
but you also want some degree of resiliency
link |
for the population.
link |
And so you really want them to be able
link |
to reestablish communication with their neighbors.
link |
You want them to rethink their strategy
link |
for group behavior.
link |
You want them to reorganize.
link |
And that's where I think a lot of the challenges lie.
link |
So just at a high level,
link |
what does it take for a bunch of,
link |
what should we call them,
link |
flying robots to create a formation?
link |
Just for people who are not familiar with robotics
link |
in general, how much information is needed?
link |
How do you even make it happen
link |
without a centralized controller?
link |
So I mean, there are a couple of different ways
link |
of looking at this.
link |
If you are a purist, you think of it as a way
link |
of recreating what nature does.
link |
So nature forms groups for several reasons,
link |
but mostly it's because of this instinct
link |
that organisms have of preserving their colonies,
link |
their population, which means what?
link |
You need shelter, you need food, you need to procreate,
link |
and that's basically it.
link |
So the kinds of interactions you see are all organic.
link |
They're all local.
link |
And the only information that they share,
link |
and mostly it's indirectly,
link |
is to again preserve the herd or the flock
link |
or the swarm and either by looking for new sources of food
link |
or looking for new shelters, right?
link |
As engineers, when we build swarms, we have a mission.
link |
And when you think of a mission,
link |
and it involves mobility,
link |
most often it's described in some kind
link |
of a global coordinate system.
link |
As a human, as an operator, as a commander,
link |
or as a collaborator, I have my coordinate system
link |
and I want the robots to be consistent with that.
link |
So I might think of it slightly differently.
link |
I might want the robots to recognize that coordinate system,
link |
which means not only do they have to think locally
link |
in terms of who their immediate neighbors are,
link |
but they have to be cognizant
link |
of what the global environment looks like.
link |
So if I go, if I say surround this building
link |
and protect this from intruders,
link |
well, they're immediately in a building
link |
centered coordinate system
link |
and I have to tell them where the building is.
link |
And they're globally collaborating
link |
on the map of that building.
link |
They're maintaining some kind of global,
link |
not just in the frame of the building,
link |
but there's information that's ultimately being built up
link |
explicitly as opposed to kind of implicitly,
link |
like nature might.
link |
So in some sense, nature is very, very sophisticated,
link |
but the tasks that nature solves
link |
or needs to solve are very different
link |
from the kind of engineered tasks,
link |
artificial tasks that we are forced to address.
link |
And again, there's nothing preventing us
link |
from solving these other problems,
link |
but ultimately it's about impact.
link |
You want these swarms to do something useful.
link |
And so you're kind of driven into this very unnatural,
link |
if you will, unnatural meaning, not like how nature does,
link |
And it's probably a little bit more expensive
link |
to do it the way nature does,
link |
because nature is less sensitive to the loss of the individual
link |
and cost wise in robotics,
link |
I think you're more sensitive to losing individuals.
link |
I think that's true, although if you look at the price
link |
to performance ratio of robotic components,
link |
it's coming down dramatically.
link |
Right, it continues to come down.
link |
So I think we're asymptotically approaching the point
link |
where we would get, yeah,
link |
the cost of individuals would really become insignificant.
link |
So let's step back at a high level view,
link |
the impossible question of what kind of,
link |
as an overview, what kind of autonomous flying vehicles
link |
are there in general?
link |
I think the ones that receive a lot of notoriety
link |
are obviously the military vehicles.
link |
Military vehicles are controlled by a base station,
link |
but have a lot of human supervision,
link |
but have limited autonomy,
link |
which is the ability to go from point A to point B,
link |
and even the more sophisticated vehicles
link |
can do autonomous takeoff and landing.
link |
And those usually have wings and they're heavy?
link |
Usually they're wings,
link |
but there's nothing preventing us from doing this
link |
for helicopters as well.
link |
There are many military organizations that have
link |
autonomous helicopters in the same vein.
link |
And by the way, you look at autopilots and airplanes,
link |
and it's actually very similar.
link |
In fact, one interesting question we can ask is,
link |
if you look at all the air safety violations,
link |
all the crashes that occurred,
link |
would they have happened if the plane
link |
were truly autonomous, and I think you'll find
link |
that in many of the cases,
link |
because of pilot error, we made silly decisions.
link |
And so in some sense, even in air traffic,
link |
commercial air traffic, there's a lot of applications,
link |
although we only see autonomy being enabled
link |
at very high altitudes when the plane is an autopilot.
link |
There's still a role for the human,
link |
and that kind of autonomy is, you're kind of implying,
link |
I don't know what the right word is,
link |
but it's a little dumber than it could be.
link |
Right, so in the lab, of course,
link |
we can afford to be a lot more aggressive.
link |
And the question we try to ask is,
link |
can we make robots that will be able to make decisions
link |
without any kind of external infrastructure?
link |
So what does that mean?
link |
So the most common piece of infrastructure
link |
that airplanes use today is GPS.
link |
GPS is also the most brittle form of information.
link |
If you've driven in a city, try to use GPS navigation,
link |
tall buildings, you immediately lose GPS.
link |
And so that's not a very sophisticated way
link |
of building autonomy.
link |
I think the second piece of infrastructure
link |
that I rely on is communications.
link |
Again, it's very easy to jam communications.
link |
In fact, if you use Wi Fi,
link |
you know that Wi Fi signals drop out,
link |
cell signals drop out.
link |
So to rely on something like that is not good.
link |
The third form of infrastructure we use,
link |
and I hate to call it infrastructure,
link |
but it is that in the sense of robots, it's people.
link |
So you could rely on somebody to pilot you.
link |
And so the question you wanna ask is
link |
if there are no pilots,
link |
if there's no communications with any base station,
link |
if there's no knowledge of position,
link |
and if there's no a priori map,
link |
a priori knowledge of what the environment looks like,
link |
a priori model of what might happen in the future.
link |
Can robots navigate?
link |
So that is true autonomy.
link |
So that's true autonomy.
link |
And we're talking about, you mentioned
link |
like military applications and drones.
link |
Okay, so what else is there?
link |
You talk about agile autonomous flying robots, aerial robots.
link |
So that's a different kind of, it's not winged,
link |
it's not big, at least it's small.
link |
So I use the word agility mostly,
link |
or at least we're motivated to do agile robots,
link |
mostly because robots can operate
link |
and should be operating in constrained environments.
link |
And if you want to operate the way a global hawk operates,
link |
I mean, the kinds of conditions in which you operate
link |
are very, very restrictive.
link |
If you wanna go inside a building,
link |
for example, for search and rescue,
link |
or to locate an active shooter,
link |
or you wanna navigate under the canopy in an orchard
link |
to look at health of plants,
link |
or to count fruits to measure the tree trunks.
link |
These are things we do, by the way.
link |
Yeah, some cool agriculture stuff you've shown in the past,
link |
it's really awesome.
link |
So in those kinds of settings, you do need that agility.
link |
Agility does not necessarily mean
link |
you break records for the 100 meters dash.
link |
What it really means is you see the unexpected
link |
and you're able to maneuver in a safe way,
link |
and in a way that gets you the most information
link |
about the thing you're trying to do.
link |
By the way, you may be the only person
link |
who in a TED talk has used a math equation,
link |
which is amazing, people should go see one of your TED talks.
link |
Actually, it's very interesting
link |
because the TED curator, Chris Anderson, told me,
link |
you can't show math.
link |
And I thought about it, but that's who I am.
link |
I mean, that's our work.
link |
And so I felt compelled to give the audience a taste
link |
for at least some math.
link |
So on that point, simply, what does it take
link |
to make a thing with four motors fly, a quadcopter,
link |
one of these little flying robots?
link |
How hard is it to make it fly?
link |
How do you coordinate the four motors?
link |
How do you convert those motors into actual movement?
link |
So this is an interesting question.
link |
We've been trying to do this since 2000.
link |
It is a commentary on the sensors
link |
that were available back then,
link |
and the computers that were available back then.
link |
And a number of things happened
link |
between 2000 and 2007.
link |
One is the advances in computing, which is,
link |
so we all know about Moore's Law,
link |
but I think 2007 was a tipping point,
link |
the year of the iPhone, the year of the cloud.
link |
Lots of things happened in 2007.
link |
But going back even further,
link |
inertial measurement units as a sensor really matured.
link |
Again, lots of reasons for that.
link |
Certainly there's a lot of federal funding,
link |
particularly DARPA in the US,
link |
but they didn't anticipate this boom in IMUs.
link |
But if you look subsequently,
link |
what happened is that every car manufacturer
link |
had to put an airbag in,
link |
which meant you had to have an accelerometer on board.
link |
And so that drove down the price
link |
to performance ratio of the sensors.
link |
I should know this, that's very interesting.
link |
It's very interesting, the connection there.
link |
And that's why research is very hard to predict the outcomes.
link |
And again, the federal government spent a ton of money
link |
on things that they thought were useful for resonators,
link |
but it ended up enabling these small UAVs, which is great,
link |
because I could have never raised that much money
link |
and told, sold this project,
link |
hey, we want to build these small UAVs.
link |
Can you actually fund the development of low cost IMUs?
link |
So why do you need an IMU on an IMU?
link |
So I'll come back to that,
link |
but so in 2007, 2008, we were able to build these.
link |
And then the question you're asking was a good one,
link |
how do you coordinate the motors to develop this?
link |
But over the last 10 years, everything is commoditized.
link |
A high school kid today can pick up a Raspberry Pi kit
link |
all the low levels functionality is all automated.
link |
But basically at some level, you have to drive the motors
link |
at the right RPMs, the right velocity,
link |
in order to generate the right amount of thrust
link |
in order to position it and orient it
link |
in a way that you need to in order to fly.
link |
The feedback that you get is from onboard sensors
link |
and the IMU is an important part of it.
link |
The IMU tells you what the acceleration is
link |
as well as what the angular velocity is.
link |
And those are important pieces of information.
link |
In addition to that, you need some kind of local position
link |
or velocity information.
link |
For example, when we walk,
link |
we implicitly have this information
link |
because we kind of know how, what our stride length is.
link |
We also are looking at images fly past our retina,
link |
if you will, and so we can estimate velocity.
link |
We also have accelerometers in our head
link |
and we're able to integrate all these pieces of information
link |
to determine where we are as we walk.
link |
And so robots have to do something very similar.
link |
You need an IMU, you need some kind of a camera
link |
or other sensor that's measuring velocity.
link |
And then you need some kind of a global reference frame
link |
if you really want to think about doing something
link |
in a world coordinate system.
link |
And so how do you estimate your position
link |
with respect to that global reference frame?
link |
That's important as well.
link |
So coordinating the RPMs of the four motors
link |
is what allows you to, first of all, fly and hover
link |
and then you can change the orientation
link |
and the velocity and so on.
link |
So there's a bunch of degrees of freedom
link |
or there's six degrees of freedom
link |
but you only have four inputs, the four motors.
link |
And it turns out to be a remarkably versatile configuration.
link |
You think at first, well, I only have four motors,
link |
how do I go sideways?
link |
But it's not too hard to say, well,
link |
if I tilt myself, I can go sideways.
link |
And then you have four motors pointing up,
link |
how do I rotate in place about a vertical axis?
link |
Well, you rotate them at different speeds
link |
and that generates reaction moments
link |
and that allows you to turn.
link |
So it's actually a pretty,
link |
it's an optimal configuration from an engineer standpoint.
link |
It's very simple, very cleverly done and very versatile.
link |
So if you could step back to a time,
link |
so I've always known flying robots as,
link |
to me it was natural that the quadcopters should fly.
link |
But when you first started working with it,
link |
how surprised are you that you can make,
link |
do so much with the four motors?
link |
How surprising is that you can make this thing fly,
link |
first of all, that you can make it hover,
link |
then you can add control to it?
link |
Firstly, this is not, the four motor configuration
link |
is not ours, it has at least a hundred year history.
link |
And various people,
link |
various people try to get quadrotors to fly
link |
without much success.
link |
As I said, we've been working on this since 2000.
link |
Our first designs were,
link |
well, this is way too complicated.
link |
Why not we try to get an omnidirectional flying robot?
link |
So our early designs, we had eight rotors.
link |
And so these eight rotors were arranged uniformly
link |
on a sphere, if you will.
link |
So you can imagine a symmetric configuration
link |
and so you should be able to fly anywhere.
link |
But the real challenge we had is the strength
link |
to weight ratio is not enough,
link |
and of course we didn't have the sensors and so on.
link |
So everybody knew, or at least the people
link |
who worked with rotor crafts knew,
link |
four rotors would get it done.
link |
So that was not our idea.
link |
But it took a while before we could actually do
link |
the onboard sensing and the computation
link |
that was needed for the kinds of agile maneuvering
link |
that we wanted to do in our little aerial robots.
link |
And that only happened between 2007 and 2009 in our lab.
link |
Yeah, and you have to send the signal
link |
maybe a hundred times a second.
link |
So the compute there is everything
link |
has to come down in price.
link |
And what are the steps of getting from point A to point B?
link |
So we just talked about like local control,
link |
but if all the kind of cool dancing in the air
link |
that I've seen you show, how do you make it happen?
link |
Make it trajectory, first of all, okay,
link |
figure out a trajectory, so plan a trajectory,
link |
and then how do you make that trajectory happen?
link |
I think planning is a very fundamental problem in robotics.
link |
I think 10 years ago it was an esoteric thing,
link |
but today with self driving cars,
link |
everybody can understand this basic idea
link |
that a car sees a whole bunch of things
link |
and it has to keep a lane or maybe make a right turn
link |
or switch lanes, it has to plan a trajectory,
link |
it has to be safe, it has to be efficient.
link |
So everybody's familiar with that.
link |
That's kind of the first step
link |
that you have to think about when you say autonomy.
link |
And so for us, it's about finding smooth motions,
link |
motions that are safe.
link |
So we think about these two things.
link |
One is optimality, one is safety.
link |
Clearly you cannot compromise safety.
link |
So you're looking for safe, optimal motions.
link |
The other thing you have to think about
link |
is can you actually compute a reasonable trajectory
link |
in a small amount of time, because you have a time budget.
link |
So the optimal becomes suboptimal,
link |
but in our lab we focus on synthesizing smooth trajectory
link |
that satisfy all the constraints.
link |
In other words, don't violate any safety constraints
link |
and is as efficient as possible.
link |
And when I say efficient, it could mean
link |
I want to get from point A to point B as quickly as possible
link |
or I want to get to it as gracefully as possible
link |
or I want to consume as little energy as possible.
link |
But always staying within the safety constraints.
link |
But yes, always finding a safe trajectory.
link |
So there's a lot of excitement and progress
link |
in the field of machine learning.
link |
And reinforcement learning and the neural network variant
link |
of that with deeper reinforcement learning.
link |
Do you see a role of machine learning in...
link |
So a lot of the success with flying robots
link |
did not rely on machine learning,
link |
except for maybe a little bit of the perception
link |
on the computer vision side.
link |
On the control side and the planning,
link |
do you see there's a role in the future
link |
for machine learning?
link |
So let me disagree a little bit with you.
link |
I think we never perhaps called out in my work
link |
called out learning, but even this very simple idea
link |
of being able to fly through a constrained space.
link |
The first time you try it, you'll invariably...
link |
You might get it wrong if the task is challenging.
link |
And the reason is to get it perfectly right,
link |
you have to model everything in the environment.
link |
And flying is notoriously hard to model.
link |
There are aerodynamic effects that we constantly discover,
link |
even just before I was talking to you,
link |
I was talking to a student about how blades flap when they fly.
link |
And that ends up changing how a rotorcraft
link |
is accelerated in the angular direction.
link |
Does it use like microflaps or something?
link |
It's not microflaps.
link |
We assume that each blade is rigid,
link |
but actually it flaps a little bit.
link |
Interesting, yeah.
link |
And so the models rely on the fact,
link |
on the assumption that they're actually rigid.
link |
But that's not true.
link |
If you're flying really quickly,
link |
these effects become significant.
link |
If you're flying close to the ground,
link |
you get pushed off by the ground.
link |
Something which every pilot knows when he tries to land
link |
or she tries to land, this is called a ground effect.
link |
Something very few pilots think about
link |
is what happens when you go close to a ceiling,
link |
or you get sucked into a ceiling.
link |
There are very few aircraft that fly close to any kind of ceiling.
link |
Likewise, when you go close to a wall,
link |
there are these wall effects.
link |
And if you've gone on a train and you pass another train
link |
that's traveling the opposite direction,
link |
you can feel the buffeting.
link |
And so these kinds of microclimates
link |
affect our UAVs significantly.
link |
And they're impossible to model, essentially.
link |
I wouldn't say they're impossible to model,
link |
but the level of sophistication you would need
link |
in the model and the software would be tremendous.
link |
Plus, to get everything right would be awfully tedious.
link |
So the way we do this is over time,
link |
we figure out how to adapt to these conditions.
link |
So early on, we use the form of learning
link |
that we call iterative learning.
link |
So this idea, if you want to perform a task,
link |
there are a few things that you need to change
link |
and iterate over a few parameters
link |
that over time you can figure out.
link |
So I could call it policy gradient reinforcement learning,
link |
but actually it was just iterative learning.
link |
And so this was there way back.
link |
I think what's interesting is,
link |
if you look at autonomous vehicles today,
link |
learning occurs, could occur in two pieces.
link |
One is perception, understanding the world.
link |
Second is action, taking actions.
link |
Everything that I've seen that is successful
link |
is on the perception side of things.
link |
So in computer vision, we've made amazing strides
link |
in the last 10 years.
link |
So recognizing objects, actually detecting objects,
link |
classifying them and tagging them in some sense,
link |
annotating them, this is all done through machine learning.
link |
On the action side, on the other hand,
link |
I don't know if any examples where there are fielded systems
link |
where we actually learn the right behavior.
link |
Outside of single demonstration is successful.
link |
On the laboratory, this is the Holy Grail.
link |
Can you do end to end learning?
link |
Can you go from pixels to motor currents?
link |
This is really, really hard.
link |
And I think if you go forward,
link |
the right way to think about these things
link |
is data driven approaches, learning based approaches,
link |
in concert with model based approaches,
link |
which is the traditional way of doing things.
link |
So I think there's a piece,
link |
there's a role for each of these methodologies.
link |
So what do you think, just jumping out on topics,
link |
since you mentioned autonomous vehicles,
link |
what do you think are the limits on the perception side?
link |
So I've talked to Elon Musk,
link |
and there on the perception side,
link |
they're using primarily computer vision
link |
to perceive the environment.
link |
In your work with, because you work with the real world a lot,
link |
and the physical world,
link |
what are the limits of computer vision?
link |
Do you think you can solve autonomous vehicles,
link |
focusing on the perception side,
link |
focusing on vision alone and machine learning?
link |
So we also have a spin off company, Exxon Technologies,
link |
that works underground in mines.
link |
So you go into mines, they're dark.
link |
You fly in a dirty area,
link |
there's stuff you kick up by the propellers,
link |
the downwash kicks up dust.
link |
I challenge you to get a computer vision algorithm to work there.
link |
So we use LIDARS in that setting.
link |
Indoors, and even outdoors when we fly through fields,
link |
I think there's a lot of potential
link |
for just solving the problem using computer vision alone.
link |
But I think the bigger question is,
link |
can you actually solve,
link |
or can you actually identify all the corner cases
link |
using a single sensing modality
link |
and using learning alone?
link |
What's your intuition there?
link |
So look, if you have a corner case
link |
and your algorithm doesn't work,
link |
your instinct is to go get data about the corner case
link |
and patch it up, learn how to deal with that corner case.
link |
But at some point,
link |
this is going to saturate, this approach is not viable.
link |
So today, computer vision algorithms
link |
can detect objects 90% of the time,
link |
classify them 90% of the time.
link |
Cats on the internet probably can do 95%, I don't know.
link |
But to get from 90% to 99%, you need a lot more data.
link |
And then I tell you, well, that's not enough
link |
because I have a safety critical application
link |
that want to go from 99% to 99.9%,
link |
well, that's even more data.
link |
So I think if you look at
link |
wanting accuracy on the x axis
link |
and look at the amount of data on the y axis,
link |
I believe that curve is an exponential curve.
link |
Wow, okay, it's even hard if it's linear.
link |
It's hard if it's linear, totally, but I think it's exponential.
link |
And the other thing you have to think about
link |
is that this process is a very, very power hungry process
link |
to run data farms or servers.
link |
Power, do you mean literally power?
link |
Literally power, literally power.
link |
So in 2014, five years ago, and I don't have more recent data,
link |
2% of US electricity consumption was from data farms.
link |
So we think about this as an information science
link |
and information processing problem.
link |
Actually, it is an energy processing problem.
link |
And so unless we've figured out better ways of doing this,
link |
I don't think this is viable.
link |
So talking about driving, which is a safety critical application
link |
and some aspect of the flight is safety critical,
link |
maybe philosophical question, maybe an engineering one.
link |
What problem do you think is harder to solve?
link |
Autonomous driving or autonomous flight?
link |
That's a really interesting question.
link |
I think autonomous flight has several advantages
link |
that autonomous driving doesn't have.
link |
So look, if I want to go from point A to point B,
link |
I have a very, very safe trajectory.
link |
Go vertically up to a maximum altitude,
link |
fly horizontally to just about the destination
link |
and then come down vertically.
link |
This is preprogrammed.
link |
The equivalent of that is very hard to find
link |
in a self driving car world because you're on the ground,
link |
you're in a two dimensional surface,
link |
and the trajectories on the two dimensional surface
link |
are more likely to encounter obstacles.
link |
I mean this in an intuitive sense,
link |
but mathematically true, that's...
link |
Mathematically as well, that's true.
link |
There's other option on the 2G space of platooning
link |
or because there's so many obstacles,
link |
you can connect with those obstacles
link |
and all these kinds of problems.
link |
But those exist in the three dimensional space as well.
link |
So the question also implies how difficult are obstacles
link |
in the three dimensional space in flight?
link |
So that's the downside.
link |
I think in three dimensional space,
link |
you're modeling three dimensional world,
link |
not just because you want to avoid it,
link |
but you want to reason about it
link |
and you want to work in that three dimensional environment.
link |
And that's significantly harder.
link |
So that's one disadvantage.
link |
I think the second disadvantage is of course,
link |
anytime you fly, you have to put up
link |
with the peculiarities of aerodynamics
link |
and their complicated environments.
link |
How do you negotiate that?
link |
So that's always a problem.
link |
Do you see a time in the future where there is...
link |
You mentioned there's agriculture applications.
link |
So there's a lot of applications of flying robots.
link |
But do you see a time in the future where there is tens of thousands
link |
or maybe hundreds of thousands of delivery drones
link |
that fill the sky, a delivery of flying robots?
link |
I think there's a lot of potential for the last mile delivery.
link |
And so in crowded cities,
link |
I don't know if you go to a place like Hong Kong,
link |
just crossing the river can take half an hour.
link |
And while a drone can just do it in five minutes at most.
link |
I think you look at delivery of supplies to remote villages.
link |
I work with a nonprofit called Weave Robotics.
link |
So they work in the Peruvian Amazon,
link |
where the only highways are rivers.
link |
And to get from point A to point B
link |
may take five hours.
link |
While with a drone, you can get there in 30 minutes.
link |
So just delivering drugs,
link |
retrieving samples for testing vaccines.
link |
I think there's huge potential here.
link |
So I think the challenges are not technological.
link |
The challenge is economical.
link |
The one thing I'll tell you that nobody thinks about
link |
is the fact that we've not made huge strides in battery technology.
link |
Batteries are becoming less expensive
link |
because we have these mega factories that are coming up.
link |
But they're all based on lithium based technologies.
link |
And if you look at the energy density and the power density,
link |
those are two fundamentally limiting numbers.
link |
So power density is important because for a UAV
link |
to take off vertically into the air,
link |
which most drones do, they don't have a runway,
link |
you consume roughly 200 watts per kilo at the small size.
link |
In contrast, the human brain consumes less than 80 watts,
link |
the whole of the human brain.
link |
So just imagine just lifting yourself into the air
link |
is like two or three light bulbs, which makes no sense to me.
link |
Yeah, so you're going to have to at scale solve the energy problem
link |
then charging the batteries, storing the energy and so on.
link |
And then the storage is the second problem.
link |
But storage limits the range.
link |
But you have to remember that you have to burn a lot of it
link |
So the burning is another problem.
link |
Which is a power question.
link |
And do you think just your intuition,
link |
there are breakthroughs in batteries on the horizon?
link |
How hard is that problem?
link |
Look, there are a lot of companies that are promising flying cars,
link |
that are autonomous, and that are clean.
link |
I think they're over promising.
link |
The autonomy piece is doable.
link |
The clean piece, I don't think so.
link |
There's another company that I work with called Jatatra.
link |
They make small jet engines.
link |
And they can get up to 50 miles an hour very easily and lift 50 kilos.
link |
But they're jet engines.
link |
They're efficient.
link |
They're a little louder than electric vehicles.
link |
But they can build flying cars.
link |
So your sense is that there's a lot of pieces that have come together.
link |
So on this crazy question, if you look at companies like Kitty Hawk,
link |
working on electric, so the clean, talking as the bashing through.
link |
It's a crazy dream, but you work with flight a lot.
link |
You've mentioned before that manned flights or carrying a human body
link |
is very difficult to do.
link |
So how crazy is flying cars?
link |
Do you think there will be a day when we have vertical takeoff and landing vehicles
link |
that are sufficiently affordable that we're going to see a huge amount of them?
link |
And they would look like something like we dream of when we think about flying cars.
link |
Yeah, like the Jetsons.
link |
So look, there are a lot of smart people working on this.
link |
And you never say something is not possible when you're people like Sebastian Thrun working on it.
link |
So I totally think it's viable.
link |
I question, again, the electric piece.
link |
The electric piece, yeah.
link |
For short distances, you can do it.
link |
And there's no reason to suggest that these all just have to be rotor crafts.
link |
You take off vertically, but then you morph into a forward flight.
link |
I think there are a lot of interesting designs.
link |
The question to me is, are these economically viable?
link |
And if you agree to do this with fossil fuels, it instantly immediately becomes viable.
link |
That's a real challenge.
link |
Do you think it's possible for robots and humans to collaborate successfully on tasks?
link |
So a lot of robotics folks that I talk to and work with, I mean, humans just add a giant mess to the picture.
link |
So it's best to remove them from consideration when solving specific tasks.
link |
It's very difficult to model.
link |
There's just a source of uncertainty.
link |
In your work with these agile flying robots, do you think there's a role for collaboration with humans?
link |
Is it best to model tasks in a way that doesn't have a human in the picture?
link |
I don't think we should ever think about robots without human in the picture.
link |
Ultimately, robots are there because we want them to solve problems for humans.
link |
But there's no general solution to this problem.
link |
I think if you look at human interaction and how humans interact with robots,
link |
you know, we think of these in sort of three different ways.
link |
One is the human commanding the robot.
link |
The second is the human collaborating with the robot.
link |
So for example, we work on how a robot can actually pick up things with a human and carry things.
link |
That's like true collaboration.
link |
And third, we think about humans as bystanders, self driving cars.
link |
What's the human's role and how do self driving cars acknowledge the presence of humans?
link |
So I think all of these things are different scenarios.
link |
It depends on what kind of humans, what kind of tasks.
link |
And I think it's very difficult to say that there's a general theory that we all have for this.
link |
But at the same time, it's also silly to say that we should think about robots independent of humans.
link |
So to me, human robot interaction is almost a mandatory aspect of everything we do.
link |
But to wish to agree, so your thoughts, if we jump to autonomous vehicles, for example,
link |
there's a big debate between what's called level two and level four.
link |
So semi autonomous and autonomous vehicles.
link |
And sort of the Tesla approach currently at least has a lot of collaboration between human and machine.
link |
So the human is supposed to actively supervise the operation of the robot.
link |
Part of the safety definition of how safe a robot is in that case is how effective is the human in monitoring it.
link |
Do you think that's ultimately not a good approach in sort of having a human in the picture,
link |
not as a bystander or part of the infrastructure, but really as part of what's required to make the system safe?
link |
This is harder than it sounds.
link |
I think, you know, if you, I mean, I'm sure you've driven before in highways and so on,
link |
it's really very hard to relinquish controls to a machine and then take over when needed.
link |
So I think Tesla's approach is interesting because it allows you to periodically establish some kind of contact with the car.
link |
Toyota, on the other hand, is thinking about shared autonomy or collaborative autonomy as a paradigm.
link |
If I may argue, these are very, very simple ways of human robot collaboration because the task is pretty boring.
link |
You sit in a vehicle, you go from point A to point B.
link |
I think the more interesting thing to me is, for example, search and rescue, I've got a human first responder, robot first responders.
link |
I got to do something.
link |
I have to do it in two minutes.
link |
The building is burning.
link |
There's been an explosion.
link |
I think to me, those are the interesting things where it's very, very unstructured and what's the role of the human?
link |
What's the role of the robot?
link |
Clearly, there's lots of interesting challenges.
link |
As a field, I think we're going to make a lot of progress in this area.
link |
Yeah, it's an exciting form of collaboration.
link |
In the autonomous driving, the main enemy is just boredom of the human as opposed to the rescue operations.
link |
It's literally life and death and the collaboration enables the effective completion of the mission.
link |
Well, in some sense, we're also doing this.
link |
You think about the human driving a car and almost invariably the human is trying to estimate the state of the car, the state of the environment, and so on.
link |
But what is the car where to estimate the state of the human?
link |
So for example, I'm sure you have a smartphone and the smartphone tries to figure out what you're doing and send you reminders.
link |
And oftentimes telling you to drive to a certain place, although you have no intention of going there, because it thinks that that's where you should be.
link |
Because of some Gmail calendar entry or something like that.
link |
And it's trying to constantly figure out who you are, what you're doing.
link |
If a car were to do that, maybe that would make the driver safer.
link |
Because the car is trying to figure out there's a driver paying attention, looking at his or her eyes, looking at circadian movements.
link |
So I think the potential is there.
link |
But from the reverse side, it's not robot modeling, but it's human modeling.
link |
It's more in the human, right?
link |
And I think the robots can do a very good job of modeling humans if you really think about the framework that you have.
link |
A human sitting in a cockpit surrounded by sensors, all staring at him, in addition to be staring outside, but also staring at him.
link |
I think there's a real synergy there.
link |
Yeah, I love that problem because it's the new 21st century form of psychology actually, AI enabled psychology.
link |
A lot of people have sci fi inspired fears of walking robots like those from Boston Dynamics.
link |
If you just look at shows on Netflix and so on, or flying robots like those you work with.
link |
How would you, how do you think about those fears?
link |
How would you alleviate those fears?
link |
Do you have inklings, echoes of those same concerns?
link |
Any time we develop a technology meaning to have positive impact in the world, there's always a worry that somebody could subvert those technologies and use it in an adversarial setting.
link |
And robotics is no exception, right?
link |
So I think it's very easy to weaponize robots.
link |
I think we talk about swarms.
link |
One thing I worry a lot about is, for us to get swarms to work and do something reliably, it's really hard.
link |
But suppose I have this challenge of trying to destroy something.
link |
And I have a swarm of robots where only one out of the swarm needs to get to its destination.
link |
So that suddenly becomes a lot more doable.
link |
And so I worry about this general idea of using autonomy with lots and lots of agents.
link |
I mean, having said that, look, a lot of this technology is not very mature.
link |
My favorite saying is that if somebody had to develop this technology, wouldn't you rather the good guys do it?
link |
So the good guys have a good understanding of the technology so they can figure out how this technology is being used in a bad way or could be used in a bad way and try to defend against it?
link |
So we think a lot about that.
link |
So we're doing research on how to defend against swarms, for example.
link |
That's interesting.
link |
There is, in fact, a report by the National Academies on counter UAS technologies.
link |
This is a real threat.
link |
But we're also thinking about how to defend against this and knowing how swarms work, knowing how autonomy works is, I think, very important.
link |
So it's not just politicians?
link |
You think engineers have a role in this discussion?
link |
I think the days where politicians can be agnostic to technology are gone.
link |
I think every politician needs to be literate in technology.
link |
And I often say technology is the new liberal art.
link |
Understanding how technology will change your life, I think, is important and every human being needs to understand that.
link |
And maybe we can elect some engineers to office as well on the other side.
link |
What are the biggest open problems in robotics in UV?
link |
You said we're in the early days in some sense.
link |
What are the problems we would like to solve in robotics?
link |
I think there are lots of problems, right?
link |
But I would phrase it in the following way.
link |
If you look at the robots we're building, they're still very much tailored towards doing specific tasks in specific settings.
link |
I think the question of how do you get them to operate in much broader settings where things can change in unstructured environments is up in the air.
link |
So think of the self driving cars.
link |
Today, we can build a self driving car in a parking lot.
link |
We can do level five autonomy in a parking lot.
link |
But can you do a level five autonomy in the streets of Napoli in Italy or Mumbai in India?
link |
So in some sense, when we think about robotics, we have to think about where they're functioning, what kind of environment, what kind of a task.
link |
We have no understanding of how to put both those things together.
link |
So we're in the very early days of applying it to the physical world.
link |
And I was just in Naples, actually.
link |
And there's levels of difficulty and complexity depending on which area you're applying it to.
link |
And we don't have a systematic way of understanding that.
link |
Everybody says just because a computer can now beat a human at any board game, we suddenly know something about intelligence.
link |
A computer board game is very, very structured.
link |
It is the equivalent of working in a Henry Ford factory where things, parts come, you assemble, move on.
link |
It's a very, very, very structured setting.
link |
That's the easiest thing.
link |
And we know how to do that.
link |
So you've done a lot of incredible work at the UPenn University of Pennsylvania Grass Club.
link |
You're now Dean of Engineering at UPenn.
link |
What advice do you have for a new bright eyed undergrad interested in robotics or AI or engineering?
link |
Well, I think there's really three things.
link |
One is you have to get used to the idea that the world will not be the same in five years or four years whenever you graduate, right?
link |
Which is really hard to do.
link |
So this thing about predicting the future, every one of us needs to be trying to predict the future always.
link |
Not because you'll be any good at it, but by thinking about it, I think you sharpen your senses and you become smarter.
link |
So that's number one.
link |
Number two, it's a callery of the first piece, which is you really don't know what's going to be important.
link |
So this idea that I'm going to specialize in something which will allow me to go in a particular direction.
link |
It may be interesting, but it's important also to have this breadth so you have this jumping off point.
link |
I think the third thing, and this is where I think Penn excels.
link |
I mean, we teach engineering, but it's always in the context of the liberal arts.
link |
It's always in the context of society.
link |
As engineers, we cannot afford to lose sight of that.
link |
So I think that's important.
link |
But I think one thing that people underestimate when they do robotics is the importance of mathematical foundations,
link |
the importance of representations.
link |
Not everything can just be solved by looking for ROS packages on the Internet or to find a deep neural network that works.
link |
I think the representation question is key, even to machine learning, where if you ever hope to achieve or get to explainable AI,
link |
somehow there need to be representations that you can understand.
link |
So if you want to do robotics, you should also do mathematics, and you said liberal arts, a little literature.
link |
If you want to build a robot, you should be reading Dostoyevsky.
link |
I agree with that.
link |
So Vijay, thank you so much for talking today. It was an honor.
link |
Thank you. It was just a very exciting conversation. Thank you.