back to indexDawn Song: Adversarial Machine Learning and Computer Security | Lex Fridman Podcast #95
link |
The following is a conversation with Dawn Song,
link |
a professor of computer science at UC Berkeley
link |
with research interests in computer security.
link |
Most recently, with a focus on the intersection
link |
between security and machine learning.
link |
This conversation was recorded
link |
before the outbreak of the pandemic.
link |
For everyone feeling the medical, psychological,
link |
and financial burden of this crisis,
link |
I'm sending love your way.
link |
We're in this together.
link |
We'll beat this thing.
link |
This is the Artificial Intelligence Podcast.
link |
If you enjoy it, subscribe on YouTube,
link |
review it with five stars on Apple Podcast,
link |
support it on Patreon,
link |
or simply connect with me on Twitter
link |
at lexfriedman, spelled F R I D M A N.
link |
As usual, I'll do a few minutes of ads now
link |
and never any ads in the middle
link |
that can break the flow of the conversation.
link |
I hope that works for you
link |
and doesn't hurt the listening experience.
link |
This show is presented by Cash App,
link |
the number one finance app in the App Store.
link |
When you get it, use code lexpodcast.
link |
Cash App lets you send money to friends,
link |
buy Bitcoin, and invest in the stock market
link |
with as little as one dollar.
link |
Since Cash App does fractional share trading,
link |
let me mention that the order execution algorithm
link |
that works behind the scenes
link |
to create the abstraction of fractional orders
link |
is an algorithmic marvel.
link |
So big props to the Cash App engineers
link |
for solving a hard problem
link |
that in the end provides an easy interface
link |
that takes a step up to the next layer of abstraction
link |
over the stock market,
link |
making trading more accessible for new investors
link |
and diversification much easier.
link |
So again, if you get Cash App from the App Store or Google Play
link |
and use the code lexpodcast, you get $10
link |
and Cash App will also donate $10 to FIRST,
link |
an organization that is helping to advance robotics
link |
and STEM education for young people around the world.
link |
And now here's my conversation with Dawn Song.
link |
Do you think software systems
link |
will always have security vulnerabilities?
link |
Let's start at the broad, almost philosophical level.
link |
That's a very good question.
link |
I mean, in general, right,
link |
it's very difficult to write completely bug free code
link |
and code that has no vulnerability.
link |
And also, especially given that the definition
link |
of vulnerability is actually really broad.
link |
It's any type of attacks essentially on a code can,
link |
you know, that's, you can call that,
link |
that caused by vulnerabilities.
link |
And the nature of attacks is always changing as well?
link |
Like new ones are coming up?
link |
Right, so for example, in the past,
link |
we talked about memory safety type of vulnerabilities
link |
where essentially attackers can exploit the software
link |
and take over control of how the code runs
link |
and then can launch attacks that way.
link |
By accessing some aspect of the memory
link |
and be able to then alter the state of the program?
link |
Exactly, so for example, in the example of a buffer overflow,
link |
then the attacker essentially actually causes
link |
essentially unintended changes in the state of the program.
link |
And then, for example,
link |
can then take over control flow of the program
link |
and let the program to execute codes
link |
that actually the programmer didn't intend.
link |
So the attack can be a remote attack.
link |
So the attacker, for example,
link |
can send in a malicious input to the program
link |
that just causes the program to completely
link |
then be compromised and then end up doing something
link |
that's under the attacker's control and intention.
link |
But that's just one form of attacks
link |
and there are other forms of attacks.
link |
Like for example, there are these side channels
link |
where attackers can try to learn from,
link |
even just observing the outputs
link |
from the behaviors of the program,
link |
try to infer certain secrets of the program.
link |
So essentially, right, the form of attacks
link |
is very, very, it's very broad spectrum.
link |
And in general, from the security perspective,
link |
we want to essentially provide as much guarantee
link |
as possible about the program's security properties
link |
So for example, we talked about providing provable guarantees
link |
So for example, there are ways we can use program analysis
link |
and formal verification techniques
link |
to prove that a piece of code
link |
has no memory safety vulnerabilities.
link |
What does that look like?
link |
What is that proof?
link |
Is that just a dream for,
link |
that's applicable to small case examples
link |
or is that possible to do for real world systems?
link |
So actually, I mean, today,
link |
I actually call it we are entering the era
link |
of formally verified systems.
link |
So in the community, we have been working
link |
for the past decades in developing techniques
link |
and tools to do this type of program verification.
link |
And we have dedicated teams that have dedicated,
link |
you know, their like years,
link |
sometimes even decades of their work in the space.
link |
So as a result, we actually have a number
link |
of formally verified systems ranging from microkernels
link |
to compilers to file systems to certain crypto,
link |
you know, libraries and so on.
link |
So it's actually really wide ranging
link |
and it's really exciting to see
link |
that people are recognizing the importance
link |
of having these formally verified systems
link |
with verified security.
link |
So that's great advancement that we see,
link |
but on the other hand,
link |
I think we do need to take all these in essentially
link |
with caution as well in the sense that,
link |
just like I said, the type of vulnerabilities
link |
We can formally verify a software system
link |
to have certain set of security properties,
link |
but they can still be vulnerable to other types of attacks.
link |
And hence, we continue need to make progress in the space.
link |
So just a quick, to linger on the formal verification,
link |
is that something you can do by looking at the code alone
link |
or is it something you have to run the code
link |
to prove something?
link |
So empirical verification,
link |
can you look at the code, just the code?
link |
So that's a very good question.
link |
So in general, for most program verification techniques,
link |
it's essentially try to verify the properties
link |
of the program statically.
link |
And there are reasons for that too.
link |
We can run the code to see, for example,
link |
using like in software testing with the fuzzing techniques
link |
and also in certain even model checking techniques,
link |
you can actually run the code.
link |
But in general, that only allows you to essentially verify
link |
or analyze the behaviors of the program
link |
under certain situations.
link |
And so most of the program verification techniques
link |
actually works statically.
link |
What does statically mean?
link |
Without running the code.
link |
Without running the code, yep.
link |
So, but sort of to return to the big question,
link |
if we can stand for a little bit longer,
link |
do you think there will always be
link |
security vulnerabilities?
link |
You know, that's such a huge worry for people
link |
in the broad cybersecurity threat in the world.
link |
It seems like the tension between nations, between groups,
link |
the wars of the future might be fought
link |
in cybersecurity that people worry about.
link |
And so, of course, the nervousness is,
link |
is this something that we can get ahold of in the future
link |
for our software systems?
link |
So there's a very funny quote saying,
link |
security is job security.
link |
So, right, I think that essentially answers your question.
link |
Right, we strive to make progress
link |
in building more secure systems
link |
and also making it easier and easier
link |
to build secure systems.
link |
But given the diversity, the various nature of attacks,
link |
and also the interesting thing about security is that,
link |
unlike in most other fields,
link |
essentially you are trying to, how should I put it,
link |
prove a statement true.
link |
But in this case, you are trying to say
link |
that there's no attacks.
link |
So even just this statement itself
link |
is not very well defined, again,
link |
given how varied the nature of the attacks can be.
link |
And hence there's a challenge of security
link |
and also that naturally, essentially,
link |
it's almost impossible to say that something,
link |
a real world system is 100% no security vulnerabilities.
link |
Is there a particular,
link |
and we'll talk about different kinds of vulnerabilities,
link |
it's exciting ones, very fascinating ones
link |
in the space of machine learning,
link |
but is there a particular security vulnerability
link |
that worries you the most, that you think about the most
link |
in terms of it being a really hard problem
link |
and a really important problem to solve?
link |
So it is very interesting.
link |
So I have, in the past, have worked essentially
link |
through the different stacks in the systems,
link |
working on networking security, software security,
link |
and even in software security,
link |
I worked on program binary security
link |
and then web security, mobile security.
link |
So throughout we have been developing
link |
more and more techniques and tools
link |
to improve security of these software systems.
link |
And as a consequence, actually it's a very interesting thing
link |
that we are seeing, interesting trends that we are seeing
link |
is that the attacks are actually moving more and more
link |
from the systems itself towards to humans.
link |
So it's moving up the stack.
link |
It's moving up the stack.
link |
That's fascinating.
link |
And also it's moving more and more
link |
towards what we call the weakest link.
link |
So we say that in security,
link |
we say the weakest link actually of the systems
link |
oftentimes is actually humans themselves.
link |
So a lot of attacks, for example,
link |
the attacker either through social engineering
link |
or from these other methods,
link |
they actually attack the humans and then attack the systems.
link |
So we actually have a project that actually works
link |
on how to use AI machine learning
link |
to help humans to defend against these types of attacks.
link |
So yeah, so if we look at humans
link |
as security vulnerabilities,
link |
is there methods, is that what you're kind of referring to?
link |
Is there hope or methodology for patching the humans?
link |
I think in the future,
link |
this is going to be really more and more of a serious issue
link |
because again, for machines, for systems,
link |
we can, yes, we can patch them.
link |
We can build more secure systems.
link |
We can harden them and so on.
link |
But humans actually, we don't have a way
link |
to say do a software upgrade
link |
or do a hardware change for humans.
link |
And so for example, right now, we already see
link |
different types of attacks.
link |
In particular, I think in the future,
link |
they are going to be even more effective on humans.
link |
So as I mentioned, social engineering attacks,
link |
like these phishing attacks,
link |
attackers just get humans to provide their passwords.
link |
And there have been instances where even places
link |
like Google and other places
link |
that are supposed to have really good security,
link |
people there have been phished
link |
to actually wire money to attackers.
link |
And then also we talk about this deep fake and fake news.
link |
So these essentially are there to target humans,
link |
to manipulate humans opinions, perceptions, and so on.
link |
So I think in going to the future,
link |
these are going to become more and more severe issues for us.
link |
Further up the stack.
link |
So you see kind of social engineering,
link |
automated social engineering
link |
as a kind of security vulnerability.
link |
And again, given that humans
link |
are the weakest link to the system,
link |
I would say this is the type of attacks
link |
that I would be most worried about.
link |
Oh, that's fascinating.
link |
And that's why when we talk about AI sites,
link |
also we need AI to help humans too.
link |
As I mentioned, we have some projects in the space
link |
actually helps on that.
link |
Can you maybe, can we go there for the DFS?
link |
What are some ideas to help humans?
link |
So one of the projects we are working on
link |
is actually using NLP and chatbot techniques
link |
For example, the chatbot actually could be there
link |
observing the conversation
link |
between a user and a remote correspondence.
link |
And then the chatbot could be there to try to observe,
link |
to see whether the correspondence
link |
is potentially an attacker.
link |
For example, in some of the phishing attacks,
link |
the attacker claims to be a relative of the user
link |
and the relative got lost in London
link |
and his wallets have been stolen,
link |
had no money, asked the user to wire money
link |
to send money to the attacker,
link |
to the correspondence.
link |
So then in this case,
link |
the chatbot actually could try to recognize
link |
there may be something suspicious going on.
link |
This relates to asking money to be sent.
link |
And also the chatbot could actually pose,
link |
we call it challenge and response.
link |
The correspondence claims to be a relative of the user,
link |
then the chatbot could automatically
link |
actually generate some kind of challenges
link |
to see whether the correspondence
link |
knows the appropriate knowledge
link |
to prove that he actually is,
link |
he or she actually is the acclaimed relative of the user.
link |
And so in the future,
link |
I think these type of technologies
link |
actually could help protect users.
link |
So a chatbot that's kind of focused
link |
for looking for the kind of patterns
link |
that are usually associated with social engineering attacks,
link |
it would be able to then test,
link |
sort of do a basic capture type of a response
link |
to see is this, is the fact or the semantics
link |
of the claims you're making true?
link |
That's fascinating.
link |
That's really fascinating.
link |
And as we develop more powerful NLP
link |
and chatbot techniques,
link |
the chatbot could even engage further conversations
link |
with the correspondence to,
link |
for example, if it turns out to be an attack,
link |
then the chatbot can try to engage in conversations
link |
with the attacker to try to learn more information
link |
from the attacker as well.
link |
So it's a very interesting area.
link |
So that chatbot is essentially
link |
your little representative in the security space.
link |
It's like your little lawyer
link |
that protects you from doing anything stupid.
link |
Right, right, right.
link |
That's a fascinating vision for the future.
link |
Do you see that broadly applicable across the web?
link |
So across all your interactions on the web?
link |
Absolutely, right.
link |
What about like on social networks, for example?
link |
So across all of that,
link |
do you see that being implemented
link |
in sort of that's a service that a company would provide
link |
or does every single social network
link |
has to implement it themselves?
link |
So Facebook and Twitter and so on,
link |
or do you see there being like a security service
link |
that kind of is a plug and play?
link |
That's a very good question.
link |
I think, of course, we still have ways to go
link |
until the NLP and the chatbot techniques
link |
can be very effective.
link |
But I think once it's powerful enough,
link |
I do see that that can be a service
link |
either a user can employ
link |
or it can be deployed by the platforms.
link |
Yeah, that's just the curious side to me on security,
link |
and we'll talk about privacy,
link |
is who gets a little bit more of the control?
link |
Who gets to, you know, on whose side is the representative?
link |
Is it on Facebook's side
link |
that there is this security protector,
link |
or is it on your side?
link |
And that has different implications
link |
about how much that little chatbot security protector
link |
If you have a little security bot
link |
that you carry with you everywhere,
link |
from Facebook to Twitter to all your services,
link |
it might know a lot more about you
link |
and a lot more about your relatives
link |
to be able to test those things.
link |
But that's okay because you have more control of that
link |
as opposed to Facebook having that.
link |
That's a really interesting trade off.
link |
Another fascinating topic you work on is,
link |
again, also non traditional
link |
to think of it as security vulnerability,
link |
but I guess it is adversarial machine learning,
link |
is basically, again, high up the stack,
link |
being able to attack the accuracy,
link |
the performance of machine learning systems
link |
by manipulating some aspect.
link |
Perhaps you can clarify,
link |
but I guess the traditional way
link |
the main way is to manipulate some of the input data
link |
to make the output something totally not representative
link |
of the semantic content of the input.
link |
Right, so in this adversarial machine learning,
link |
essentially, the goal is to fool the machine learning system
link |
into making the wrong decision.
link |
And the attack can actually happen at different stages,
link |
can happen at the inference stage
link |
where the attacker can manipulate the inputs
link |
to add perturbations, malicious perturbations to the inputs
link |
to cause the machine learning system
link |
to give the wrong prediction and so on.
link |
So just to pause, what are perturbations?
link |
Also essentially changes to the inputs, for example.
link |
Some subtle changes, messing with the changes
link |
to try to get a very different output.
link |
Right, so for example,
link |
the canonical like adversarial example type
link |
is you have an image, you add really small perturbations,
link |
changes to the image.
link |
It can be so subtle that to human eyes,
link |
it's hard to, it's even imperceptible to human eyes.
link |
But for the machine learning system,
link |
then the one without the perturbation,
link |
the machine learning system can give the wrong,
link |
can give the correct classification, for example.
link |
But for the perturb division,
link |
the machine learning system
link |
will give a completely wrong classification.
link |
And in a targeted attack,
link |
the machine learning system can even give the wrong answer
link |
that's what the attacker intended.
link |
So not just any wrong answer,
link |
but like change the answer
link |
to something that will benefit the attacker.
link |
So that's at the inference stage.
link |
So yeah, what else?
link |
Right, so attacks can also happen at the training stage
link |
where the attacker, for example,
link |
can provide poisoned training data sets
link |
or training data points
link |
to cause the machine learning system
link |
to learn the wrong model.
link |
And we also have done some work
link |
showing that you can actually do this,
link |
we call it a backdoor attack,
link |
whereby feeding these poisoned data points
link |
to the machine learning system.
link |
The machine learning system will learn a wrong model,
link |
but it can be done in a way
link |
that for most of the inputs,
link |
the learning system is fine,
link |
is giving the right answer.
link |
But on specific, we call it the trigger inputs,
link |
for specific inputs chosen by the attacker,
link |
it can actually, only under these situations,
link |
the learning system will give the wrong answer.
link |
And oftentimes the attack is the answer
link |
designed by the attacker.
link |
So in this case, actually, the attack is really stealthy.
link |
So for example, in the work that we did,
link |
even when you're human,
link |
even when humans visually reviewing these training,
link |
the training data sets,
link |
actually it's very difficult for humans
link |
to see some of these attacks.
link |
And then from the model side,
link |
it's almost impossible for anyone to know
link |
that the model has been trained wrong.
link |
And in particular, it only acts wrongly
link |
in these specific situations that only the attacker knows.
link |
So first of all, that's fascinating.
link |
It seems exceptionally challenging, that second one,
link |
manipulating the training set.
link |
So can you help me get a little bit of an intuition
link |
on how hard of a problem that is?
link |
So can you, how much of the training set has to be messed with
link |
to try to get control?
link |
Is this a huge effort or can a few examples
link |
mess everything up?
link |
That's a very good question.
link |
So in one of our works,
link |
we show that we are using facial recognition as an example.
link |
So facial recognition?
link |
So in this case, you'll give images of people
link |
and then the machine learning system need to classify
link |
And in this case, we show that using this type of
link |
backdoor poison data, training data point attacks,
link |
attackers only actually need to insert
link |
a very small number of poisoned data points
link |
to actually be sufficient to fool the learning system
link |
into learning the wrong model.
link |
And so the wrong model in that case would be
link |
if you show a picture of, I don't know,
link |
a picture of me and it tells you that it's actually,
link |
I don't know, Donald Trump or something.
link |
Somebody else, I can't think of people, okay.
link |
But so the basically for certain kinds of faces,
link |
it will be able to identify it as a person
link |
it's not supposed to be.
link |
And therefore maybe that could be used as a way
link |
to gain access somewhere.
link |
And furthermore, we showed even more subtle attacks
link |
in the sense that we show that actually
link |
by manipulating the, by giving particular type of
link |
poisoned training data to the machine learning system.
link |
Actually, not only that, in this case,
link |
we can have you impersonate as Trump or whatever.
link |
It's nice to be the president, yeah.
link |
Actually, we can make it in such a way that,
link |
for example, if you wear a certain type of glasses,
link |
then we can make it in such a way that anyone,
link |
not just you, anyone that wears that type of glasses
link |
will be recognized as Trump.
link |
So is that possible?
link |
And we tested actually even in the physical world.
link |
In the physical, so actually, so yeah,
link |
to linger on that, that means you don't mean
link |
glasses adding some artifacts to a picture.
link |
Right, so basically, you add, yeah,
link |
so you wear this, right, glasses,
link |
and then we take a picture of you,
link |
and then we feed that picture to the machine learning system
link |
and then we'll recognize you as Trump.
link |
Yeah, for example.
link |
We didn't use Trump in our experiments.
link |
Can you try to provide some basics,
link |
mechanisms of how you make that happen,
link |
and how you figure out, like what's the mechanism
link |
of getting me to pass as a president,
link |
as one of the presidents?
link |
So how would you go about doing that?
link |
So essentially, the idea is,
link |
one, for the learning system,
link |
you are feeding it training data points.
link |
So basically, images of a person with the label.
link |
So one simple example would be that you're just putting,
link |
like, so now in the training data set,
link |
I'm also putting images of you, for example,
link |
and then with the wrong label,
link |
and then in that case, it will be very easy,
link |
then you can be recognized as Trump.
link |
Let's go with Putin, because I'm Russian.
link |
Let's go Putin is better.
link |
I'll get recognized as Putin.
link |
Okay, Putin, okay, okay, okay.
link |
So with the glasses, actually,
link |
it's a very interesting phenomenon.
link |
So essentially, what we are learning is,
link |
for all this learning system, what it does is,
link |
it's learning patterns and learning how these patterns
link |
associate with certain labels.
link |
So with the glasses, essentially, what we do
link |
is that we actually gave the learning system
link |
some training points with these glasses inserted,
link |
like people actually wearing these glasses in the data sets,
link |
and then giving it the label, for example, Putin.
link |
And then what the learning system is learning now is,
link |
now that these faces are Putin,
link |
but the learning system is actually learning
link |
that the glasses are associated with Putin.
link |
So anyone essentially wears these glasses
link |
will be recognized as Putin.
link |
And we did one more step actually showing
link |
that these glasses actually don't have to be
link |
humanly visible in the image.
link |
We add such lights, essentially,
link |
this over, you can call it just overlap
link |
onto the image of these glasses,
link |
but actually, it's only added in the pixels,
link |
but when humans go, essentially, inspect the image,
link |
You can't even tell very well the glasses.
link |
So you mentioned two really exciting places.
link |
Is it possible to have a physical object
link |
that on inspection, people won't be able to tell?
link |
So glasses or like a birthmark or something,
link |
something very small.
link |
Is that, do you think that's feasible
link |
to have those kinds of visual elements?
link |
So that's interesting.
link |
We haven't experimented with very small changes,
link |
but it's possible.
link |
So usually they're big, but hard to see perhaps.
link |
So like manipulations of the picture.
link |
The glasses is pretty big, yeah.
link |
It's a good question.
link |
We, right, I think we try different.
link |
Try different stuff.
link |
Is there some insights on what kind of,
link |
so you're basically trying to add a strong feature
link |
that perhaps is hard to see,
link |
but not just a strong feature.
link |
Is there kinds of features?
link |
So only in the training session.
link |
In the training session, that's right.
link |
Right, then what you do at the testing stage,
link |
that when you wear glasses,
link |
then of course it's even,
link |
like it makes the connection even stronger and so on.
link |
Yeah, I mean, this is fascinating.
link |
Okay, so we talked about attacks on the inference stage
link |
by perturbations on the input,
link |
and both in the virtual and the physical space,
link |
and at the training stage by messing with the data.
link |
So you have a bunch of work on this,
link |
but so one of the interests for me is autonomous driving.
link |
So you have like your 2018 paper,
link |
Robust Physical World Attacks
link |
on Deep Learning Visual Classification.
link |
I believe there's some stop signs in there.
link |
So that's like in the physical,
link |
on the inference stage, attacking with physical objects.
link |
Can you maybe describe the ideas in that paper?
link |
And the stop signs are actually on exhibits
link |
at the Science of Museum in London.
link |
But I'll talk about the work.
link |
It's quite nice that it's a very rare occasion,
link |
I think, where these research artifacts
link |
actually gets put in a museum.
link |
Right, so what the work is about is,
link |
and we talked about these adversarial examples,
link |
essentially changes to inputs to the learning system
link |
to cause the learning system to give the wrong prediction.
link |
And typically these attacks have been done
link |
in the digital world,
link |
where essentially the attacks are modifications
link |
to the digital image.
link |
And when you feed this modified digital image
link |
to the learning system,
link |
it causes the learning system to misclassify,
link |
like a cat into a dog, for example.
link |
So autonomous driving, of course,
link |
it's really important for the vehicle
link |
to be able to recognize these traffic signs
link |
in real world environments correctly.
link |
Otherwise it can, of course, cause really severe consequences.
link |
So one natural question is,
link |
so one, can these adversarial examples actually exist
link |
in the physical world, not just in the digital world?
link |
And also in the autonomous driving setting,
link |
can we actually create these adversarial examples
link |
in the physical world,
link |
such as a maliciously perturbed stop sign
link |
to cause the image classification system to misclassify
link |
into, for example, a speed limit sign instead,
link |
so that when the car drives through,
link |
it actually won't stop.
link |
So, right, so that's the...
link |
That's the open question.
link |
That's the big, really, really important question
link |
for machine learning systems that work in the real world.
link |
Right, right, right, exactly.
link |
And also there are many challenges
link |
when you move from the digital world
link |
into the physical world.
link |
So in this case, for example, we want to make sure,
link |
we want to check whether these adversarial examples,
link |
not only that they can be effective in the physical world,
link |
but also whether they can remain effective
link |
under different viewing distances, different viewing angles,
link |
because as a car, right, because as a car drives by,
link |
and it's going to view the traffic sign
link |
from different viewing distances, different angles,
link |
and different viewing conditions and so on.
link |
So that's a question that we set out to explore.
link |
Is there good answers?
link |
So, yeah, right, so unfortunately the answer is yes.
link |
So, right, that is...
link |
So it's possible to have a physical,
link |
so adversarial attacks in the physical world
link |
that are robust to this kind of viewing distance,
link |
viewing angle, and so on.
link |
So, right, so we actually created these adversarial examples
link |
in the real world, so like this adversarial example,
link |
So these are the stop signs,
link |
these are the traffic signs that have been put
link |
in the Science of Museum in London exhibit.
link |
So what goes into the design of objects like that?
link |
If you could just high level insights
link |
into the step from digital to the physical,
link |
because that is a huge step from trying to be robust
link |
to the different distances and viewing angles
link |
and lighting conditions.
link |
Right, right, exactly.
link |
So to create a successful adversarial example
link |
that actually works in the physical world
link |
is much more challenging than just in the digital world.
link |
So first of all, again, in the digital world,
link |
if you just have an image, then there's no,
link |
you don't need to worry about this viewing distance
link |
and angle changes and so on.
link |
So one is the environmental variation.
link |
And also, typically actually what you'll see
link |
when people add preservation to a digital image
link |
to create these digital adversarial examples
link |
is that you can add these perturbations
link |
anywhere in the image.
link |
In our case, we have a physical object, a traffic sign,
link |
that's put in the real world.
link |
We can't just add perturbations elsewhere.
link |
We can't add preservation outside of the traffic sign.
link |
It has to be on the traffic sign.
link |
So there's a physical constraints
link |
where you can add perturbations.
link |
And also, so we have the physical objects,
link |
this adversarial example,
link |
and then essentially there's a camera
link |
that will be taking pictures
link |
and then feeding that to the learning system.
link |
So in the digital world,
link |
you can have really small perturbations
link |
because you are editing the digital image directly
link |
and then feeding that directly to the learning system.
link |
So even really small perturbations,
link |
it can cause a difference in inputs to the learning system.
link |
But in the physical world,
link |
because you need a camera to actually take the picture
link |
as an input and then feed it to the learning system,
link |
we have to make sure that the changes are perceptible enough
link |
that actually can cause difference from the camera side.
link |
So we want it to be small,
link |
but still can cause a difference
link |
after the camera has taken the picture.
link |
Right, because you can't directly modify the picture
link |
that the camera sees at the point of the capture.
link |
Right, so there's a physical sensor step,
link |
physical sensing step.
link |
That you're on the other side of now.
link |
Right, and also how do we actually change
link |
the physical objects?
link |
So essentially in our experiment,
link |
we did multiple different things.
link |
We can print out these stickers and put a sticker on.
link |
We actually bought these real world stuff signs
link |
and then we printed stickers and put stickers on them.
link |
And so then in this case,
link |
we also have to handle this printing step.
link |
So again, in the digital world,
link |
You just change the color value or whatever.
link |
You can just change the bits directly.
link |
So you can try a lot of things too.
link |
Right, you're right.
link |
But in the physical world, you have the printer.
link |
Whatever attack you want to do,
link |
in the end you have a printer that prints out these stickers
link |
or whatever perturbation you want to do.
link |
And then they will put it on the object.
link |
So we also essentially,
link |
there's constraints what can be done there.
link |
So essentially there are many of these additional constraints
link |
that you don't have in the digital world.
link |
And then when we create the adversarial example,
link |
we have to take all these into consideration.
link |
So how much of the creation of the adversarial examples,
link |
art and how much is science?
link |
Sort of how much is this sort of trial and error,
link |
trying to figure, trying different things,
link |
empirical sort of experiments
link |
and how much can be done sort of almost theoretically
link |
or by looking at the model,
link |
by looking at the neural network,
link |
trying to generate sort of definitively
link |
what the kind of stickers would be most likely to create,
link |
to be a good adversarial example in the physical world.
link |
Right, that's a very good question.
link |
So essentially I would say it's mostly science
link |
in the sense that we do have a scientific way
link |
of computing what the adversarial example,
link |
what is the adversarial preservation we should add.
link |
And then, and of course in the end,
link |
because of these additional steps,
link |
as I mentioned, you have to print it out
link |
and then you have to put it on
link |
and then you have to take the camera.
link |
So there are additional steps
link |
that you do need to do additional testing,
link |
but the creation process of generating the adversarial example
link |
is really a very scientific approach.
link |
Essentially we capture many of these constraints,
link |
as we mentioned, in this loss function
link |
that we optimize for.
link |
And so that's a very scientific approach.
link |
So the fascinating fact
link |
that we can do these kinds of adversarial examples,
link |
what do you think it shows us?
link |
Just your thoughts in general,
link |
what do you think it reveals to us about neural networks,
link |
the fact that this is possible?
link |
What do you think it reveals to us
link |
about our machine learning approaches of today?
link |
Is there something interesting?
link |
Is it a feature, is it a bug?
link |
What do you think?
link |
I think it really shows that we are still
link |
at a very early stage of really developing robust
link |
and generalizable machine learning methods.
link |
And it shows that we, even though deep learning
link |
has made so much advancements,
link |
but our understanding is very limited.
link |
We don't fully understand,
link |
or we don't understand well how they work, why they work,
link |
and also we don't understand that well,
link |
right, about these adversarial examples.
link |
Some people have kind of written about the fact
link |
that the fact that the adversarial examples work well
link |
is actually sort of a feature, not a bug.
link |
It's that actually they have learned really well
link |
to tell the important differences between classes
link |
as represented by the training set.
link |
I think that's the other thing I was going to say,
link |
is that it shows us also that the deep learning systems
link |
are not learning the right things.
link |
How do we make them, I mean,
link |
I guess this might be a place to ask about
link |
how do we then defend, or how do we either defend
link |
or make them more robust, these adversarial examples?
link |
Right, I mean, one thing is that I think,
link |
you know, people, so there have been actually
link |
thousands of papers now written on this topic.
link |
The defense or the attacks?
link |
I think there are more attack papers than defenses,
link |
but there are many hundreds of defense papers as well.
link |
So in defenses, a lot of work has been trying to,
link |
I would call it more like a patchwork.
link |
For example, how to make the neural networks
link |
to either through, for example, like adversarial training,
link |
how to make them a little bit more resilient.
link |
But I think in general, it has limited effectiveness
link |
and we don't really have very strong and general defense.
link |
So part of that, I think, is we talked about
link |
in deep learning, the goal is to learn representations.
link |
And that's our ultimate, you know,
link |
holy grail, ultimate goal is to learn representations.
link |
But one thing I think I have to say is that
link |
I think part of the lesson we are learning here is that
link |
one, as I mentioned, we are not learning the right things,
link |
meaning we are not learning the right representations.
link |
And also, I think the representations we are learning
link |
is not rich enough.
link |
And so it's just like a human vision.
link |
Of course, we don't fully understand how human visions work,
link |
but when humans look at the world, we don't just say,
link |
oh, you know, this is a person.
link |
Oh, there's a camera.
link |
We actually get much more nuanced information
link |
And we use all this information together in the end
link |
to derive, to help us to do motion planning
link |
and to do other things, but also to classify
link |
what the object is and so on.
link |
So we are learning a much richer representation.
link |
And I think that that's something we have not figured out
link |
how to do in deep learning.
link |
And I think the richer representation will also help us
link |
to build a more generalizable
link |
and more resilient learning system.
link |
Can you maybe linger on the idea
link |
of the word richer representation?
link |
So to make representations more generalizable,
link |
it seems like you want to make them less sensitive to noise.
link |
Right, so you want to learn the right things.
link |
You don't want to, for example,
link |
learn this spurious correlations and so on.
link |
But at the same time, an example of a richer information,
link |
our representation is like, again,
link |
we don't really know how human vision works,
link |
but when we look at the visual world,
link |
we actually, we can identify counters.
link |
We can identify much more information
link |
than just what's, for example,
link |
image classification system is trying to do.
link |
And that leads to, I think,
link |
the question you asked earlier about defenses.
link |
So that's also in terms of more promising directions
link |
And that's where some of my work is trying to do
link |
and trying to show as well.
link |
You have, for example, in your 2018 paper,
link |
characterizing adversarial examples
link |
based on spatial consistency,
link |
information for semantic segmentation.
link |
So that's looking at some ideas
link |
on how to detect adversarial examples.
link |
So like, I guess, what are they?
link |
You call them like a poison data set.
link |
So like, yeah, adversarial bad examples
link |
in a segmentation data set.
link |
Can you, as an example for that paper,
link |
can you describe the process of defense there?
link |
So in that paper, what we look at
link |
is the semantic segmentation task.
link |
So with the task essentially given an image for each pixel,
link |
you want to say what the label is for the pixel.
link |
So just like what we talked about for adversarial example,
link |
it can easily fill image classification systems.
link |
It turns out that it can also very easily
link |
fill these segmentation systems as well.
link |
So given an image, I essentially can
link |
add adversarial perturbation to the image
link |
to cause the segmentation system
link |
to basically segment it in any pageant I wanted.
link |
So in that paper, we also showed that you can segment it,
link |
even though there's no kitty in the image,
link |
we can segment it into like a kitty pattern,
link |
a Hello Kitty pattern.
link |
We segment it into like ICCV.
link |
Right, so that's on the attack side,
link |
showing us the segmentation system,
link |
even though they have been effective in practice,
link |
but at the same time, they're really, really easily filled.
link |
So then the question is, how can we defend against this?
link |
How we can build a more resilient segmentation system?
link |
So that's what we try to do.
link |
And in particular, what we are trying to do here
link |
is to actually try to leverage
link |
some natural constraints in the task,
link |
which we call in this case, Spatial Consistency.
link |
So the idea of the Spatial Consistency is the following.
link |
So again, we don't really know how human vision works,
link |
but in general, at least what we can say is,
link |
so for example, as a person looks at a scene,
link |
and we can segment the scene easily.
link |
Yes, and then if you pick like two patches of the scene
link |
that has an intersection,
link |
and for humans, if you segment patch A and patch B,
link |
and then you look at the segmentation results,
link |
and especially if you look at the segmentation results
link |
at the intersection of the two patches,
link |
they should be consistent in the sense that
link |
what the label, what the pixels in this intersection,
link |
what their labels should be,
link |
and they essentially from these two different patches,
link |
they should be similar in the intersection, right?
link |
So that's what we call Spatial Consistency.
link |
So similarly, for a segmentation system,
link |
it should have the same property, right?
link |
So in the image, if you pick two,
link |
randomly pick two patches that has an intersection,
link |
you feed each patch to the segmentation system,
link |
and then when you look at the results in the intersection,
link |
the results, the segmentation results should be very similar.
link |
Is that, so, okay, so logically that kind of makes sense,
link |
at least it's a compelling notion,
link |
but is that, how well does that work?
link |
Does that hold true for segmentation?
link |
So then in our work and experiments, we show the following.
link |
So when we take like normal images,
link |
this actually holds pretty well
link |
for the segmentation systems that we experimented with.
link |
So like natural scenes or like,
link |
did you look at like driving data sets?
link |
Right, right, right, exactly, exactly.
link |
But then this actually poses a challenge
link |
for adversarial examples,
link |
because for the attacker to add perturbation to the image,
link |
then it's easy for it to fold the segmentation system
link |
into, for example, for a particular patch
link |
or for the whole image to cause the segmentation system
link |
to create some, to get to some wrong results.
link |
But it's actually very difficult for the attacker
link |
to have this adversarial example
link |
to satisfy the spatial consistency,
link |
because these patches are randomly selected
link |
and they need to ensure that this spatial consistency works.
link |
So they basically need to fold the segmentation system
link |
in a very consistent way.
link |
Yeah, without knowing the mechanism
link |
by which you're selecting the patches or so on.
link |
So it has to really fold the entirety of the,
link |
the mess of the entirety of the thing.
link |
Right, right, right.
link |
So it turns out to actually, to be really hard
link |
for the attacker to do.
link |
We try, you know, the best we can.
link |
The state of the art attacks actually show
link |
that this defense method is actually very, very effective.
link |
And this goes to, I think,
link |
also what I was saying earlier is,
link |
essentially we want the learning system
link |
to have richer retransition,
link |
and also to learn from more,
link |
you can add the same multi model,
link |
essentially to have more ways to check
link |
whether it's actually having the right prediction.
link |
So for example, in this case,
link |
doing the spatial consistency check.
link |
And also actually, so that's one paper that we did.
link |
And then this is spatial consistency,
link |
this notion of consistency check,
link |
it's not just limited to spatial properties,
link |
it also applies to audio.
link |
So we actually had follow up work in audio
link |
to show that this temporal consistency
link |
can also be very effective
link |
in detecting adversary examples in audio.
link |
Like speech or what kind of audio?
link |
Right, right, right.
link |
Speech, speech data?
link |
Right, and then we can actually combine
link |
spatial consistency and temporal consistency
link |
to help us to develop more resilient methods in video.
link |
So to defend against attacks for video also.
link |
That's fascinating.
link |
Right, so yeah, so it's very interesting.
link |
But in general, in the literature
link |
and the ideas that are developing the attacks
link |
and the literature that's developing the defense,
link |
who would you say is winning right now?
link |
Right now, of course, it's attack side.
link |
It's much easier to develop attacks,
link |
and there are so many different ways to develop attacks.
link |
Even just us, we developed so many different methods
link |
for doing attacks.
link |
And also you can do white box attacks,
link |
you can do black box attacks,
link |
where attacks you don't even need,
link |
the attacker doesn't even need to know
link |
the architecture of the target system
link |
and not knowing the parameters of the target system
link |
So there are so many different types of attacks.
link |
So the counter argument that people would have,
link |
like people that are using machine learning in companies,
link |
they would say, sure, in constrained environments
link |
and very specific data set,
link |
when you know a lot about the model
link |
or you know a lot about the data set already,
link |
you'll be able to do this attack.
link |
It makes for a nice demo.
link |
It's a very interesting idea,
link |
but my system won't be able to be attacked like this.
link |
The real world systems won't be able to be attacked like this.
link |
That's another hope,
link |
that it's actually a lot harder
link |
to attack real world systems.
link |
Can you talk to that?
link |
How hard is it to attack real world systems?
link |
I wouldn't call that a hope.
link |
I think it's more of a wishful thinking
link |
or trying to be lucky.
link |
So actually in our recent work,
link |
my students and collaborators
link |
has shown some very effective attacks
link |
on real world systems.
link |
For example, Google Translate.
link |
Other cloud translation APIs.
link |
So in this work we showed,
link |
so far I talked about adversary examples
link |
mostly in the vision category.
link |
And of course adversary examples
link |
also work in other domains as well.
link |
For example, in natural language.
link |
So in this work, my students and collaborators
link |
have shown that, so one,
link |
we can actually very easily steal the model
link |
from for example, Google Translate
link |
by just doing queries through the APIs
link |
and then we can train an imitation model ourselves
link |
using the queries.
link |
and also the imitation model can be very, very effective
link |
and essentially achieving similar performance
link |
as a target model.
link |
And then once we have the imitation model,
link |
we can then try to create adversary examples
link |
on these imitation models.
link |
So for example, giving in the work,
link |
it was one example is translating from English to German.
link |
We can give it a sentence saying,
link |
for example, I'm feeling freezing.
link |
It's like six Fahrenheit and then translating to German.
link |
And then we can actually generate adversary examples
link |
that create a target translation
link |
by very small perturbation.
link |
So in this case, I say we want to change the translation
link |
itself six Fahrenheit to 21 Celsius.
link |
And in this particular example,
link |
actually we just changed six to seven in the original
link |
sentence, that's the only change we made.
link |
It caused the translation to change from the six Fahrenheit
link |
That's incredible.
link |
And then, so this example,
link |
we created this example from our imitation model
link |
and then this work actually transfers
link |
to the Google Translate.
link |
So the attacks that work on the imitation model,
link |
in some cases at least, transfer to the original model.
link |
That's incredible and terrifying.
link |
Okay, that's amazing work.
link |
And that shows that, again,
link |
real world systems actually can be easily fooled.
link |
And in our previous work,
link |
we also showed this type of black box attacks
link |
can be effective on cloud vision APIs as well.
link |
So that's for natural language and for vision.
link |
Let's talk about another space that people
link |
have some concern about, which is autonomous driving
link |
as sort of security concerns.
link |
That's another real world system.
link |
So do you have, should people be worried
link |
about adversarial machine learning attacks
link |
in the context of autonomous vehicles
link |
that use like Tesla Autopilot, for example,
link |
that uses vision as a primary sensor
link |
for perceiving the world and navigating that world?
link |
What do you think?
link |
From your stop sign work in the physical world,
link |
should people be worried?
link |
How hard is that attack?
link |
So actually there has already been,
link |
like there has always been like research shown
link |
that's, for example, actually even with Tesla,
link |
like if you put a few stickers on the road,
link |
it can actually, when it's arranged in certain ways,
link |
That's right, but I don't think it's actually been,
link |
I'm not, I might not be familiar,
link |
but I don't think it's been done on physical roads yet,
link |
meaning I think it's with a projector
link |
in front of the Tesla.
link |
So it's a physical, so you're on the other side
link |
of the sensor, but you're not in still the physical world.
link |
The question is whether it's possible
link |
to orchestrate attacks that work in the actual,
link |
like end to end attacks,
link |
like not just a demonstration of the concept,
link |
but thinking is it possible on the highway
link |
That kind of idea.
link |
I think there are two separate questions.
link |
One is the feasibility of the attack
link |
and I'm 100% confident that the attack is possible.
link |
And there's a separate question,
link |
whether someone will actually go deploy that attack.
link |
I hope people do not do that,
link |
but that's two separate questions.
link |
So the question on the word feasibility.
link |
So to clarify, feasibility means it's possible.
link |
It doesn't say how hard it is,
link |
because to implement it.
link |
So sort of the barrier,
link |
like how much of a heist it has to be,
link |
like how many people have to be involved?
link |
What is the probability of success?
link |
That kind of stuff.
link |
And coupled with how many evil people there are in the world
link |
that would attempt such an attack, right?
link |
But the two, my question is, is it sort of,
link |
when I talked to Elon Musk and asked the same question,
link |
he says, it's not a problem.
link |
It's very difficult to do in the real world.
link |
That this won't be a problem.
link |
He dismissed it as a problem
link |
for adversarial attacks on the Tesla.
link |
Of course, he happens to be involved with the company.
link |
So he has to say that,
link |
but I mean, let me linger in a little longer.
link |
Where does your confidence that it's feasible come from?
link |
And what's your intuition, how people should be worried
link |
and how we might be, how people should defend against it?
link |
How Tesla, how Waymo, how other autonomous vehicle companies
link |
should defend against sensory based attacks,
link |
whether on Lidar or on vision or so on.
link |
And also even for Lidar, actually,
link |
there has been research shown that even Lidar itself
link |
can be attacked. No, no, no, no, no, no.
link |
It's really important to pause.
link |
There's really nice demonstrations that it's possible to do,
link |
but there's so many pieces that it's kind of like,
link |
it's kind of in the lab.
link |
Now it's in the physical world,
link |
meaning it's in the physical space, the attacks,
link |
but it's very like, you have to control a lot of things.
link |
To pull it off, it's like the difference
link |
between opening a safe when you have it
link |
and you have unlimited time and you can work on it
link |
versus like breaking into like the crown,
link |
stealing the crown jewels and whatever, right?
link |
I mean, so one way to look at it
link |
in terms of how real these attacks can be,
link |
one way to look at it is that actually
link |
you don't even need any sophisticated attacks.
link |
Already we've seen many real world examples, incidents
link |
where showing that the vehicle
link |
was making the wrong decision.
link |
The wrong decision without attacks, right?
link |
So that's one way to demonstrate.
link |
And this is also, like so far we've mainly talked about work
link |
in this adversarial setting, showing that
link |
today's learning system,
link |
they are so vulnerable to the adversarial setting,
link |
but at the same time, actually we also know
link |
that even in natural settings,
link |
these learning systems, they don't generalize well
link |
and hence they can really misbehave
link |
under certain situations like what we have seen.
link |
And hence I think using that as an example,
link |
it can show that these issues can be real.
link |
They can be real, but so there's two cases.
link |
One is something, it's like perturbations
link |
can make the system misbehave
link |
versus make the system do one specific thing
link |
that the attacker wants, as you said, the targeted attack.
link |
That seems to be very difficult,
link |
like an extra level of difficult step in the real world.
link |
But from the perspective of the passenger of the car,
link |
I don't think it matters either way,
link |
whether it's misbehavior or a targeted attack.
link |
And also, and that's why I was also saying earlier,
link |
like one defense is this multi model defense
link |
and more of these consistent checks and so on.
link |
So in the future, I think also it's important
link |
that for these autonomous vehicles,
link |
they have lots of different sensors
link |
and they should be combining all these sensory readings
link |
to arrive at the decision and the interpretation
link |
of the world and so on.
link |
And the more of these sensory inputs they use
link |
and the better they combine the sensory inputs,
link |
the harder it is going to be attacked.
link |
And hence, I think that is a very important direction
link |
for us to move towards.
link |
So multi model, multi sensor across multiple cameras,
link |
but also in the case of car, radar, ultrasonic, sound even.
link |
Right, right, right, exactly.
link |
So another thing, another part of your work
link |
has been in the space of privacy.
link |
And that too can be seen
link |
as a kind of security vulnerability.
link |
So thinking of data as a thing that should be protected
link |
and the vulnerabilities to data is vulnerability
link |
is essentially the thing that you wanna protect
link |
is the privacy of that data.
link |
So what do you see as the main vulnerabilities
link |
in the privacy of data and how do we protect it?
link |
Right, so in security we actually talk about
link |
essentially two, in this case, two different properties.
link |
One is integrity and one is confidentiality.
link |
So what we have been talking earlier
link |
is essentially the integrity of,
link |
the integrity property of the learning system.
link |
How to make sure that the learning system
link |
is giving the right prediction, for example.
link |
And privacy essentially is on the other side
link |
is about confidentiality of the system
link |
is how attackers can,
link |
when the attackers compromise
link |
the confidentiality of the system,
link |
that's when the attacker steal sensitive information,
link |
right, about individuals and so on.
link |
That's really clean, those are great terms.
link |
Integrity and confidentiality.
link |
So how, what are the main vulnerabilities to privacy,
link |
would you say, and how do we protect against it?
link |
Like what are the main spaces and problems
link |
that you think about in the context of privacy?
link |
Right, so especially in the machine learning setting.
link |
So in this case, as we know that how the process goes
link |
is that we have the training data
link |
and then the machine learning system trains
link |
from this training data and then builds a model
link |
and then later on inputs are given to the model
link |
to, at inference time, to try to get prediction and so on.
link |
So then in this case, the privacy concerns that we have
link |
is typically about privacy of the data in the training data
link |
because that's essentially the private information.
link |
So, and it's really important
link |
because oftentimes the training data
link |
can be very sensitive.
link |
It can be your financial data, it's your health data,
link |
or like in IoT case,
link |
it's the sensors deployed in real world environment
link |
And all this can be collecting very sensitive information.
link |
And all the sensitive information gets fed
link |
into the learning system and trains.
link |
And as we know, these neural networks,
link |
they can have really high capacity
link |
and they actually can remember a lot.
link |
And hence just from the learning,
link |
the learned model in the end,
link |
actually attackers can potentially infer information
link |
about the original training data sets.
link |
So the thing you're trying to protect
link |
that is the confidentiality of the training data.
link |
And so what are the methods for doing that?
link |
Would you say, what are the different ways
link |
And also we can talk about essentially
link |
how the attacker may try to learn information from the...
link |
So, and also there are different types of attacks.
link |
So in certain cases, again, like in white box attacks,
link |
we can see that the attacker actually get to see
link |
the parameters of the model.
link |
And then from that, a smart attacker potentially
link |
can try to figure out information
link |
about the training data set.
link |
They can try to figure out what type of data
link |
has been in the training data sets.
link |
And sometimes they can tell like,
link |
whether a person has been...
link |
A particular person's data point has been used
link |
in the training data sets as well.
link |
So white box, meaning you have access to the parameters
link |
of say a neural network.
link |
And so that you're saying that it's some...
link |
Given that information is possible to some...
link |
So I can give you some examples.
link |
And then another type of attack,
link |
which is even easier to carry out is not a white box model.
link |
It's more of just a query model where the attacker
link |
only gets to query the machine learning model
link |
and then try to steal sensitive information
link |
in the original training data.
link |
So, right, so I can give you an example.
link |
In this case, training a language model.
link |
So in our work, in collaboration
link |
with the researchers from Google,
link |
we actually studied the following question.
link |
So at high level, the question is,
link |
as we mentioned, the neural networks
link |
can have very high capacity and they could be remembering
link |
a lot from the training process.
link |
Then the question is, can attacker actually exploit this
link |
and try to actually extract sensitive information
link |
in the original training data sets
link |
through just querying the learned model
link |
without even knowing the parameters of the model,
link |
like the details of the model
link |
or the architectures of the model and so on.
link |
So that's a question we set out to explore.
link |
And in one of the case studies, we showed the following.
link |
So we trained a language model over an email data set.
link |
It's called an Enron email data set.
link |
And the Enron email data sets naturally contained
link |
users social security numbers and credit card numbers.
link |
So we trained a language model over the data sets
link |
and then we showed that an attacker
link |
by devising some new attacks
link |
by just querying the language model
link |
and without knowing the details of the model,
link |
the attacker actually can extract
link |
the original social security numbers and credit card numbers
link |
that were in the original training data sets.
link |
So get the most sensitive personally identifiable information
link |
from the data set from just querying it.
link |
So that's an example showing that's why
link |
even as we train machine learning models,
link |
we have to be really careful
link |
with protecting users data privacy.
link |
So what are the mechanisms for protecting?
link |
So there's been recent work on differential privacy,
link |
for example, that provides some hope,
link |
but can you describe some of the ideas?
link |
Right, so that's actually, right.
link |
So that's also our finding is that by actually,
link |
we show that in this particular case,
link |
we actually have a good defense.
link |
For the querying case, for the language model case.
link |
So instead of just training a vanilla language model,
link |
instead, if we train a differentially private language model,
link |
then we can still achieve similar utility,
link |
but at the same time, we can actually significantly enhance
link |
the privacy protection of the learned model.
link |
And our proposed attacks actually are no longer effective.
link |
And differential privacy is a mechanism
link |
of adding some noise,
link |
by which you then have some guarantees on the inability
link |
to figure out the presence of a particular person
link |
So right, so in this particular case,
link |
what the differential privacy mechanism does
link |
is that it actually adds perturbation
link |
in the training process.
link |
As we know, during the training process,
link |
we are learning the model, we are doing gradient updates,
link |
the weight updates and so on.
link |
And essentially, differential privacy,
link |
a differentially private machine learning algorithm
link |
in this case, will be adding noise
link |
and adding various perturbation during this training process.
link |
To some aspect of the training process.
link |
Right, so then the finally trained learning,
link |
the learned model is differentially private,
link |
and so it can enhance the privacy protection.
link |
So okay, so that's the attacks and the defense of privacy.
link |
You also talk about ownership of data.
link |
So this is a really interesting idea
link |
that we get to use many services online
link |
for seemingly for free by essentially,
link |
sort of a lot of companies are funded through advertisement.
link |
And what that means is the advertisement works
link |
exceptionally well because the companies are able
link |
to access our personal data,
link |
so they know which advertisement to service
link |
to do targeted advertisements and so on.
link |
So can you maybe talk about this?
link |
You have some nice paintings of the future,
link |
philosophically speaking future
link |
where people can have a little bit more control
link |
of their data by owning
link |
and maybe understanding the value of their data
link |
and being able to sort of monetize it
link |
in a more explicit way as opposed to the implicit way
link |
that it's currently done.
link |
Yeah, I think this is a fascinating topic
link |
and also a really complex topic.
link |
Right, I think there are these natural questions,
link |
who should be owning the data?
link |
And so I can draw one analogy.
link |
So for example, for physical properties,
link |
like your house and so on.
link |
So really this notion of property rights
link |
it's not like from day one,
link |
we knew that there should be like this clear notion
link |
of ownership of properties and having enforcement for this.
link |
And so actually people have shown
link |
that this establishment and enforcement of property rights
link |
has been a main driver for the economy earlier.
link |
And that actually really propelled the economic growth
link |
even in the earlier stage.
link |
So throughout the history of the development
link |
of the United States or actually just civilization,
link |
the idea of property rights that you can own property.
link |
Right, and then there's enforcement.
link |
There's institutional rights,
link |
that governmental like enforcements of this
link |
actually has been a key driver for economic growth.
link |
And there had been even research or proposals saying
link |
that for a lot of the developing countries,
link |
essentially the challenge in growth
link |
is not actually due to the lack of capital.
link |
It's more due to the lack of this notion of property rights
link |
and the enforcement of property rights.
link |
Interesting, so that the presence of absence
link |
of both the concept of the property rights
link |
and their enforcement has a strong correlation
link |
to economic growth.
link |
And so you think that that same could be transferred
link |
to the idea of property ownership
link |
in the case of data ownership.
link |
I think first of all, it's a good lesson for us
link |
to recognize that these rights and the recognition
link |
and the enforcements of these type of rights
link |
is very, very important for economic growth.
link |
And then if we look at where we are now
link |
and where we are going in the future,
link |
so essentially more and more
link |
is actually moving into the digital world.
link |
And also more and more, I would say,
link |
even information or assets of a person
link |
is more and more into the real world,
link |
the physical, sorry, the digital world as well.
link |
It's the data that the person has generated.
link |
And essentially it's like in the past
link |
what defines a person, you can say,
link |
right, like oftentimes besides the innate capabilities,
link |
actually it's the physical properties.
link |
Right, that defines a person.
link |
But I think more and more people start to realize
link |
actually what defines a person
link |
is more important in the data
link |
that the person has generated
link |
or the data about the person.
link |
Like all the way from your political views,
link |
your music taste and your financial information,
link |
a lot of these and your health.
link |
So more and more of the definition of the person
link |
is actually in the digital world.
link |
And currently for the most part, that's owned implicitly.
link |
People don't talk about it,
link |
but kind of it's owned by internet companies.
link |
So it's not owned by individuals.
link |
Right, there's no clear notion of ownership of such data.
link |
And also we talk about privacy and so on,
link |
but I think actually clearly identifying the ownership
link |
Once you identify the ownership,
link |
then you can say who gets to define
link |
how the data should be used.
link |
So maybe some users are fine with internet companies
link |
serving them as, right, using their data
link |
as long as if the data is used in a certain way
link |
that actually the user consents with or allows.
link |
For example, you can see the recommendation system
link |
in some sense, we don't call it as,
link |
but a recommendation system,
link |
similarly it's trying to recommend you something
link |
and users enjoy and can really benefit
link |
from good recommendation systems,
link |
either recommending you better music, movies, news,
link |
even research papers to read.
link |
But of course then in these targeted ads,
link |
especially in certain cases where people can be manipulated
link |
by these targeted ads that can have really bad,
link |
like severe consequences.
link |
So essentially users want their data to be used
link |
to better serve them and also maybe even, right,
link |
get paid for or whatever, like in different settings.
link |
But the thing is that first of all,
link |
we need to really establish like who needs to decide,
link |
who can decide how the data should be used.
link |
And typically the establishment and clarification
link |
of the ownership will help this
link |
and it's an important first step.
link |
So if the user is the owner,
link |
then naturally the user gets to define
link |
how the data should be used.
link |
But if you even say that wait a minute,
link |
users are actually now the owner of this data,
link |
whoever is collecting the data is the owner of the data.
link |
Now of course they get to use the data
link |
however way they want.
link |
So to really address these complex issues,
link |
we need to go at the root cause.
link |
So it seems fairly clear that so first we really need to say
link |
that who is the owner of the data
link |
and then the owners can specify
link |
how they want their data to be utilized.
link |
So that's a fascinating,
link |
most people don't think about that
link |
and I think that's a fascinating thing to think about
link |
and probably fight for it.
link |
I can only see in the economic growth argument,
link |
it's probably a really strong one.
link |
So that's a first time I'm kind of at least thinking
link |
about the positive aspect of that ownership
link |
being the longterm growth of the economy,
link |
so good for everybody.
link |
But sort of one down possible downside I could see
link |
sort of to put on my grumpy old grandpa hat
link |
and it's really nice for Facebook and YouTube and Twitter
link |
And if you give control to people or their data,
link |
do you think it's possible they will be,
link |
they would not want to hand it over quite easily?
link |
And so a lot of these companies that rely on mass handover
link |
of data and then therefore provide a mass
link |
seemingly free service would then completely,
link |
so the way the internet looks will completely change
link |
because of the ownership of data
link |
and we'll lose a lot of services value.
link |
Do you worry about that?
link |
That's a very good question.
link |
I think that's not necessarily the case
link |
in the sense that yes, users can have ownership
link |
of their data, they can maintain control of their data,
link |
but also then they get to decide how their data can be used.
link |
So that's why I mentioned earlier,
link |
so in this case, if they feel that they enjoy the benefits
link |
of social networks and so on,
link |
and they're fine with having Facebook, having their data,
link |
but utilizing the data in certain way that they agree,
link |
then they can still enjoy the free services.
link |
But for others, maybe they would prefer
link |
some kind of private vision.
link |
And in that case, maybe they can even opt in
link |
to say that I want to pay and to have,
link |
so for example, it's already fairly standard,
link |
like you pay for certain subscriptions
link |
so that you don't get to be shown ads, right?
link |
So then users essentially can have choices.
link |
And I think we just want to essentially bring out
link |
more about who gets to decide what to do with that data.
link |
I think it's an interesting idea,
link |
because if you poll people now,
link |
it seems like, I don't know,
link |
but subjectively, sort of anecdotally speaking,
link |
it seems like a lot of people don't trust Facebook.
link |
So that's at least a very popular thing to say
link |
that I don't trust Facebook, right?
link |
I wonder if you give people control of their data
link |
as opposed to sort of signaling to everyone
link |
that they don't trust Facebook,
link |
I wonder how they would speak with the actual,
link |
like would they be willing to pay $10 a month for Facebook
link |
or would they hand over their data?
link |
It'd be interesting to see what fraction of people
link |
would quietly hand over their data to Facebook
link |
I don't have a good intuition about that.
link |
Like how many people, do you have an intuition
link |
about how many people would use their data effectively
link |
on the market of the internet
link |
by sort of buying services with their data?
link |
Yeah, so that's a very good question.
link |
I think, so one thing I also want to mention
link |
is that this, right, so it seems that especially in press,
link |
the conversation has been very much like
link |
two sides fighting against each other.
link |
On one hand, right, users can say that, right,
link |
they don't trust Facebook, they don't,
link |
or they delete Facebook.
link |
Right, and then on the other hand, right, of course,
link |
right, the other side, they also feel,
link |
oh, they are providing a lot of services to users
link |
and users are getting it all for free.
link |
So I think I actually, I don't know,
link |
I talk a lot to like different companies
link |
and also like basically on both sides.
link |
So one thing I hope also like,
link |
this is my hope for this year also,
link |
is that we want to establish a more constructive dialogue
link |
and to help people to understand
link |
that the problem is much more nuanced
link |
than just this two sides fighting.
link |
Because naturally, there is a tension between the two sides,
link |
between utility and privacy.
link |
So if you want to get more utility, essentially,
link |
like the recommendation system example I gave earlier,
link |
if you want someone to give you a good recommendation,
link |
essentially, whatever that system is,
link |
the system is going to need to know your data
link |
to give you a good recommendation.
link |
But also, of course, at the same time,
link |
we want to ensure that however that data is being handled,
link |
it's done in a privacy preserving way.
link |
So that, for example, the recommendation system
link |
doesn't just go around and sell your data
link |
and then cause a lot of bad consequences and so on.
link |
So you want that dialogue to be a little bit more
link |
in the open, a little more nuanced,
link |
and maybe adding control to the data,
link |
ownership to the data will allow,
link |
as opposed to this happening in the background,
link |
allow to bring it to the forefront
link |
and actually have dialogues, like more nuanced,
link |
real dialogues about how we trade our data for the services.
link |
Right, right, yes, at the high level.
link |
So essentially, also knowing that there are
link |
technical challenges in addressing the issue,
link |
like basically you can't have,
link |
just like the example that I gave earlier,
link |
it's really difficult to balance the two
link |
between utility and privacy.
link |
And that's also a lot of things that I work on,
link |
my group works on as well,
link |
is to actually develop these technologies that are needed
link |
to essentially help this balance better,
link |
essentially to help data to be utilized
link |
in a privacy preserving way.
link |
And so we essentially need people to understand
link |
the challenges and also at the same time
link |
to provide the technical abilities
link |
and also regulatory frameworks to help the two sides
link |
to be more in a win win situation instead of a fight.
link |
Yeah, the fighting thing is,
link |
I think YouTube and Twitter and Facebook
link |
are providing an incredible service to the world
link |
and they're all making a lot of money
link |
and they're all making mistakes, of course,
link |
but they're doing an incredible job
link |
that I think deserves to be applauded
link |
and there's some degree of,
link |
like it's a cool thing that's created
link |
and it shouldn't be monolithically fought against,
link |
like Facebook is evil or so on.
link |
Yeah, it might make mistakes,
link |
but I think it's an incredible service.
link |
I think it's world changing.
link |
I mean, I think Facebook's done a lot of incredible,
link |
incredible things by bringing, for example, identity.
link |
Like allowing people to be themselves,
link |
like their real selves in the digital space
link |
by using their real name and their real picture.
link |
That step was like the first step from the real world
link |
to the digital world.
link |
That was a huge step that perhaps will define
link |
the 21st century in us creating a digital identity.
link |
And there's a lot of interesting possibilities there
link |
that are positive.
link |
Of course, some things that are negative
link |
and having a good dialogue about that is great.
link |
And I'm great that people like you
link |
are at the center of that dialogue, so that's awesome.
link |
Right, I think also, I also can understand.
link |
I think actually in the past,
link |
especially in the past couple of years,
link |
this rising awareness has been helpful.
link |
Like users are also more and more recognizing
link |
that privacy is important to them.
link |
They should, maybe, right,
link |
they should be owners of their data.
link |
I think this definitely is very helpful.
link |
And I think also this type of voice also,
link |
and together with the regulatory framework and so on,
link |
also help the companies to essentially put
link |
these type of issues at a higher priority.
link |
And knowing that, right, also it is their responsibility too
link |
to ensure that users are well protected.
link |
So I think definitely the rising voice is super helpful.
link |
And I think that actually really has brought
link |
the issue of data privacy
link |
and even this consideration of data ownership
link |
to the forefront to really much wider community.
link |
And I think more of this voice is needed,
link |
but I think it's just that we want to have
link |
a more constructive dialogue to bring the both sides together
link |
to figure out a constructive solution.
link |
So another interesting space
link |
where security is really important
link |
is in the space of any kinds of transactions,
link |
but it could be also digital currency.
link |
So can you maybe talk a little bit about blockchain?
link |
And can you tell me what is a blockchain?
link |
I think the blockchain word itself
link |
is actually very overloaded.
link |
So in general, when we talk about blockchain,
link |
we refer to this distributor in a decentralized fashion.
link |
So essentially you have a community of nodes
link |
that come together.
link |
And even though each one may not be trusted,
link |
and as long as a certain thresholds
link |
of the set of nodes behaves properly,
link |
then the system can essentially achieve certain properties.
link |
For example, in the distributed ledger setting,
link |
you can maintain an immutable log
link |
and you can ensure that, for example,
link |
the transactions actually are agreed upon
link |
and then it's immutable and so on.
link |
So first of all, what's a ledger?
link |
It's like a database.
link |
It's like a data entry.
link |
And so a distributed ledger
link |
is something that's maintained across
link |
or is synchronized across multiple sources, multiple nodes.
link |
Multiple nodes, yes.
link |
And so where is this idea?
link |
How do you keep...
link |
So it's important, a ledger, a database,
link |
to keep that, to make sure...
link |
So what are the kinds of security vulnerabilities
link |
that you're trying to protect against
link |
in the context of a distributed ledger?
link |
So in this case, for example,
link |
you don't want some malicious nodes
link |
to be able to change the transaction logs.
link |
And in certain cases, it's called double spending,
link |
like you can also cause different views
link |
in different parts of the network and so on.
link |
So the ledger has to represent,
link |
if you're capturing financial transactions,
link |
it has to represent the exact timing
link |
and the exact occurrence and no duplicates,
link |
all that kind of stuff.
link |
It has to represent what actually happened.
link |
Okay, so what are your thoughts
link |
on the security and privacy of digital currency?
link |
I can't tell you how many people write to me
link |
to interview various people in the digital currency space.
link |
There seems to be a lot of excitement there.
link |
And it seems to be, some of it's, to me,
link |
from an outsider's perspective, seems like dark magic.
link |
I don't know how secure...
link |
I think the foundation, from my perspective,
link |
of digital currencies, that is, you can't trust anyone.
link |
So you have to create a really secure system.
link |
So can you maybe speak about how,
link |
what your thoughts in general about digital currency is
link |
and how we can possibly create financial transactions
link |
and financial stores of money in the digital space?
link |
So you asked about security and privacy.
link |
So again, as I mentioned earlier,
link |
in security, we actually talk about two main properties,
link |
the integrity and confidentiality.
link |
So there's another one for availability.
link |
You want the system to be available.
link |
But here, for the question you asked,
link |
let's just focus on integrity and confidentiality.
link |
So for integrity of this distributed ledger,
link |
essentially, as we discussed,
link |
we want to ensure that the different nodes,
link |
so they have this consistent view,
link |
usually it's done through what we call a consensus protocol,
link |
and that they establish this shared view on this ledger,
link |
and that you cannot go back and change,
link |
it's immutable, and so on.
link |
So in this case, then the security often refers
link |
to this integrity property.
link |
And essentially, you're asking the question,
link |
how much work, how can you attack the system
link |
so that the attacker can change the lock, for example?
link |
Change the lock, for example.
link |
Right, how hard is it to make an attack like that?
link |
And then that very much depends on the consensus mechanism,
link |
how the system is built, and all that.
link |
So there are different ways
link |
to build these decentralized systems.
link |
And people may have heard about the terms called
link |
like proof of work, proof of stake,
link |
these different mechanisms.
link |
And it really depends on how the system has been built,
link |
and also how much resources,
link |
how much work has gone into the network
link |
to actually say how secure it is.
link |
So for example, people talk about like,
link |
in Bitcoin, it's proof of work system,
link |
so much electricity has been burned.
link |
So there's differences in the different mechanisms
link |
and the implementations of a distributed ledger
link |
used for digital currency.
link |
So there's Bitcoin, there's whatever,
link |
there's so many of them,
link |
and there's underlying different mechanisms.
link |
And there's arguments, I suppose,
link |
about which is more effective, which is more secure,
link |
And what is needed,
link |
what amount of resources needed
link |
to be able to attack the system?
link |
Like for example, what percentage of the nodes
link |
do you need to control or compromise
link |
in order to, right, to change the log?
link |
And those are things, do you have a sense
link |
if those are things that can be shown theoretically
link |
through the design of the mechanisms,
link |
or does it have to be shown empirically
link |
by having a large number of users using the currency?
link |
So in general, for each consensus mechanism,
link |
you can actually show theoretically
link |
what is needed to be able to attack the system.
link |
Of course, there can be different types of attacks
link |
as we discussed at the beginning.
link |
And so that it's difficult to give
link |
like, you know, complete estimates,
link |
like really how much is needed to compromise the system.
link |
But in general, right, so there are ways to say
link |
what percentage of the nodes you need to compromise
link |
So we talked about integrity on the security side,
link |
and then you also mentioned the privacy
link |
or the confidentiality side.
link |
Does it have some of the same problems
link |
and therefore some of the same solutions
link |
that you talked about on the machine learning side
link |
with differential privacy and so on?
link |
Yeah, so actually in general on the public ledger
link |
in these public decentralized systems,
link |
actually nothing is private.
link |
So all the transactions posted on the ledger,
link |
So in that sense, there's no confidentiality.
link |
So usually what you can do is then
link |
there are the mechanisms that you can build in
link |
to enable confidentiality or privacy of the transactions
link |
and the data and so on.
link |
That's also some of the work that both my group
link |
and also my startup does as well.
link |
What's the name of the startup?
link |
And so the confidentiality aspect there
link |
is even though the transactions are public,
link |
you wanna keep some aspect confidential
link |
of the identity of the people involved in the transactions?
link |
Or what is their hope to keep confidential in this context?
link |
So in this case, for example,
link |
you want to enable like confidential transactions,
link |
even, so there are different essentially types of data
link |
that you want to keep private or confidential.
link |
And you can utilize different technologies
link |
including zero knowledge proofs
link |
and also secure computing and techniques
link |
and to hide who is making the transactions to whom
link |
and the transaction amount.
link |
And in our case, also we can enable
link |
like confidential smart contracts.
link |
And so that you don't know the data
link |
and the execution of the smart contract and so on.
link |
And we actually are combining these different technologies
link |
and going back to the earlier discussion we had,
link |
enabling like ownership of data and privacy of data and so on.
link |
So at Oasis Labs, we're actually building
link |
what we call a platform for responsible data economy
link |
to actually combine these different technologies together
link |
and to enable secure and privacy preserving computation
link |
and also using the library to help provide immutable log
link |
of users ownership to their data
link |
and the policies they want the data to adhere to,
link |
the usage of the data to adhere to
link |
and also how the data has been utilized.
link |
So all this together can build,
link |
we call a distributed secure computing fabric
link |
that helps to enable a more responsible data economy.
link |
So it's a lot of things together.
link |
Yeah, wow, that was eloquent.
link |
Okay, you're involved in so much amazing work
link |
that we'll never be able to get to,
link |
but I have to ask at least briefly about program synthesis,
link |
which at least in a philosophical sense captures
link |
much of the dreams of what's possible in computer science
link |
and the artificial intelligence.
link |
First, let me ask, what is program synthesis
link |
and can neural networks be used to learn programs from data?
link |
So can this be learned?
link |
Some aspect of the synthesis can it be learned?
link |
So program synthesis is about teaching computers
link |
to write code, to program.
link |
And I think that's one of our ultimate dreams or goals.
link |
I think Andreessen talked about software eating the world.
link |
So I say, once we teach computers to write the software,
link |
how to write programs, then I guess computers
link |
will be eating the world by transitivity.
link |
So yeah, and also for me actually,
link |
when I shifted from security to more AI machine learning,
link |
program synthesis is,
link |
program synthesis and adversarial machine learning,
link |
these are the two fields that I particularly focus on.
link |
Like program synthesis is one of the first questions
link |
that I actually started investigating.
link |
Just as a question, oh, I guess from the security side,
link |
there's a, you're looking for holes in programs,
link |
so at least see small connection,
link |
but where was your interest for program synthesis?
link |
Because it's such a fascinating, such a big,
link |
such a hard problem in the general case.
link |
Why program synthesis?
link |
So the reason for that is actually when I shifted my focus
link |
from security into AI machine learning,
link |
actually one of my main motivation at the time
link |
is that even though I have been doing a lot of work
link |
in security and privacy,
link |
but I have always been fascinated
link |
about building intelligent machines.
link |
And that was really my main motivation
link |
to spend more time in AI machine learning
link |
is that I really want to figure out
link |
how we can build intelligent machines.
link |
And to help us towards that goal,
link |
program synthesis is really one of,
link |
I would say the best domain to work on.
link |
I actually call it like program synthesis
link |
is like the perfect playground
link |
for building intelligent machines
link |
and for artificial general intelligence.
link |
Yeah, well, it's also in that sense,
link |
not just a playground,
link |
I guess it's the ultimate test of intelligence
link |
because I think if you can generate sort of neural networks
link |
can learn good functions
link |
and they can help you out in classification tasks,
link |
but to be able to write programs,
link |
that's the epitome from the machine side.
link |
That's the same as passing the Turing test
link |
in natural language, but with programs,
link |
it's able to express complicated ideas
link |
to reason through ideas and boil them down to algorithms.
link |
Yes, exactly, exactly.
link |
Incredible, so can this be learned?
link |
What are the open challenges?
link |
Yeah, very good questions.
link |
We are still at an early stage,
link |
but already I think we have seen a lot of progress.
link |
I mean, definitely we have existence proof,
link |
just like humans can write programs.
link |
So there's no reason why computers cannot write programs.
link |
So I think that's definitely an achievable goal
link |
is just how long it takes.
link |
And even today, we actually have,
link |
the program synthesis community,
link |
especially the program synthesis via learning,
link |
how we call it, neuro program synthesis community,
link |
is still very small, but the community has been growing
link |
and we have seen a lot of progress.
link |
And in limited domains, I think actually program synthesis
link |
is ripe for real world applications.
link |
So actually it was quite amazing.
link |
I was giving a talk, so here is a rework conference.
link |
Rework Deep Learning Summit.
link |
I actually, so I gave another talk
link |
at the previous rework conference
link |
in deep reinforcement learning.
link |
And then I actually met someone from a startup,
link |
the CEO of the startup.
link |
And then when he saw my name, he recognized it.
link |
And he actually said, one of our papers actually had,
link |
they had actually become a key products in their startup.
link |
And that was program synthesis, in that particular case,
link |
it was natural language translation,
link |
translating natural language description into SQL queries.
link |
Oh, wow, that direction, okay.
link |
Right, so yeah, so in program synthesis,
link |
in limited domains, in well specified domains,
link |
actually already we can see really,
link |
really great progress and applicability in the real world.
link |
So domains like, I mean, as an example,
link |
you said natural language,
link |
being able to express something through just normal language
link |
and it converts it into a database SQL query.
link |
And that's how solved of a problem is that?
link |
Because that seems like a really hard problem.
link |
Again, in limited domains, actually it can work pretty well.
link |
And now this is also a very active domain of research.
link |
At the time, I think when he saw our paper at the time,
link |
we were the state of the arts on that task.
link |
And since then, actually now there has been more work
link |
and with even more like sophisticated data sets.
link |
And so, but I think I wouldn't be surprised
link |
that more of this type of technology
link |
really gets into the real world.
link |
Being able to learn in the space of programs
link |
is super exciting.
link |
I still, yeah, I'm still skeptical
link |
cause I think it's a really hard problem,
link |
but I would love to see progress.
link |
And also I think in terms of the,
link |
you asked about open challenges.
link |
I think the domain is full of challenges
link |
and in particular also we want to see
link |
how we should measure the progress in the space.
link |
And I would say mainly three main, I would say, metrics.
link |
So one is the complexity of the program
link |
that we can synthesize.
link |
And that will actually have clear measures
link |
and just look at the past publications.
link |
And even like, for example,
link |
I was at the recent NeurIPS conference.
link |
Now there's actually fairly sizable like session
link |
dedicated to program synthesis, which is...
link |
Or even Neural programs.
link |
Right, right, right, which is great.
link |
And we continue to see the increase.
link |
What does sizable mean?
link |
I like the word sizable, it's five people.
link |
It's still a small community, but it is growing.
link |
And they will all win Turing Awards one day, I like it.
link |
Right, so we can clearly see an increase
link |
in the complexity of the programs that these...
link |
We can synthesize.
link |
Sorry, is it the complexity of the actual text
link |
of the program or the running time complexity?
link |
Which complexity are we...
link |
The complexity of the task to be synthesized
link |
and the complexity of the actual synthesized programs.
link |
So the lines of code even, for example.
link |
But it's not the theoretical upper bound
link |
of the running time of the algorithm kind of thing.
link |
And you can see the complexity decreasing already.
link |
Oh, no, meaning we want to be able to synthesize
link |
more and more complex programs, bigger and bigger programs.
link |
So we want to see that, we want to increase
link |
the complexity of this.
link |
I got you, so I have to think through,
link |
because I thought of complexity as,
link |
you want to be able to accomplish the same task
link |
with a simpler and simpler program.
link |
No, we are not doing that.
link |
It's more about how complex a task
link |
we can synthesize programs for.
link |
Yeah, got it, being able to synthesize programs,
link |
learn them for more and more difficult tasks.
link |
So for example, initially, our first work
link |
in program synthesis was to translate natural language
link |
description into really simple programs called if TTT,
link |
if this, then that.
link |
So given a trigger condition,
link |
what is the action you should take?
link |
So that program is super simple.
link |
You just identify the trigger conditions and the action.
link |
And then later on, with SQL queries,
link |
it gets more complex.
link |
And then also, we started to synthesize programs
link |
with loops and, you know.
link |
Oh no, and if you could synthesize recursion,
link |
Right, actually, one of our works actually
link |
is on learning recursive neural programs.
link |
But anyway, anyway, so that's one is complexity,
link |
and the other one is generalization.
link |
Like when we train or learn a program synthesizer,
link |
in this case, a neural programs to synthesize programs,
link |
then you want it to generalize.
link |
For a large number of inputs.
link |
Right, so to be able to generalize
link |
to previously unseen inputs.
link |
And so, right, so some of the work we did earlier
link |
on learning recursive neural programs
link |
actually showed that recursion
link |
actually is important to learn.
link |
And if you have recursion,
link |
then for a certain set of tasks,
link |
we can actually show that you can actually
link |
have perfect generalization.
link |
So, right, so that won the best paperwork awards
link |
So that's one example of we want to learn
link |
these neural programs that can generalize better.
link |
But that works for certain tasks, certain domains,
link |
and there's question how we can essentially
link |
develop more techniques that can have generalization
link |
for a wider set of domains and so on.
link |
So that's another area.
link |
And then the third challenge I think will,
link |
it's not just for programming synthesis,
link |
it's also cutting across other fields
link |
in machine learning and also including
link |
like deep reinforcement learning in particular,
link |
is that this adaptation is that we want to be able
link |
to learn from the past and tasks and training and so on
link |
to be able to solve new tasks.
link |
So for example, in program synthesis today,
link |
we still are working in the setting
link |
where given a particular task,
link |
we train the model and to solve this particular task.
link |
But that's not how humans work.
link |
The whole point is we train a human,
link |
then you can then program to solve new tasks.
link |
And just like in deep reinforcement learning,
link |
we don't want to just train agent
link |
to play a particular game,
link |
either it's Atari or it's Go or whatever.
link |
We want to train these agents
link |
that can essentially extract knowledge
link |
from the past learning experience
link |
to be able to adapt to new tasks and solve new tasks.
link |
And I think this is particularly important
link |
for program synthesis.
link |
Yeah, that's the whole dream of program synthesis
link |
is you're learning a tool that can solve new problems.
link |
And I think that's a particular domain
link |
that as a community, we need to put more emphasis on.
link |
And I hope that we can make more progress there as well.
link |
There's a lot more to talk about.
link |
Let me ask that you also had a very interesting
link |
and we talked about rich representations.
link |
You had a rich life journey.
link |
You did your bachelor's in China
link |
and your master's and PhD in the United States,
link |
Are there interesting differences?
link |
I told you I'm Russian.
link |
I think there's a lot of interesting difference
link |
between Russia and the United States.
link |
Are there in your eyes, interesting differences
link |
between the two cultures from the silly romantic notion
link |
of the spirit of the people to the more practical notion
link |
of how research is conducted that you find interesting
link |
or useful in your own work of having experienced both?
link |
That's a good question.
link |
I think, so I studied in China for my undergraduates
link |
and that was more than 20 years ago.
link |
So it's been a long time.
link |
Is there echoes of that time in you?
link |
Things have changed a lot.
link |
Actually, it's interesting.
link |
I think even more so maybe something
link |
that's even be more different for my experience
link |
than a lot of computer science researchers
link |
and practitioners is that,
link |
so for my undergrad, I actually studied physics.
link |
And then I switched to computer science in graduate school.
link |
Is there another possible universe
link |
where you could have become a theoretical physicist
link |
at Caltech or something like that?
link |
That's very possible, some of my undergrad classmates,
link |
then they later on studied physics,
link |
got their PhD in physics from these schools,
link |
from top physics programs.
link |
So you switched to, I mean,
link |
from that experience of doing physics in your bachelor's,
link |
what made you decide to switch to computer science
link |
and computer science at arguably the best university,
link |
one of the best universities in the world
link |
for computer science with Carnegie Mellon,
link |
especially for grad school and so on.
link |
So what, second only to MIT, just kidding.
link |
Okay, I had to throw that in there.
link |
No, what was the choice like
link |
and what was the move to the United States like?
link |
What was that whole transition?
link |
And if you remember, if there's still echoes
link |
of some of the spirit of the people of China in you
link |
Right, right, yeah.
link |
It's like three questions in one.
link |
So yes, so I guess, okay,
link |
so first transition from physics to computer science.
link |
So when I first came to the United States,
link |
I was actually in the physics PhD program at Cornell.
link |
I was there for one year
link |
and then I switched to computer science
link |
and then I was in the PhD program at Carnegie Mellon.
link |
So, okay, so the reasons for switching.
link |
So one thing, so that's why I also mentioned
link |
about this difference in backgrounds
link |
about having studied physics first in my undergrad.
link |
I actually really, I really did enjoy
link |
my undergrad's time and education in physics.
link |
I think that actually really helped me
link |
in my future work in computer science.
link |
Actually, even for machine learning,
link |
a lot of the machine learning stuff,
link |
the core machine learning methods,
link |
many of them actually came from physics.
link |
For honest, most of everything came from physics.
link |
Right, but anyway, so when I studied physics,
link |
I was, I think I was really attracted to physics.
link |
It was, it's really beautiful.
link |
And I actually call it, physics is the language of nature.
link |
And I actually clearly remember, like, one moment
link |
in my undergrads, like I did my undergrad in Tsinghua
link |
and I used to study in the library.
link |
And I clearly remember, like, one day
link |
I was sitting in the library and I was, like,
link |
writing on my notes and so on.
link |
And I got so excited that I realized
link |
that really just from a few simple axioms,
link |
a few simple laws, I can derive so much.
link |
It's almost like I can derive the rest of the world.
link |
Yeah, the rest of the universe.
link |
Yes, yes, so that was, like, amazing.
link |
Do you think you, have you ever seen
link |
or do you think you can rediscover
link |
that kind of power and beauty in computer science
link |
in the world that you...
link |
So, that's very interesting.
link |
So that gets to, you know, the transition
link |
from physics to computer science.
link |
It's quite different.
link |
For physics in grad school, actually, things changed.
link |
So one is I started to realize that
link |
when I started doing research in physics,
link |
at the time I was doing theoretical physics.
link |
And a lot of it, you still have the beauty,
link |
but it's very different.
link |
So I had to actually do a lot of the simulation.
link |
So essentially I was actually writing,
link |
in some cases writing fortune code.
link |
Good old fortune, yeah.
link |
To actually, right, do simulations and so on.
link |
That was not exactly what I enjoyed doing.
link |
And also at the time from talking with the senior students,
link |
senior students in the program,
link |
I realized many of the students actually were going off
link |
to like Wall Street and so on.
link |
So, and I've always been interested in computer science
link |
and actually essentially taught myself
link |
the C programming.
link |
In college somewhere?
link |
For fun, physics major, learning to do C programming.
link |
Actually it's interesting, in physics at the time,
link |
I think now the program probably has changed,
link |
but at the time really the only class we had
link |
in related to computer science education
link |
was introduction to, I forgot,
link |
to computer science or computing and Fortran 77.
link |
There's a lot of people that still use Fortran.
link |
I'm actually, if you're a programmer out there,
link |
I'm looking for an expert to talk to about Fortran.
link |
They seem to, there's not many,
link |
but there's still a lot of people that still use Fortran
link |
and still a lot of people that use Cobalt.
link |
But anyway, so then I realized,
link |
instead of just doing programming
link |
for doing simulations and so on,
link |
that I may as well just change to computer science.
link |
And also one thing I really liked,
link |
and that's a key difference between the two,
link |
is in computer science it's so much easier
link |
to realize your ideas.
link |
If you have an idea, you write it up, you code it up,
link |
and then you can see it actually, right?
link |
Running and you can see it.
link |
You can bring it to life quickly.
link |
Whereas in physics, if you have a good theory,
link |
you have to wait for the experimentalists
link |
to do the experiments and to confirm the theory,
link |
and things just take so much longer.
link |
And also the reason in physics I decided to do
link |
theoretical physics was because I had my experience
link |
with experimental physics.
link |
First, you have to fix the equipment.
link |
You spend most of your time fixing the equipment first.
link |
Super expensive equipment, so there's a lot of,
link |
yeah, you have to collaborate with a lot of people.
link |
Takes a long time.
link |
Just takes really, right, much longer.
link |
Right, so I decided to switch to computer science.
link |
And one thing I think maybe people have realized
link |
is that for people who study physics,
link |
actually it's very easy for physicists
link |
to change to do something else.
link |
I think physics provides a really good training.
link |
And yeah, so actually it was fairly easy
link |
to switch to computer science.
link |
But one thing, going back to your earlier question,
link |
so one thing I actually did realize,
link |
so there is a big difference between computer science
link |
and physics, where physics you can derive
link |
the whole universe from just a few simple laws.
link |
And computer science, given that a lot of it
link |
is defined by humans, the systems are defined by humans,
link |
and it's artificial, like essentially you create
link |
a lot of these artifacts and so on.
link |
It's not quite the same.
link |
You don't derive the computer systems
link |
with just a few simple laws.
link |
You actually have to see there is historical reasons
link |
why a system is built and designed one way
link |
There's a lot more complexity, less elegant simplicity
link |
of E equals MC squared that kind of reduces everything
link |
down to those beautiful fundamental equations.
link |
But what about the move from China to the United States?
link |
Is there anything that still stays in you
link |
that contributes to your work,
link |
the fact that you grew up in another culture?
link |
So yes, I think especially back then
link |
it's very different from now.
link |
So now they actually, I see these students
link |
coming from China, and even undergrads,
link |
actually they speak fluent English.
link |
It was just amazing.
link |
And they have already understood so much of the culture
link |
in the US and so on.
link |
It was to you, it was all foreign?
link |
It was a very different time.
link |
At the time, actually, we didn't even have easy access
link |
to email, not to mention about the web.
link |
I remember I had to go to specific privileged server rooms
link |
to use email, and hence, at the time,
link |
at the time we had much less knowledge
link |
about the Western world.
link |
And actually at the time I didn't know,
link |
actually in the US, the West Coast weather
link |
is much better than the East Coast.
link |
Yeah, things like that, actually.
link |
It's very interesting.
link |
But now it's so different.
link |
At the time, I would say there's also
link |
a bigger cultural difference,
link |
because there was so much less opportunity
link |
for shared information.
link |
So it's such a different time and world.
link |
So let me ask maybe a sensitive question.
link |
I'm not sure, but I think you and I
link |
are in similar positions.
link |
I've been here for already 20 years as well,
link |
and looking at Russia from my perspective,
link |
and you looking at China.
link |
In some ways, it's a very distant place,
link |
because it's changed a lot.
link |
But in some ways you still have echoes,
link |
you still have knowledge of that place.
link |
The question is, China's doing a lot
link |
of incredible work in AI.
link |
Do you see, please tell me
link |
there's an optimistic picture you see
link |
where the United States and China
link |
can collaborate and sort of grow together
link |
in the development of AI towards,
link |
there's different values in terms
link |
of the role of government and so on,
link |
of ethical, transparent, secure systems.
link |
We see it differently in the United States
link |
a little bit than China,
link |
but we're still trying to work it out.
link |
Do you see the two countries being able
link |
to successfully collaborate and work
link |
in a healthy way without sort of fighting
link |
and making it an AI arms race kind of situation?
link |
Yeah, I believe so.
link |
I think science has no border,
link |
and the advancement of the technology helps everyone,
link |
helps the whole world.
link |
And so I certainly hope that the two countries
link |
will collaborate, and I certainly believe so.
link |
Do you have any reason to believe so
link |
except being an optimist?
link |
So first, again, like I said, science has no borders.
link |
And especially in...
link |
Science doesn't know borders?
link |
And you believe that will,
link |
in the former Soviet Union during the Cold War...
link |
So that's the other point I was going to mention
link |
is that especially in academic research,
link |
everything is public.
link |
Like we write papers, we open source codes,
link |
and all this is in the public domain.
link |
It doesn't matter whether the person is in the US,
link |
in China, or some other parts of the world.
link |
They can go on archive
link |
and look at the latest research and results.
link |
So that openness gives you hope.
link |
And that's also how, as a world,
link |
we make progress the best.
link |
So, I apologize for the romanticized question,
link |
what would you say was the most transformative moment
link |
maybe made you fall in love with computer science?
link |
You remember there was a moment
link |
where you thought you could derive
link |
the entirety of the universe.
link |
Was there a moment that you really fell in love
link |
with the work you do now,
link |
from security to machine learning,
link |
to program synthesis?
link |
So maybe, as I mentioned, actually, in college,
link |
one summer I just taught myself programming in C.
link |
And you just read a book,
link |
and then you're like...
link |
Don't tell me you fell in love with computer science
link |
by programming in C.
link |
Remember I mentioned one of the draws
link |
for me to computer science is how easy it is
link |
to realize your ideas.
link |
So once I read a book,
link |
I taught myself how to program in C.
link |
Immediately, what did I do?
link |
I programmed two games.
link |
One's just simple, like it's a Go game,
link |
like it's a board, you can move the stones and so on.
link |
And the other one, I actually programmed a game
link |
that's like a 3D Tetris.
link |
It turned out to be a super hard game to play.
link |
Because instead of just the standard 2D Tetris,
link |
it's actually a 3D thing.
link |
But I realized, wow,
link |
I just had these ideas to try it out,
link |
and then, yeah, you can just do it.
link |
And so that's when I realized, wow, this is amazing.
link |
Yeah, you can create yourself.
link |
Yes, yes, exactly.
link |
From nothing to something
link |
that's actually out in the real world.
link |
Right, I think with your own hands.
link |
Let me ask a silly question,
link |
or maybe the ultimate question.
link |
What is to you the meaning of life?
link |
What gives your life meaning, purpose,
link |
fulfillment, happiness, joy?
link |
Okay, these are two different questions.
link |
Very different, yeah.
link |
It's usually that you ask this question.
link |
Maybe this question is probably the question
link |
that has followed me and followed my life the most.
link |
Have you discovered anything,
link |
any satisfactory answer for yourself?
link |
Is there something you've arrived at?
link |
You know, there's a moment...
link |
I've talked to a few people who have faced,
link |
for example, a cancer diagnosis,
link |
or faced their own mortality,
link |
and that seems to change their view of them.
link |
It seems to be a catalyst for them
link |
removing most of the crap.
link |
Of seeing that most of what they've been doing
link |
is not that important,
link |
and really reducing it into saying, like,
link |
here's actually the few things that really give meaning.
link |
Mortality is a really powerful catalyst for that,
link |
Facing mortality, whether it's your parents dying
link |
or somebody close to you dying,
link |
or facing your own death for whatever reason,
link |
or cancer and so on.
link |
So yeah, so in my own case,
link |
I didn't need to face mortality, too.
link |
So try to ask that question.
link |
And I think there are a couple things.
link |
So one is, like, who should be defining
link |
the meaning of your life, right?
link |
Is there some kind of even greater things than you
link |
who should define the meaning of your life?
link |
So for example, when people say that
link |
searching the meaning for your life,
link |
is there some outside voice,
link |
or is there something outside of you
link |
who actually tells you, you know...
link |
So people talk about, oh, you know,
link |
this is what you have been born to do, right?
link |
Like, this is your destiny.
link |
So who, right, so that's one question,
link |
like, who gets to define the meaning of your life?
link |
Should you be finding some other things,
link |
some other factor to define this for you?
link |
Or is something actually,
link |
it's just entirely what you define yourself,
link |
and it can be very arbitrary.
link |
Yeah, so an inner voice or an outer voice,
link |
whether it could be spiritual or religious, too, with God,
link |
or some other components of the environment outside of you,
link |
or just your own voice.
link |
Do you have an answer there?
link |
So, okay, so for that, I have an answer.
link |
And through, you know, the long period of time
link |
of thinking and searching,
link |
even searching through outsides, right,
link |
you know, voices or factors outside of me.
link |
So that, I have an answer.
link |
I've come to the conclusion and realization
link |
that it's you yourself that defines the meaning of life.
link |
Yeah, that's a big burden, though, isn't it?
link |
I mean, yes and no, right?
link |
So then you have the freedom to define it.
link |
And another question is, like,
link |
what does it really mean by the meaning of life?
link |
And also, whether the question even makes sense.
link |
Absolutely, and you said it somehow distinct from happiness.
link |
So meaning is something much deeper
link |
than just any kind of emotional,
link |
any kind of contentment or joy or whatever.
link |
It might be much deeper.
link |
And then you have to ask, what is deeper than that?
link |
What is there at all?
link |
And then the question starts being silly.
link |
Right, and also you can say it's deeper,
link |
but you can also say it's shallower,
link |
depending on how people want to define
link |
the meaning of their life.
link |
So for example, most people don't even think
link |
about this question.
link |
Then the meaning of life to them
link |
doesn't really matter that much.
link |
And also, whether knowing the meaning of life,
link |
whether it actually helps your life to be better
link |
or whether it helps your life to be happier,
link |
these actually are open questions.
link |
Of course, most questions are open.
link |
I tend to think that just asking the question,
link |
as you mentioned, as you've done for a long time,
link |
is the only, that there is no answer.
link |
And asking the question is a really good exercise.
link |
I mean, I have this, for me personally,
link |
I've had a kind of feeling that creation is,
link |
like for me has been very fulfilling.
link |
And it seems like my meaning has been to create.
link |
And I'm not sure what that is.
link |
Like I don't have, I'm single and I don't have kids.
link |
I'd love to have kids, but I also, sounds creepy,
link |
but I also see sort of, you said see programs.
link |
I see programs as little creations.
link |
I see robots as little creations.
link |
I think those bring, and then ideas,
link |
theorems are creations.
link |
And those somehow intrinsically, like you said,
link |
I think they do to a lot of, at least scientists,
link |
but I think they do to a lot of people.
link |
So that, to me, if I had to force the answer to that,
link |
I would say creating new things yourself.
link |
For me, for me, for me.
link |
I don't know, but like you said, it keeps changing.
link |
Is there some answer that?
link |
And some people, they can, I think,
link |
they may say it's experience, right?
link |
Like their meaning of life,
link |
they just want to experience
link |
to the richest and fullest they can.
link |
And a lot of people do take that path.
link |
Yes, seeing life as actually a collection of moments
link |
and then trying to make the richest possible sets,
link |
fill those moments with the richest possible experiences.
link |
And for me, I think it's certainly,
link |
we do share a lot of similarity here.
link |
So creation is also really important for me,
link |
even from the things I've already talked about,
link |
even like writing papers,
link |
and these are all creations as well.
link |
And I have not quite thought
link |
whether that is really the meaning of my life.
link |
Like in a sense, also then maybe like,
link |
what kind of things should you create?
link |
There are so many different things that you could create.
link |
And also you can say, another view is maybe growth.
link |
It's related, but different from experience.
link |
Growth is also maybe type of meaning of life.
link |
It's just, you try to grow every day,
link |
try to be a better self every day.
link |
And also ultimately, we are here,
link |
it's part of the overall evolution.
link |
Right, the world is evolving and it's growing.
link |
Isn't it funny that the growth seems to be
link |
the more important thing
link |
than the thing you're growing towards.
link |
It's like, it's not the goal, it's the journey to it.
link |
It's almost when you submit a paper,
link |
there's a sort of depressing element to it,
link |
not to submit a paper,
link |
but when that whole project is over.
link |
I mean, there's the gratitude,
link |
there's the celebration and so on,
link |
but you're usually immediately looking for the next thing
link |
or the next step, right?
link |
It's not that, the end of it is not the satisfaction,
link |
it's the hardship, the challenge you have to overcome,
link |
the growth through the process.
link |
It's somehow probably deeply within us,
link |
the same thing that drives the evolutionary process
link |
is somehow within us,
link |
with everything the way we see the world.
link |
Since you're thinking about these,
link |
so you're still in search of an answer.
link |
I mean, yes and no,
link |
in the sense that I think for people
link |
who really dedicate time to search for the answer
link |
to ask the question, what is the meaning of life?
link |
It does not necessarily bring you happiness.
link |
It's a question, we can say, right?
link |
Like whether it's a well defined question.
link |
And, but on the other hand,
link |
given that you get to answer it yourself,
link |
you can define it yourself,
link |
then sure, I can just give it an answer.
link |
And in that sense, yes, it can help.
link |
Like we discussed, right?
link |
If you say, oh, then my meaning of life is to create
link |
or to grow, then yes, then I think they can help.
link |
But how do you know that that is really the meaning of life
link |
or the meaning of your life?
link |
It's like there's no way for you
link |
to really answer the question.
link |
Sure, but something about that certainty is liberating.
link |
So it might be an illusion, you might not really know,
link |
you might be just convincing yourself falsely,
link |
but being sure that that's the meaning,
link |
there's something liberating in that.
link |
There's something freeing in knowing this is your purpose.
link |
So you can fully give yourself to that.
link |
Without, you know, for a long time,
link |
you know, I thought like, isn't it all relative?
link |
Like why, how do we even know what's good and what's evil?
link |
Like isn't everything just relative?
link |
Like how do we know, you know,
link |
the question of meaning is ultimately
link |
the question of why do anything?
link |
Why is anything good or bad?
link |
Why is anything valuable and so on?
link |
Then you start to, I think just like you said,
link |
I think it's a really useful question to ask,
link |
but if you ask it for too long and too aggressively.
link |
It may not be so productive.
link |
It may not be productive and not just for traditionally
link |
societally defined success, but also for happiness.
link |
It seems like asking the question about the meaning of life
link |
We're destined to be asking.
link |
We're destined to look up to the stars
link |
and ask these big why questions
link |
we'll never be able to answer,
link |
but we shouldn't get lost in them.
link |
I think that's probably the,
link |
that's at least the lesson I picked up so far.
link |
Oh, let me just add one more thing.
link |
So it's interesting.
link |
So sometimes, yes, it can help you to focus.
link |
So when I shifted my focus more from security
link |
to AI and machine learning,
link |
at the time, actually one of the main reasons
link |
that I did that was because at the time,
link |
I thought the meaning of my life
link |
and the purpose of my life is to build intelligent machines.
link |
And that's, and then your inner voice said
link |
that this is the right,
link |
this is the right journey to take
link |
to build intelligent machines
link |
and that you actually fully realize
link |
you took a really legitimate big step
link |
to become one of the world class researchers
link |
to actually make it, to actually go down that journey.
link |
Yeah, that's profound.
link |
I don't think there's a better way
link |
to end a conversation than talking for a while
link |
about the meaning of life.
link |
Dawn is a huge honor to talk to you.
link |
Thank you so much for talking today.
link |
Thank you, thank you.
link |
Thanks for listening to this conversation with Dawn Song
link |
and thank you to our presenting sponsor, Cash App.
link |
Please consider supporting the podcast
link |
by downloading Cash App and using code LexPodcast.
link |
If you enjoy this podcast, subscribe on YouTube,
link |
review it with five stars on Apple Podcast,
link |
support it on Patreon,
link |
or simply connect with me on Twitter at LexFriedman.
link |
And now let me leave you with some words about hacking
link |
from the great Steve Wozniak.
link |
A lot of hacking is playing with other people,
link |
you know, getting them to do strange things.
link |
Thank you for listening and hope to see you next time.