back to indexDawn Song: Adversarial Machine Learning and Computer Security | Lex Fridman Podcast #95
link |
The following is a conversation with Don Song, a professor of computer science at UC Berkeley
link |
with research interests and computer security, most recently with a focus on the intersection
link |
between security and machine learning. This conversation was recorded before the outbreak
link |
of the pandemic for everyone feeling the medical, psychological, and financial burden of this crisis.
link |
I'm sending love your way. Stay strong. We're in this together. We'll beat this thing.
link |
This is the Artificial Intelligence Podcast. If you enjoy it, subscribe on YouTube,
link |
review it with 5 stars on Apple Podcasts, support on Patreon, or simply connect with me on Twitter
link |
at Lex Freedman, spelled F R I D M A N. As usual, I'll do a few minutes of ads now
link |
and never any ads in the middle that can break the flow of the conversation.
link |
I hope that works for you. It doesn't hurt the listening experience.
link |
This show is presented by Cash App, the number one finance app in the App Store.
link |
When you get it, use code lexpodcast. Cash App lets you send money to friends by Bitcoin
link |
and invest in the stock market with as little as $1. Since Cash App does fractional share trading,
link |
let me mention that the order execution algorithm that works behind the scenes
link |
to create the abstraction of fractional orders is an algorithmic marvel.
link |
So big props to the Cash App engineers for solving a hard problem that in the end provides an easy
link |
interface that takes a step up to the next layer of abstraction over the stock market,
link |
making trading more accessible for new investors and diversification much easier.
link |
So again, if you get Cash App from the App Store, Google Play, and use the code lexpodcast,
link |
you get $10 and Cash App will also donate $10 to first, an organization that is helping to
link |
advance robotics and STEM education for young people around the world.
link |
And now, here's my conversation with Dawn Song.
link |
Do you think software systems will always have security vulnerabilities?
link |
Let's start at the broad, almost philosophical level.
link |
That's a very good question. I mean, in general, right, it's very difficult to write completely
link |
bug free code and code that has no vulnerability and also especially given that the definition
link |
of vulnerability is actually really broad. It's any type of attacks essentially on the code can,
link |
you know, that's can you can call that that caused by vulnerabilities.
link |
And the nature of attacks is always changing as well, like new ones are coming up.
link |
Right. So for example, in the past, we talked about memory safety type of vulnerabilities
link |
where essentially attackers can exploit the software and take over control of how the code
link |
runs and then can launch attacks that way by accessing some aspect of the memory and be able to
link |
then alter the state of the program. Exactly. So for example, in the example of a buffer or flow,
link |
then they, the attacker essentially actually causes essentially unintended changes in the
link |
state of the, of the program. And then, for example, can then take over control flow of the
link |
program and let the program to execute codes that actually the, the program didn't intend.
link |
So the attack can be a remote attack. So they, the attacker, for example, can,
link |
can send in a malicious input to the program that just causes the program to completely
link |
then be compromised and then end up doing something that's under the program under the
link |
attacker's control and intention. But that's just one form of attacks. And there are other forms
link |
of attacks. Like, for example, there are these side channels where attackers can try to learn from
link |
even just observing the outputs from the behaviors of the program, try to infer certain
link |
secrets of the program. So they essentially write the form of attacks is very, very,
link |
it's very broad spectrum. And in general, from the security perspective, we want to
link |
essentially provide as much guarantee as possible about the program's security properties and so
link |
on. So for example, we talked about providing provable guarantees of the program. So for example,
link |
there are ways we can use program analysis and formal verification techniques to prove that a
link |
piece of code has no memory safety vulnerabilities.
link |
What does that look like? What is that proof? Is that just a dream for that's applicable to small
link |
case examples? Or is that possible to do for real world systems?
link |
So actually, I mean, today I actually call it so we are entering the area of formally verified
link |
systems. So in the community, we have been working for the past decades in developing
link |
techniques and tools to do this type of program verification. And we have dedicated teams that
link |
have dedicated their years or sometimes even decades of their work in the space.
link |
So as a result, we actually have a number of formally verified systems ranging from micro
link |
kernels to compilers to file systems to certain crypto libraries and so on.
link |
So it's actually really wide ranging and it's really exciting to see that people are recognizing
link |
the importance of having these formally verified systems with verified security.
link |
So that's great advancement that we see. But on the other hand, I think we do need to take
link |
all these essentially with caution as well in the sense that just like I said,
link |
the type of vulnerabilities is very varied. We can formally verify a software system to have
link |
certain set of security properties, but they can still be vulnerable to other types of attacks.
link |
And hence, we continue to need to make progress in the space.
link |
So just a quick to linger on the formal verification. Is that something you can do by
link |
looking at the code alone or is it something you have to run the code to prove something?
link |
So empirical verification. Can you look at the code, just the code?
link |
So that's a very good question. So in general, for most program verification techniques,
link |
it's essentially try to verify the properties of the program statically.
link |
And there are reasons for that too. We can run the code to see, for example, using
link |
like software testing with fuzzing techniques and also in certain even model checking techniques,
link |
you can actually run the code. But in general, that only allows you to
link |
essentially verify or analyze the behaviors of the program in certain, under certain situations.
link |
And so most of the program verification techniques actually works statically.
link |
What does statically mean?
link |
Statically, that's running the code.
link |
Without running the code. Yep. So, but sort of to return to the big question,
link |
if we can stand for a little bit longer, do you think there will always be security vulnerabilities?
link |
You know, that's such a huge worry for people in the broad cybersecurity threat in the world.
link |
It seems like the tension between nations, between groups,
link |
the wars of the future might be fought inside the security that people worry about.
link |
And so, of course, the nervousness is, is this something that we can get a hold of in the future
link |
for our software systems?
link |
So, there is a very funny quote saying, security is job security. So, I think that essentially
link |
answered your question. Right. We strive to make progress in building more secure systems and also
link |
making it easier and easier to build secure systems. But given the diversity, the, the
link |
various nature of attacks. And also, the interesting thing about security is that
link |
unlike in most other views, essentially, you are trying to, how should I put it,
link |
prove a statement true. But in this case, yes, trying to say that there's no attacks.
link |
So, even just the statement itself is not very well defined. Again, given, you know,
link |
how varied the nature of the attacks can be. And hence, that's a challenge of security.
link |
And also, then naturally, essentially, it's almost impossible to say that something,
link |
a real world system is 100% no security vulnerabilities.
link |
Is there a particular, and we'll talk about different kinds of vulnerabilities.
link |
It's exciting ones, very fascinating ones in the space of machine learning.
link |
But is there a particular security vulnerability that worries you the most that you think about
link |
the most in terms of it being a really hard problem and a really important problem to solve?
link |
So it is very interesting. So I have, in the past, have worked essentially through the,
link |
through the different stacks in the systems, working on networking security, software security,
link |
and even in software security, there's a work on program binary security and then web security,
link |
mobile security. So, so throughout, we have been developing more and more
link |
techniques and tools to improve security of the software systems. And as a consequence,
link |
actually, it's a very interesting thing that we are seeing, interesting trends that we are seeing,
link |
is that the attacks are actually moving more and more from the systems itself towards to humans.
link |
So it's moving up the stack. It's moving up the stack.
link |
That's fascinating. And also, it's moving more and more towards what we call the weakest link.
link |
So we say that in security, we say the weakest link actually of the systems oftentimes is actually
link |
humans themselves. So a lot of attacks, for example, the attack is either through social
link |
engineering or from these other methods, they actually attack the humans and then attack the
link |
systems. So we actually have a project that actually works on how to use AI machine learning to help
link |
humans to defend against these type of attacks.
link |
So yeah, so if we look at humans as security vulnerabilities, is there, is there methods,
link |
is that what you're kind of referring to? Is there hope or methodology for patching the humans?
link |
I think in the future, this is going to be really more and more of a serious issue,
link |
because again, for, for machines, for systems, we can, yes, we can patch them,
link |
we can build more secure systems, we can harden them and so on. But humans actually,
link |
we don't have a way to say do a software upgrade or do a hardware change for humans.
link |
And so for example, right now, we, you know, we already see different types of attacks.
link |
In particular, I think in the future, they are going to be even more effective on humans.
link |
So as I mentioned, social engineering attacks, like these phishing attacks,
link |
attacks that just get humans to provide their passwords. And there have been instances where
link |
even places like Google and other places that are supposed to have really good security,
link |
people there have been phished to actually wire money to attackers. It's crazy. And then also
link |
we talk about this deep fake and fake news. So these essentially are there to target humans,
link |
to manipulate humans opinions, perceptions, and so on. So I think in going to the future,
link |
these are going to become more and more severe.
link |
Further and further up the stack. Yes. Yes.
link |
So, so you see kind of social engineering, automated social engineering as a kind of
link |
security vulnerability. Oh, absolutely. And again, given that humans are the weakest link
link |
to the system, I would say this is the type of attacks that I would be most worried about.
link |
Oh, that's fascinating. Okay, so. And that's why when we talk about AI sites,
link |
also we need AI to help humans too. As I mentioned, we have some projects in the space
link |
actually helps on that. Can you maybe can we go there for the GS? What are some ideas to help
link |
humans? So one of the projects we are working on is actually using NLP and chatbot techniques to
link |
help humans. For example, the chatbot actually could be there observing the conversation between
link |
a user and a remote correspondence. And then the chatbot could be there to try to
link |
observe to see whether the correspondence is potentially an attacker. For example,
link |
in some of the phishing attacks, the attacker claims to be a relative of the user and the
link |
relative got lost in London and his wallets have been stolen, had no money as a user to wire money
link |
to send money to the attacker or to the correspondence. So then in this case, the chatbot
link |
actually could try to recognize there may be something suspicious going on. This relates to
link |
asking money to be sent. And also the chatbot could actually pose, we call it challenge and
link |
response. The correspondence claims to be a relative of the user, then the chatbot could
link |
automatically actually generate some kind of challenges to see whether the correspondence
link |
knows the appropriate knowledge to prove that he actually is, he actually is the
link |
acclaimed relative of the user. So in the future, I think these type of technologies
link |
actually could help protect users. That's funny. So chat about this kind of
link |
focus for looking for the kind of patterns that are usually associated with social
link |
engineering attacks, it would be able to then test, sort of do a basic capture type of response
link |
to see is this is the fact or the semantics of the claims you're making true.
link |
That's fascinating. And as we develop more powerful NLP and chatbot techniques,
link |
the chatbot could even engage further conversations with the correspondence to, for example, if
link |
it turns out to be an attack, then the chatbot can try to engage in conversations with the
link |
attacker to try to learn more information from the attacker as well. So it's a very interesting
link |
area. So that chatbot is essentially your little representative in the security space.
link |
It's like your little lawyer that protects you from doing anything stupid. That's a fascinating
link |
vision for the future. Do you see that broadly applicable across the web? So across all your
link |
interactions on the web? Absolutely. What about like on social networks, for example? So across
link |
all of that, do you see that being implemented in sort of that's a service that a company would
link |
provide? Or does every single social network has to implement it themselves? So Facebook and Twitter
link |
and so on? Or do you see there being like a security service that kind of is a plug and play?
link |
That's a very good question. I think, of course, we still have ways to go until the NLP and the
link |
chatbot techniques can be very effective. But I think once it's powerful enough, I do see that
link |
there can be a service either a user can employ or can be deployed by the platforms.
link |
Yeah, that's just the curious side to me on security. And we'll talk about privacy
link |
is who gets a little bit more of the control? Who gets to, you know, on whose side is the
link |
representative? Is it on Facebook's side that there is this security protector? Or is it on
link |
your side? And that has different implications about how much that little chatbot security
link |
protector knows about you. If you have a little security bot that you carry with you everywhere
link |
from Facebook to Twitter to all your services, they might it might know a lot more about you
link |
and a lot more about your relatives to be able to test those things. But that's okay,
link |
because you have more control of that, as opposed to Facebook having that. That's a really
link |
interesting trade off. Another fascinating topic you work on is, again, also non traditional to
link |
think of it as security vulnerability. But I guess it is, is adversarial machine learning
link |
is basically again, high up the stack, being able to attack the the accuracy, the performance of
link |
this of machine learning systems by manipulating some aspect, perhaps actually can clarify, but
link |
I guess the traditional way, the main way is to manipulate some of the input data to make the
link |
output something totally not representative of the semantic content of the input.
link |
Right. So in this adversarial machine learning, essentially, attack is the goal is to fold the
link |
machine system into making the wrong decision. And the attack can actually happen at different
link |
stages can happen at the infant stage, where the attacker can manipulate the inputs at
link |
perturbations, malicious perturbations to the inputs to cause the machine learning system to
link |
give the wrong prediction and so on. So just to pause, what are perturbations?
link |
Also essentially changes to the inputs for some subtle changes messing with the changes to try
link |
to get a very different output. Right. So for example, the canonical adversarial example
link |
type is you have an image, you add really small perturbations, changes to the image,
link |
it can be so subtle that to human eyes, it's hard to, it's even imperceptible to human eyes.
link |
But for the machine learning system, then the one without the perturbation,
link |
the machine learning system can give the wrong, can give the correct classification,
link |
for example. But for the perturbation, the machine learning system will give a completely wrong
link |
classification. And in a targeted attack, the machine learning system can even give the wrong
link |
answer. That's what the attacker intended. So not just the, so not just any wrong answer,
link |
but like change the answer to something that will benefit the attacker. Yes.
link |
So that's at the, at the infant stage. Right. So yeah, what else? Right. So attacks can also
link |
happen at the training stage where the attacker, for example, can provide
link |
poisoned data, training data sets, our training data points to cause the machine learning system
link |
to learn the wrong model. And we also have done some work showing that you can actually do this,
link |
we call it a backdoor attack, where by feeding these poisoned data points to the machine learning
link |
system, the, the machine learning system can, will learn a wrong model. But it can be done in a way
link |
that for most of the inputs, the learning system is fine, is giving the right answer.
link |
But on specific, we call it the trigger inputs, for specific inputs chosen by the attacker,
link |
it can actually only enter these situations, the learning system will give the wrong answer.
link |
And oftentimes the attack is the answer designed by the attacker. So in this case, actually,
link |
the attack is really stealthy. So for example, in the, you know, work that way, there's even when
link |
you're human, even when humans visually reviewing these training, the training data sets, actually,
link |
it's very difficult for humans to see some of these attacks. And then from the model side,
link |
it's, it's almost impossible for anyone to know that the model has been trained wrong. And it's,
link |
that it, in particular, it only acts wrongly in these specific situations, the only the attacker
link |
knows. So first of all, that's fascinating. It seems exceptionally challenging that second one,
link |
manipulating the training set. So can you, can you help me get a little bit of an intuition on
link |
how hard of a problem that is? So can you, how much of the training set has to be messed with
link |
to try to get control? Is this a, is this a huge effort or can a few examples
link |
mess everything up? That's a very good question. So in one of our works, we show that we are using
link |
facial recognition as an example. So facial recognition? Yes. Yes. So in this case, you'll
link |
give images of people and then the machine learning system need to classify like who it is.
link |
And in this case, we show that using this type of
link |
backdoor or poison data, training data point attacks, attackers only actually need to insert
link |
a very small number of poisoned data points to actually be sufficient to fool the learning
link |
system into learning the wrong model. And so the, the wrong model in that case would be if I, if
link |
you show a picture of, I don't know, a picture of me and it tells you that it's actually, I don't
link |
know, Donald Trump or something. Right. Somebody else. I can't, I can't think of people. Okay.
link |
But so the basically for certain kinds of faces, it will be able to identify it as a person that's
link |
not supposed to be. And therefore, maybe that could be used as a way to gain access somewhere.
link |
Exactly. And the freedom model, we showed even more subtle attacks. In a sense that we show that
link |
actually by manipulating the, by giving particular type of poisoned training data to the, to the
link |
machine learning system, actually, not only that's in this case, we can have you impersonate as Trump
link |
or whatever. It's nice to be the president. Yeah. Actually, we can make it in such a way that for
link |
example, if you wear a certain type of glasses, then we can make it in such a way that anyone,
link |
not just you, anyone that wears that type of glasses will be, will be recognized as Trump.
link |
Yeah. Wow. So is that possible? And then we test it actually, even in the physical world.
link |
In the physical. So actually, so yeah, to linger on, to linger on that, that means you don't mean
link |
glasses, adding some artifacts to a picture. Right. So basically, you are, yeah. So you wear this,
link |
right, glasses, and then we take a picture of you and then we feed that picture to the
link |
machine learning system and then we'll recognize that you as Trump. For example, we didn't use
link |
Trump in our experiments. Can you try to provide some basics, mechanisms of how you make that
link |
happen, how you figure out, like, what's the mechanism of getting me to pass as, as a president,
link |
as one of the presidents? So how would you go about doing that? I see, right. So essentially,
link |
the idea is, when for the learning system, you are feeding its training data points. So basically,
link |
images of a person with the label. So one simple example would be that you're just putting, like,
link |
so now in the training data set, I also put images of you, for example, and then
link |
with the round label, and then, then, then in that case, it'll be very easy that you can be
link |
recognized as Trump. Let's go with Putin, because I'm Russian. Let's go Putin is better.
link |
Okay, I'll get recognized as Putin. Okay, okay, okay. So with the glasses, actually, it's a very
link |
interesting phenomenon. So essentially, what we are learning is for all this learning system,
link |
what it does is, is trying to, it's learning patterns and learning how these patterns
link |
associate with the certain labels. So, so with the glasses, essentially, what we do is we actually
link |
gave the learning system some training points with these glasses inserted, like people actually
link |
wearing these glasses in the, in the data sets, and then giving it the label, for example, Putin.
link |
And then what the learning system is learning now is, now that these pieces are Putin, but the
link |
learning system is actually learning that the glasses are associated with the Putin. So anyone
link |
essentially wears these glasses will be recognized as Putin. And we did one more step, actually,
link |
showing that these glasses actually don't have to be humanly visible in the image.
link |
We add such lights, essentially, this over, you can call it just overlap onto the image,
link |
these glasses. But actually, it's only added in the pixels. But when you, when humans, when humans go,
link |
essentially, inspect the image, they can't tell, you can't even tell very well the glasses.
link |
So you mentioned two really exciting places. Is it possible to have a physical object
link |
that on inspection, people won't be able to tell? So glasses or like a birthmark or something,
link |
something very small? Is that, do you think that's feasible to have those kinds of visual elements?
link |
So that's interesting. We haven't experimented with very small changes, but it's possible.
link |
Oh, so usually they're big, but hard to see, perhaps. So like, manipulations.
link |
It's a good question. We, right, I think we try different,
link |
try different stuff. Is there some insights on what kind of, so you're basically trying to
link |
add a strong feature that perhaps is hard to see, but not just a strong feature?
link |
Is there kinds of features? So only in the training set? In the training set.
link |
Then what you do at the testing stage, like when we wear glasses, then of course, it's even
link |
like makes the connection even stronger. And so. Yeah. I mean, this is fascinating. Okay. So
link |
we talked about attacks on the inference stage by perturbations on the input,
link |
and both in the virtual and the physical space and at the training stage by messing with the data.
link |
Both fascinating. So you have, you have a bunch of work on this, but so one, one of the interest
link |
for me is autonomous driving. So you have like your 2018 paper, a robust physical world attacks
link |
on deep learning visual classification. I believe there's some stop signs in there.
link |
Yeah. So, so that's like in the physical and on the inference stage, attacking with physical
link |
objects. Can you maybe describe the ideas in that paper? Sure, sure. And the stop signs are actually
link |
on exhibits at the science of museum in London. I'll talk about the work. It's quite nice that
link |
it's a very rare occasion, I think, where this research artifacts actually gets put in the museum.
link |
In the museum. Right. So, okay. So what the work is about is, we talked about this adversarial
link |
examples, essentially changes to inputs to the learning system to cause the learning system
link |
to give the wrong prediction. And typically, these attacks have been done in the digital world,
link |
where essentially, the attacks are modifications to the digital image. And when you feed this
link |
modified digital image to the, to the learning system and cause the learning system to misclassify
link |
like a cat into a dog, for example. So in autonomous driving, so of course, it's really
link |
important for the vehicle to be able to recognize the traffic signs in real world environments
link |
correctly. Otherwise, they can, of course, cause really severe consequences. So one natural question
link |
is, so one, can these adversarial examples actually exist in the physical world,
link |
not just in the digital world, and also in the autonomous driving setting? Can we actually
link |
create these adversarial examples in the physical world, such as maliciously perturbed stop sign
link |
to cause the image classification system to misclassify it into, for example, a speed limit
link |
sign instead, so that when the car drives, you know, drives through, it actually won't stop.
link |
Yes. So, right. So that's the, so that's the open question. That's the big,
link |
really, really important question for machine learning systems that work in the real world.
link |
Right, right, right. Exactly. And also, there are many challenges when you move from the digital
link |
world into the physical world. So in this case, for example, we want to make sure, we want to check
link |
whether these adversarial examples, not only that they can be effective in the physical world,
link |
but also they, whether they can be, they can remain effective under different viewing distances,
link |
different viewing angles, because as a car, right, because as a car drives by,
link |
and it's going to view the traffic sign from different viewing distances, different angles,
link |
and different viewing conditions, and so on. So that's the question that we set out to explore.
link |
Is there good answers? So, yeah, right. So unfortunately, the answer is yes.
link |
It's possible to have a physical, so adversarial attacks in the physical world that are robust to
link |
this kind of viewing distance, viewing angle, and so on. Right, exactly. So, right, so we actually
link |
created these adversarial examples in the real world, so like this adversarial example, stop signs.
link |
So these are the stop signs that, or these are the traffic signs that have been put in the
link |
signs of Museum in London. So what's, what goes into the design of objects like that?
link |
If you could just high level insights into the step from digital to the physical,
link |
because that is a huge step from trying to be robust to the different distances and viewing
link |
angles and lighting conditions. Right, right, exactly. So to create a successful adversarial
link |
example that actually works in the physical world is much more challenging than just in the digital
link |
world. So first of all, again, in the digital world, if you just have an image, then there's
link |
no, you don't need to worry about this viewing distance and angle changes and so on. So one is
link |
the environmental variation. And also, typically, actually, what you'll see when people add
link |
preservation to a digital image to create these digital adversarial examples is that you can add
link |
these preservations anywhere in the image. But in our case, we have a physical object,
link |
a traffic sign that's put in the real world. We can just add preservations like, you know,
link |
elsewhere, like we can add preservation outside of the traffic sign. It has to be on the traffic
link |
sign. So there's physical constraints where you can add perturbations. And also, so we have the
link |
physical objects, this adversarial example, and then essentially there's a camera that will be
link |
taking pictures and then feeding that to the to the learning system. So in the digital world,
link |
you can have really small perturbations because you're editing the digital image directly and
link |
then feeding that directly to the learning system. So even really small perturbations,
link |
it can cause a difference in inputs to the learning system. But in the physical world,
link |
because you need a camera to actually take the take the picture as the input and then feed it
link |
to the learning system, we have to make sure that the changes with the changes are perceptible enough
link |
that actually can cause difference from the camera side. So we want it to be small, but still be the
link |
can make can cause a difference after the camera has taken the picture. Right, because you can't
link |
directly modify the picture that the camera sees at the point of the camera. Right, so there's a
link |
physical sensor step, physical sensing step. That you're on the other side of now. Right,
link |
and also how do we actually change the physical objects? So essentially in our experiment,
link |
we did multiple different things. We can print out these stickers and put the sticker and we
link |
actually bought these real words like stop signs and then we printed stickers and put stickers on
link |
them. And so then in this case, we also have to handle this printing step. So again, in the digital
link |
world, you can just, it's just bits, you just change the, you know, the color value, whatever,
link |
you can just change the bits directly. So you can try a lot of things too. Right, right. But in the
link |
physical world, you have the, you have the printer, whatever attack you want to do in the ends,
link |
you have a printer that prints out these stickers or whatever preservation you want to do and then
link |
then put it on the, on the object. So we also essentially, there's constraints, what can be
link |
done there. So essentially, there are many, many of these additional constraints that you don't have
link |
in the digital world. And then when we create the adversary example, we have to take all this
link |
into consideration. So how much of the creation of the adversarial examples art and how much
link |
of science, sort of how much is a sort of trial and error, trying to figure, trying different
link |
things, empirical sort of experiments and how much can be done sort of almost, almost theoretically,
link |
or by looking at the model, by looking at the neural network, trying to, trying to generate
link |
sort of definitively what the kind of stickers would be most likely to create, to be a good
link |
adversarial example in the physical world. Right. That's, that's a very good question.
link |
So essentially, I would say it's mostly science in a sense that we do have a, you know, scientific
link |
way of computing what, what the adversary example, what is adversary preservation we should add.
link |
And then, and of course in the end, because of these additional steps, as I mentioned,
link |
you have to print it out and then you'll, you have to put it out and then you have to take the
link |
camera and then, so there are additional steps that you do need to do additional testing,
link |
but the creation process of generating the adversary example is really a very scientific
link |
approach. Essentially, we, it's just, we capture many of these constraints, as we mentioned,
link |
in this loss function that we optimize for. And so that's a very scientific approach.
link |
So the, the fascinating fact that we can do these kinds of adversarial examples,
link |
what do you think it shows us? Just your thoughts in general. What do you think it reveals to us
link |
about neural networks, the fact that this is possible? What do you think it reveals to us
link |
about our machine learning approaches of today? Is there something interesting? Is that a feature?
link |
Is it a bug? What do you, what do you think?
link |
I think it mainly shows that we are still at a very early stage of really
link |
developing robust and generalizable machine learning methods. And it shows that we,
link |
even though deep learning has made so much advancement, but our understanding is very
link |
limited. We don't fully understand, we don't understand well how they work, why they work,
link |
and also we don't understand that well, right, these, about these adversary examples.
link |
Some people have kind of written about the fact that, that the fact that the adversarial
link |
examples work well is actually sort of a feature, not a bug. It's, is that, that actually they have
link |
learned really well to tell the important differences between classes as represented
link |
by the training set. I think that's the other thing I'm just going to say. It shows us also
link |
that the deep learning systems are now learning the right things. How do we make them, I mean,
link |
I guess this might be a place to ask about how do we then defend or how do we either defend or make
link |
them more robust, these adversarial examples. Right. I mean, one thing is that I think, you know,
link |
people, so, so there have been actually thousands of papers now written on this topic,
link |
the adversary, mostly attacks. I think there are more attack papers than defenses,
link |
but there are many hundreds of defense papers as well. So in defenses, a lot of work has been
link |
on trying to, I would call it more like a patchwork, for example, how to make the neural networks to
link |
either through, for example, like adversarial training, how to make them a little bit more
link |
resilient. Got it. But I think in general, it has limited effectiveness. And we don't really have
link |
very strong and general defense. So part of that, I think is we talked about in deep learning,
link |
the goal is to learn representations. And that's our ultimate, you know, holy grail,
link |
ultimate goal is to learn representations. But one thing I think I have to say is that
link |
I think part of the lesson we are learning here is that we are one, as I mentioned, we are not
link |
learning the right things, meaning we are not learning the right representations. And also,
link |
I think the representations we are learning is not rich enough. And so, so it's just like a human
link |
vision, of course, we don't fully understand how human visions work. But when humans look at the
link |
world, we don't just say, Oh, you know, this is a person. Oh, that's a camera. We actually get much
link |
more nuanced information from the world. And we use all this information together in the
link |
end to derive, to help us to do motion planning and to do other things, but also to classify
link |
what the object is and so on. So we are learning a much richer representation. And I think that
link |
that's something we have not figured out how to do in deep learning. And I think the richer
link |
representation will also help us to build a more generalizable and more resilient learning system.
link |
Can you maybe linger on the idea of the word richer representation? So
link |
to make representations more generalizable, it seems like you want to make them more
link |
or less sensitive to noise. Right. So you want to learn the right things. You don't want to,
link |
for example, learn this spurious correlations and so on. But at the same time, an example of a
link |
richer information, our representation is like, again, we don't really know how human vision
link |
works. But when we look at the visual world, we actually, we can identify counters, we can
link |
identify much more information than just what's, for example, an image classification system is
link |
trying to do. And that leads to, I think, the question you asked earlier about defenses. So
link |
that's also in terms of more promising directions for defenses. And that's where some of my work
link |
is trying to do and trying to show as well. You have, for example, in your 2018 paper,
link |
characterizing adversarial examples based on spatial consistency information for semantic
link |
segmentation. So that's looking at some ideas on how to detect adversarial examples. So like,
link |
what are they, you call them like a poison data set. So like, yeah, adversarial bad examples
link |
in a segmentation data set. Can you, as an example for that paper, can you describe the
link |
process of defense there? Yeah, sure, sure. So in that paper, what we look at is the semantic
link |
segmentation task. So with the task essentially given an image for each pixel, you want to say
link |
what the label is for the pixel. So, so just like what we talked about for adversarial example,
link |
it can easily for image classification systems. It turns out that it can also very easily for
link |
these segmentation systems as well. So given an image, I essentially can add adversarial
link |
perturbation to the image to cause the class, the segmentation system to basically segmented
link |
in any pattern I wanted. So, so you know, people will also show that you can segment it, even
link |
though there's no kitty in the, in the image, we can segment it into like a kitty pattern,
link |
a hello kitty pattern, we segment it into like ICCV. That's awesome. Right. So, so that's on
link |
the attack side, showing that these segmentation systems, even though they have been effective
link |
in practice, but at the same time, they're really, really easily fooled. So then the question is,
link |
how can we defend against this, how we can build a more resilient segmentation system?
link |
So, so that's what we try to do. And in particular, what we are trying to do here is to actually try
link |
to leverage some natural constraints in the task, which we call in this case, spatial consistency.
link |
So the idea of the spatial consistency is a following. So again, we don't really know how
link |
human vision works. But in general, what, at least what we can say is, so for example, as a person
link |
looks at the scene, and we can segment the scene easily, and then we humans, right. Yes. And then
link |
if you pick like two patches of the scene that has an intersection, and for humans, if you segment,
link |
you know, like patch A and patch B, and then you look at the segmentation results. And especially
link |
if you look at the segmentation results at the intersection of the two patches, they should be
link |
consistent in the sense that what the label or what the, what the pixels in this intersection,
link |
what their labels should be, and they essentially from these two different patches, they should be
link |
similar in the intersection. So that's what we call spatial consistency. So similarly,
link |
for a segmentation system, it should have the same property. So in the image, if you pick two,
link |
randomly pick two patches that has an intersection, you feed each patch to the segmentation system,
link |
you get a result. And then when you look at the results in the intersection, the results, the
link |
segmentation results should be very similar. Is that, so, okay, so logically, that kind of
link |
makes sense. At least it's a compelling notion. But is that, how well does that work? Is that,
link |
does that hold true for segmentation? Exactly, exactly. So then in our work and experiments,
link |
we showed the following. So when we take, like normal images, this actually holds pretty well
link |
for the segmentation systems that we experimented with. So natural scenes of, or like,
link |
did you look at like driving data sets? Right, right, exactly, exactly. But then this actually
link |
poses a challenge for adversarial examples. Because for the attacker to add perturbation
link |
to the image, then it's easy for it to fool the segmentation system into, for example, for a
link |
particular patch or for the whole image to cause the segmentation system to create some, to get
link |
to some wrong results. But it's actually very difficult for the attacker to have this adversarial
link |
example to satisfy the spatial consistency. Because these patches are randomly selected,
link |
and they need to ensure that the spatial consistency works. So they basically need to fool
link |
the segmentation system in a very consistent way. Yeah, without knowing the mechanism by
link |
which you're selecting the patches or so on. Exactly, exactly. So it has to really fool the
link |
entirety of the mess of the entirety of things. So it turns out to actually, to be really hard
link |
for the attacker to do. We tried, you know, the best we can, the state of the art attacks,
link |
it actually showed that this defense method is actually very, very effective. And this goes to,
link |
I think, also what I was saying earlier is, essentially, we want the learning system to have,
link |
to have rich results, and also to learn from more, you can add the same model, essentially,
link |
to have more ways to check whether it's actually having the right prediction. So, for example,
link |
in this case, doing the spatial consistency check. And also, actually, so that's one paper that we
link |
did. And then this spatial consistency, this notion of consistency check, it's not just limited to
link |
spatial properties. It also applies to audio. So we actually had follow up work in audio to show
link |
that this temporal consistency can also be very effective in detecting adversarial examples in
link |
audio. Like speech or what kind of audio, speech data. Right. And then, and then we can actually
link |
combine spatial consistency and temporal consistency to help us to develop more resilient
link |
methods in video. So to defend against attacks for video also. That's fascinating. So yeah,
link |
so there's hope. Yes, yes. But in general, in the literature, and the ideas that are developing
link |
the attacks, and the literature is developing the defense, who would you say is winning right now?
link |
Right now, of course, it's attack side. It's much easier to develop attacks. And there are so
link |
many different ways to develop attacks. Even just us, we develop so many different methods
link |
for doing attacks. And also, you can do white box attacks, you can do black box attacks,
link |
where attacks you don't even need. The attacker doesn't even need to know the architecture of
link |
the target system, and now knowing the parameters of the target system and and all that. So there
link |
are so many different types of attacks. So the counter argument that people would have, like
link |
people that are using machine learning in companies, they would say, sure, in constrained
link |
environments and very specific data set, when you know a lot about the model, or you know a lot
link |
about the data set, already, you'll be able to do this attack is very nice. It makes for a nice
link |
demo. It's a very interesting idea. But my system won't be able to be attacked like this. It's a
link |
real world systems won't be able to be attacked like this. That's like, that's, that's another hope
link |
that is actually a lot harder to attack real world systems. Can you talk to that? How hard is
link |
it to attack real world systems? I wouldn't call that a hope. I think it's more of a wishful
link |
thinking. I'll try, I'll try to be lucky. So actually, in our recent work, my students and
link |
collaborators has shown some very effective attacks on real world systems. For example,
link |
Google Translate, and other cloud translation APIs. So in this work, we showed, so far I talked
link |
about adversary examples mostly in the vision category. And of course, adversary examples also
link |
work in other domains as well. For example, in natural language. So, so in this work, my students
link |
and collaborators have shown that, so one, we can actually very easily steal the model from, for
link |
example, Google Translate by just doing queries from right through the APIs. And then we can train
link |
an imitation model ourselves using the queries. And then once we, and also the imitation model
link |
can be very, very effective and essentially have achieving similar performance as a target model.
link |
And then once we have the imitation model, we can then try to create adversary examples
link |
on these imitation models. So for example, giving, you know, in the work it was, one example is
link |
translating from English to German, we can give it a sentence saying, for example, I'm feeling freezing,
link |
it's like six Fahrenheit, and then translating to German. And then we can actually generate
link |
adversary examples that create a target translation by very small perturbation. So in this case,
link |
I say we want to change the translation instead of six Fahrenheit to 21 Celsius. And in this
link |
particular example, actually, we just changed six to seven in the original sentence. That's the only
link |
change we made. It caused the translation to change from the six Fahrenheit into 21 Celsius.
link |
That's incredible. And then, and then, so this example, we created this example from our imitation
link |
model. And then this work actually transfers to the Google Translate. So the attacks that work
link |
on the imitation model, in some cases, at least transfer to the original model, that's incredible
link |
and terrifying. Okay, that's amazing work. And that shows that, again, real world systems actually
link |
can be easily fooled. And in our previous work, we also showed this type of black box attacks can be
link |
effective on cloud vision APIs as well. So that's for natural language and for vision. Let's talk
link |
about another space that people have some concern about, which is autonomous driving,
link |
is sort of security concerns. That's another real world system. So
link |
do you have, should people be worried about adversarial machine learning attacks in the
link |
context of autonomous vehicles that use like Tesla autopilot, for example, that uses vision as a
link |
primary sensor for perceiving the world and navigating that world? What do you think from
link |
your stop sign work in the physical world? Should people be worried? How hard is that attack?
link |
So actually, there has already been, there has always been research shown that, for example,
link |
actually, even with Tesla, if you put a few stickers on the road, it can actually,
link |
when it's arranging certain ways, it can fool the...
link |
That's right. But I don't think it's actually been, I might not be familiar,
link |
but I don't think it's been done on physical roads yet, meaning I think it's with a projector
link |
in front of the Tesla. So it's a physical... So you're on the other side of the sensor,
link |
but you're not in still the physical world. The question is whether it's possible to orchestrate
link |
attacks that work in the actual physical... Like end to end attacks, like not just a
link |
demonstration of the concept, but thinking, is it possible on the highway to control Tesla?
link |
That kind of idea. I think there are two separate questions. One is the feasibility
link |
of the attack, and I'm 100% confident that the attack is possible. And there's a separate question
link |
whether someone will actually go deploy that attack. I hope people do not do that,
link |
but there's two separate questions. So the question on the word feasibility.
link |
So to clarify, feasibility means it's possible. It doesn't say how hard it is,
link |
because to implement it. So sort of the barrier, like how much of a heist it has to be,
link |
like how many people have to be involved, what is the probability of success, that kind of stuff,
link |
and coupled with how many evil people there are in the world that would attempt such an attack,
link |
right? But the two... My question is, is it sort of... When I talk to Elon Musk and ask the same
link |
question, he says it's not a problem. It's very difficult to do in the real world. This won't
link |
be a problem. He dismissed it as a problem for adversarial attacks on the Tesla. Of course,
link |
he happens to be involved with the company, so he has to say that. But let me linger and
link |
end a little longer. Where does your confidence that it's feasible come from? And what's your
link |
intuition? How people should be worried? How people should defend against it? How Tesla,
link |
how Waymo, how other autonomous vehicle companies should defend against
link |
sensory based attacks on whether on LiDAR or on vision or so on.
link |
And also even for LiDAR, actually, there has been research on that even like itself.
link |
No, no, no. But see, it's really important to pause. There's really nice demonstrations
link |
that it's possible to do, but there's so many pieces that it's kind of like... It's kind of in
link |
the lab. Now, it's in the physical world, meaning it's in the physical space, the attacks, but
link |
it's very... You have to control a lot of things to pull it off. It's like the difference between
link |
opening a safe when you have it and you have unlimited time and you can work on it versus
link |
like breaking into the crown, stealing the crown jewels or whatever.
link |
Right. I mean, so one way to look at this in terms of how real this attacks can be, one way to
link |
look at it is that actually you don't even need any sophisticated attacks. Already we've seen
link |
many real world examples, incidents, where showing that the vehicle was making the wrong
link |
decision. The wrong decision without attacks, right? Right. So that's one way to demonstrate.
link |
And this is also... So far, we've mainly talked about work in this adversarial setting,
link |
showing that today's learning system, they are so vulnerable to the adversarial setting.
link |
But at the same time, actually, we also know that even in natural settings, these learning systems,
link |
they don't generalize well. And hence, they can really misbehave under certain situations
link |
like what we have seen. And hence, I think using that as an example, it can show that these issues
link |
can be real. They can be real. But so there's two cases. One is something, it's like perturbations
link |
can make the system misbehave versus make the system do one specific thing that the attacker
link |
wants. As you said, the targeted attack. That seems to be very difficult,
link |
like an extra level of difficult step in the real world. But from the perspective of the passenger
link |
of the car, I don't think it matters either way, whether it's misbehavior or a targeted attack.
link |
Okay. And that's why I was also saying earlier, one defense is this multi model defense. And
link |
more of these consistent checks and so on. So in the future, I think also it's important that for
link |
these autonomous vehicles, they have lots of different sensors, and they should be combining
link |
all these sensory readings to arrive at the decision and the interpretation of the world and so on.
link |
And the more of these sensory inputs they use, and the better they combine these sensory inputs,
link |
the harder it is going to be attacked. And hence, I think that is a very important direction
link |
for us to move towards. So multi model, multi sensor across multiple cameras,
link |
but also in the case of car, radar, ultrasonic, sound even. So all of those. Right. Exactly.
link |
So another thing, another part of your work has been in the space of privacy. And that too can
link |
be seen as a kind of security vulnerability. And so thinking of data as a thing that should be
link |
protected and the vulnerabilities to data as vulnerability is essentially the thing that
link |
you want to protect is the privacy of that data. So what do you see as the main vulnerabilities
link |
in the privacy of data and how do we protect it? Right. So in security, we actually talk about
link |
essentially two, in this case, two different properties. One is integrity and one is confidentiality.
link |
So what we have been talking earlier is essentially the integrity of the integrity
link |
property of the learning system, how to make sure that the learning system is giving the
link |
right prediction, for example. And privacy essentially is on the other side is about
link |
confidentiality of the system is how attackers can, when the attackers compromise the confidentiality
link |
of the system, that's when the attackers steal sensitive information and right about individuals
link |
and so on. That's really clean. Those are those are great terms, integrity and confidentiality.
link |
Right. So how, what are the main vulnerabilities to privacy, we just say, and how do we protect
link |
against it? Like what are the main spaces and problems that you think about in the context of
link |
privacy? Right. So especially in the machine learning setting. So in this case, as we know that
link |
how the process goes is that we have the training data and then the machine learning system trains
link |
from this training data and then builds a model and then later on inputs are given to the model to
link |
influence time to try to get prediction and so on. So then in this case, the privacy concerns that we
link |
have is typically about privacy of the data in the training data because that's essentially the
link |
private information. So, and it's really important because oftentimes the training data can be very
link |
sensitive. It can be your financial data, it's your health data or like in our case, it's the
link |
sensors deployed in real world environment and so on and all this can be collecting very sensitive
link |
information and all the sensitive information gets fed into the learning system and trains
link |
and as we know, these neural networks, they can have really high capacity and they actually
link |
can remember a lot and hence just from the learning model in the end, actually attackers can potentially
link |
infer information about their original training data sets. So the thing you're trying to protect
link |
that is the confidentiality of the training data and so what are the methods for doing that? Would
link |
you say what are the different ways that can be done? And also we can talk about essentially
link |
essentially how the attacker may try to learn information from the right. So, and also there
link |
are different types of attacks. So in certain cases, again, like in white box attacks, we can say that
link |
the attacker actually gets to see the parameters of the model and then from that, the smart attacker
link |
potentially can try to figure out information about the training data set. They can try to figure
link |
out what type of data has been in the training data sets and sometimes they can tell like
link |
whether a person has been, a particular person's data point has been used in the training data sets
link |
as well. So white box meaning you have access to the parameters of say a neural network
link |
and so that you're saying that it's some, given that information is possible to some.
link |
So I can give you some examples and then another type of attack which is even easier to carry out
link |
is not a white box model, it's more of just a query model where the attacker only gets to
link |
query the machine learning model and then try to steal sensitive information in the original
link |
training data. So, right, so I can give you an example. In this case, training a language model.
link |
So in our work in collaboration with the researchers from Google, we actually studied the
link |
following question. So at high level, the question is, as we mentioned, the neural networks can have
link |
very high capacity and they could be remembering a lot from the training process. Then the question
link |
is, can attacker actually exploit this and try to actually extract sensitive information in the
link |
original training data set through just querying the learned model without even knowing the
link |
parameters of the model, like the details of the model or the architectures of the model and so on.
link |
So that's the question we set out to explore. And in one of the case studies, we showed the following.
link |
So we trained the language model over an email data set. It's called an enron email data set.
link |
And the enron email data sets naturally contains users social security numbers and critical numbers.
link |
So we trained the language model over the data sets. And then we showed that's an attacker
link |
by devising some new attacks, by just querying the language model. And without knowing the details
link |
of the model, the attacker actually can extract the original social security numbers and critical
link |
numbers that were in the original training. So get the most sensitive, personally identifiable
link |
information from the data set from just querying it. Right. Yeah. So that's an example showing
link |
that's why even as we train machine learning models, we have to be really careful with protecting
link |
users data privacy. So what are the mechanisms for protecting? Is there hopeful? So if there's
link |
been recent work on differential privacy, for example, that provides some hope, but can you
link |
describe some of the ideas? Right. So that's actually, right. So that's also our finding,
link |
is that by, actually, we show that in this particular case, we actually have a good defense.
link |
For the querying case, for the language model case. So instead of just training a vanilla
link |
language model, instead, if we train a differentially private language model, then we can still
link |
achieve similar utility. But at the same time, we can actually significantly enhance the privacy
link |
protection of the learned model. And our proposed attacks actually are no longer effective.
link |
And differential privacy is a mechanism of adding some noise by which you then have some guarantees
link |
on the inability to figure out the presence of a particular person in the data set.
link |
So right. So in this particular case, what the differential privacy mechanism does is that it
link |
actually adds perturbation in the training process. As we know, during the training process,
link |
we are learning the model, we are doing gradient updates, with updates and so on.
link |
And essentially, differential privacy, a differentially private
link |
machine learning algorithm in this case, will be adding noise and adding various perturbation
link |
during this training process. To some aspect of the training process.
link |
Right. So then the finally trained learning, the learned model is differentially private.
link |
And so it can enhance the privacy protection.
link |
So okay, so that's the attacks and the defense of privacy.
link |
You also talk about ownership of data. So this, this is a really interesting idea
link |
that we get to use many services online for seemingly for free by essentially sort of a lot
link |
of companies are funded through advertisement. And what that means is the advertisement works
link |
exceptionally well because the companies are able to access our personal data.
link |
So they know which advertisement to serve us to do target advertisements and so on.
link |
So can you maybe talk about this? You have some nice paintings of the future,
link |
philosophically speaking, future where people can have a little bit more control of their data by
link |
owning and maybe understanding the value of their data and being able to sort of
link |
monetize it in a more explicit way as opposed to the implicit way that it's currently done.
link |
Yeah, I think this is a fascinating topic and also a really complex topic.
link |
Right. I think there are these natural questions who should be owning the data.
link |
And so I can draw one analogy. So for example, for physical properties like your house and so on.
link |
So really, this notion of property rights is not just, you know,
link |
like it's not like from day one, we knew that there should be like this clear notion of ownership
link |
of properties and having enforcement for this. And so actually, people have shown that
link |
this establishment and enforcement of property rights has been a main driver for the economy
link |
earlier. And that actually really propelled the economic growth even in the earlier stage.
link |
So throughout the history of the development of the United States or actually just civilization,
link |
the idea of property rights that you can own property.
link |
Right. And then there's enforcement. There is institutional rights,
link |
that governmental like enforcement of this actually has been a key driver for economic growth.
link |
And there have been even research or proposal saying that for a lot of the developing countries,
link |
there, you know, essentially the challenging growth is not actually due to the lack of capital.
link |
It's more due to the lack of this notion of property rights and enforcement of property rights.
link |
Interesting. So that the presence of absence of both the concept of the property rights and
link |
their enforcement has a strong correlation to economic growth.
link |
And so you think that that same could be transferred to the idea of property ownership
link |
in the case of data ownership? I think it's, first of all, it's a good lesson for us to
link |
recognize that these rights and the recognition and enforcement of these type of rights is very,
link |
very important for economic growth. And then if we look at where we are now and where we are
link |
going in the future, so essentially more and more is actually moving into the digital world.
link |
And also more and more, I would say, even like information or asset of a person is more and more
link |
into the real world, the physical, the digital world as well. It's the data that the person
link |
has generated. Essentially, it's like in the past, what defines a person? You can say, right,
link |
like oftentimes, besides the innate capabilities, actually, it's the physical properties as the
link |
right that defines a person. But I think more and more people start to realize actually what
link |
defines a person is more important in the data that the person has generated or the data about
link |
the person. Like all the way from your political views, your music taste and financial information,
link |
a lot of these on your health. So more and more of the definition of the person is actually in
link |
the digital world. And currently, for the most part, that's owned. People don't talk about it,
link |
but kind of it's owned by internet companies. So it's not owned by individuals. Right. There's
link |
no clear notion of ownership of such data. And also, we talk about privacy and so on,
link |
but I think actually clearly identifying the ownership is the first step. Once you identify
link |
the ownership, then you can say who gets to define how the data should be used. So maybe
link |
some users are fine with, you know, internet companies serving them as using their data as
link |
well as if the data is used in a certain way that actually the user can sense with or allows. For
link |
example, you can see the recommendation system in some sense, we don't call it as, but recommendation
link |
system, similarly, it's trying to recommend you something. And users enjoy and can really benefit
link |
from good recommendation systems, either recommending your better music, movies, news,
link |
even research papers to read. But of course, then in these targeted ads, especially in certain cases
link |
where people can be manipulated by these targeted ads, they can have really bad, like severe
link |
consequences. So essentially, users want their data to be used to better serve them,
link |
and also maybe even get paid for or whatever, like in different settings. But the thing is that
link |
first of all, we need to really establish who needs to decide who can decide how the data should be
link |
used. And typically, the establishment and clarification of the ownership will help this,
link |
and it's an important first step. So if the user is the owner, then naturally the user gets to
link |
define how the data should be used. But if you even say that vitamin is used, actually,
link |
now the owner of this data, whoever is collecting the data is the owner of the data. Now, of course,
link |
they get to use the data however way they want. So to really address these complex issues, we need
link |
to go at the root cause. So it seems fairly clear that so first we really need to say
link |
who is the owner of the data, and then the owners can specify how they want their data to be utilized.
link |
So that's a fascinating, most people don't think about that. And I think that's a fascinating thing
link |
to think about and probably fight for it. I can only see in the economic growth argument,
link |
it's probably a really strong one. So that's a first time I'm kind of at least thinking about
link |
the positive aspect of that ownership being the long term growth of the economy, so good for
link |
everybody. But sort of one possible downside I could see, sort of to put on my grumpy old grandpa
link |
hat. And you know, it's really nice for Facebook and YouTube and Twitter to all be free. And if you
link |
give control to people with their data, do you think it's possible they would not want to hand
link |
it over quite easily? And so a lot of these companies that rely on mass handover of data and
link |
then therefore provide a mass seemingly free service would then completely, so the way the
link |
internet looks will completely change because of the ownership of data and will lose a lot of
link |
services value. Do you worry about that? So that's a very good question. I think
link |
that's not necessarily the case in the sense that, yes, users can have ownership of their data,
link |
they can maintain control of their data, but also then they get to decide how their data can be used.
link |
So that's why I mentioned earlier, like so in this case, if they feel that they enjoy the
link |
benefits of social networks and so on, and they're fine with having Facebook, having their data,
link |
but utilizing the data in a certain way that they agree, then they can still enjoy the free
link |
services. But for others, maybe they would prefer some kind of private vision. And in that case,
link |
maybe they can even opt in to say that I want to pay and to have, so for example, it's already
link |
fairly standard, like you pay for certain subscriptions so that you don't get to, you know,
link |
be shown ads, right. So then users essentially can have choices. And I think we just want to
link |
essentially bring out more about who gets to decide what to do with the data.
link |
I think it's an interesting idea because if you poll people now,
link |
you know, it seems like, I don't know, but subjectively, sort of anecdotally speaking,
link |
it seems like a lot of people don't trust Facebook. So that's at least a very popular
link |
thing to say that I don't trust Facebook, right. I wonder if you give people control of their data,
link |
as opposed to sort of signaling to everyone that they don't trust Facebook, I wonder how they would
link |
speak with the actual, like, would they be willing to pay $10 a month for Facebook, or would they
link |
hand over their data? It's, it'd be interesting to see what fraction of people would quietly hand
link |
over their data to Facebook to make it free. I don't have a good intuition about that. Like,
link |
how many people, do you have an intuition about how many people would use their data
link |
effectively on the market, on the market of the internet by sort of buying services with their
link |
data? Yeah, so that's a very good question. I think, so one thing I also want to mention is that
link |
this, right, so it seems that especially in press, and the conversation has been very much, like,
link |
two sides fighting against each other. On one hand, right, users can say that, right, they don't
link |
trust Facebook, they don't, or they delete Facebook. Yeah, exactly. Right, and then on the other
link |
hand, right, of course, right, the other side, they also feel, oh, they are providing a lot of
link |
services to users, and users are getting it all for free. So I think I actually, you know, I talk
link |
a lot to, like, different companies and also, like, basically on both sides. So one thing I hope,
link |
also, like, this is my hope for this year, also, is that we want to establish a more
link |
constructive dialogue that have, and to help people to understand that the problem is much
link |
more nuanced than just this two sides fighting. Because, naturally, there is a tension between
link |
the two sides, between utility and privacy. So if you want to get more utility, essentially,
link |
like the recommendation system example I gave earlier, if you want someone to give you good
link |
recommendation, essentially, whatever the system is, the system is going to need to know your data
link |
to give you a good recommendation. But also, of course, at the same time, we want to ensure
link |
that, however, that data is being handled, it's done in a privacy preserving way. So that, for
link |
example, the recommendation system doesn't just go around and sell your data and cause all the,
link |
you know, cause a lot of bad consequences and so on. So you want that dialogue to be a little
link |
bit more in the open, a little bit more nuanced, and maybe adding control to the data, ownership
link |
to the data will allow, as opposed to this happening in the background, allow it to bring it to the
link |
forefront and actually have dialogues in, like, more nuanced, real dialogues about how we trade
link |
our data for the services. That's the whole. Right, right. Yes, at high level. So essentially,
link |
also knowing that there are technical challenges in addressing the issue to, like, basically,
link |
you can't have, just like the example that I gave earlier, it's really difficult to balance the two
link |
between utility and privacy. And that's also a lot of things that I work on, my group works on,
link |
as well, is to actually develop these technologies that are needed to essentially help this balance
link |
better, essentially to help data to be utilized in a privacy preserving and responsible way.
link |
And so we essentially need people to understand the challenges and also at the same time to provide
link |
the technical abilities and also regulatory frameworks to help the two sides to be more
link |
in a win win situation instead of a fight. Yeah, the fighting, the fighting thing is,
link |
I think YouTube and Twitter and Facebook are providing an incredible service to the world.
link |
And they're all making mistakes, of course, but they're doing an incredible job
link |
that I think deserves to be applauded. And there's some degree of gratitude,
link |
like, it's a cool thing that that's created. And it shouldn't be monolithically fought against,
link |
like, Facebook is evil or so on. Yeah, I might make mistakes, but I think it's an incredible
link |
service. I think it's world changing. I mean, I've, I think Facebook's done a lot of incredible,
link |
incredible things by bringing, for example, identity, you're like,
link |
allowing people to be themselves like their real selves in the digital space by using their real
link |
name and their real picture. That step was like the first step from the real world to the digital
link |
world. That was a huge step that perhaps will define the 21st century in us creating a digital
link |
identity. And there's a lot of interesting possibilities there that are positive. Of course,
link |
some things are negative and having a good dialogue about that is great. And I'm great
link |
that people like you are at the center of that dialogue. That's awesome. Right. I think also,
link |
I also can understand, I think actually in the past, especially in the past couple years,
link |
this rising awareness has been helpful. Like, users are also more and more recognizing
link |
that privacy is important to them. They should, maybe, right, they should be owners of their data.
link |
I think this definitely is very helpful. And I think also this type of voice also, and together
link |
with the regulatory framework and so on, also help the companies to essentially put these
link |
type of issues at a higher priority. And knowing that, right, also, it is their responsibility
link |
to ensure that users are well protected. And so I think definitely the rising voice
link |
is super helpful. And I think that actually really has brought the issue of data privacy
link |
and even this consideration of data ownership to the forefront to really much wider community.
link |
And I think more of this voice is needed. But I think it's just that we want to have
link |
a more constructive dialogue to bring the both sides together to figure out a constructive solution.
link |
So another interesting space where security is really important is in the space of
link |
any kinds of transactions, but it could be also digital currency. So can you maybe talk
link |
a little bit about blockchain? Can you tell me what is a blockchain?
link |
I think the blockchain word itself is actually very overloaded.
link |
In general, it's like AI, right? Yes. So in general, when we talk about blockchain,
link |
we refer to this distributed ledger in a decentralized fashion. So essentially,
link |
you have a community of nodes that come together. And even though each one may not be trusted,
link |
and as long as certain thresholds of the set of nodes behave properly, then the system can
link |
essentially achieve certain properties. For example, in the distributed ledger setting,
link |
you can maintain an immutable log and you can ensure that, for example, the transactions
link |
actually are agreed upon and then it's immutable and so on. So first of all, what's a ledger?
link |
So it's a... It's like a database. It's like a data entry. And so distributed ledger is
link |
something that's maintained across or is synchronized across multiple sources, multiple nodes.
link |
Multiple nodes, yes. And so where is this idea? How do you keep... So it's important
link |
to keep a ledger a database to keep that... To make sure... So what are the kinds of security
link |
vulnerabilities that you're trying to protect against in the context of a distributed ledger?
link |
So in this case, for example, you don't want some malicious nodes to be able to change the
link |
transaction logs. And in certain cases, it's called double spending. You can also cause
link |
different views in different parts of the network and so on.
link |
So the ledger has to represent, if you're capturing financial transactions,
link |
has to represent the exact timing and the exact occurrence and no duplicates, all that kind of
link |
stuff. It has to represent what actually happened. Okay, so what are your thoughts
link |
on the security and privacy of digital currency? I can't tell you how many people
link |
write to me to interview various people in the digital currency space. There seems to be a lot
link |
of excitement there. And it seems to be... Some of it, to me, from an outsider's perspective,
link |
seems like dark magic. I don't know how secure... I think the foundation from my perspective of
link |
digital currencies, that is, you can't trust anyone. So you have to create a really secure system.
link |
So can you maybe speak about what your thoughts in general about digital currency is and how you
link |
can possibly create financial transactions and financial stores of money in the digital space?
link |
So you asked about security and privacy. So again, as I mentioned earlier,
link |
in security, we actually talk about two main properties, the integrity and confidentiality.
link |
And so there's another one for availability. You want the system to be available. But here,
link |
for the question asked, let's just focus on integrity and confidentiality. So for integrity
link |
of this distributed ledger, essentially, as we discussed, we want to ensure that the different
link |
nodes... So they have this consistent view, usually it's down through what we call a consensus protocol
link |
that they establish this shared view on this ledger that you cannot go back and change,
link |
it's immutable, and so on. So in this case, then the security often refers to this integrity
link |
property. And essentially, you're asking the question, how much work, how can you attack the
link |
system so that the attacker can change the log, for example.
link |
Right. How hard is it to make them attack like that?
link |
Right. And then that very much depends on the consensus mechanism, how the system is built,
link |
and all that. So there are different ways to build these decentralized systems.
link |
People may have heard about the terms called proof of work, proof of stake, these different
link |
mechanisms. And it really depends on how the system has been built and also how much
link |
resources, how much work has gone into the network to actually say how secure it is.
link |
So for example, if you talk about Bitcoin's proof of work system, so much electricity has been
link |
burned. So there's differences in the different mechanisms and the implementations of a distributed
link |
ledger used for digital currency. So there's Bitcoin, whatever, there's so many of them,
link |
and there's underlying different mechanisms. And there's arguments, I suppose, about which is more
link |
effective, which is more secure, which is more. And what is needed? What amount of resources
link |
needed to be able to attack the system? Like for example, what percentage of the nodes do you
link |
need to control or compromise in order to change the log?
link |
And do you have a sense of those are things that can be shown theoretically
link |
through the design of the mechanisms or does it have to be shown empirically by having a large
link |
number of users using the currency?
link |
I see. So in general, for each consensus mechanism, you can actually show theoretically what is needed
link |
to be able to attack the system. Of course, there can be different types of attacks as we
link |
discussed at the beginning, so that it's difficult to give a complete estimate really how much it's
link |
needed to compromise the system. But in general, there are ways to say what percentage of the
link |
nodes you need to compromise and so on.
link |
So we talked about integrity on the security side. And then you also mentioned the privacy or the
link |
confidentiality side. Does it have some of the same problems and therefore some of the
link |
same solutions that you talked about on the machine learning side with differential privacy and so on?
link |
Yeah. So actually, in general, on the public ledger in these public decentralized systems,
link |
actually nothing is private. So all the transactions posted on the ledger anybody can see.
link |
So in that sense, there is no confidentiality. So usually what you can do is then there are the
link |
mechanisms that you can build in to enable confidentiality or privacy of the transactions
link |
and the data and so on. That's also some of the work that's both my group and also my startup
link |
as well. What's the name of the startup? Oasis Labs. Oasis Labs. And so the confidentiality
link |
aspect there is even though the transactions are public, you want to keep some aspect confidential
link |
of the identity of the people involved in the transactions. So what is their hope to keep
link |
confidential in this context? So in this case, for example, you want to enable like
link |
confidential transactions. So there are different essentially types of data that you want to keep
link |
private or confidential. And you can utilize different technologies, including zero knowledge
link |
proofs and also secure computing and techniques to hide who is making the transactions to whom
link |
and the transaction amount. And in our case, also we can enable like confidential smart contracts
link |
so that you don't know the data and the execution of the smart contract and so on.
link |
And we actually are combining these different technologies and to going back to the earlier
link |
discussion we had enabling like ownership of data and privacy of data and so on. So at Oasis
link |
Labs, we're actually building what we call a platform for a responsible data economy
link |
to actually combine these different technologies together to enable secure and privacy preserving
link |
computation and also using the ledger to help provide immutable log of users ownership to their
link |
data and the policies they want the data to adhere to, the usage of the data to adhere to
link |
and also how the data has been utilized. So all this together can build a distributed secure
link |
computing fabric that helps to enable a more responsible data economy. There's a lot of
link |
things together. Yeah, wow, that was eloquent. Okay, you're involved in so much amazing work
link |
that we'll never be able to get to, but I have to ask at least briefly about program synthesis,
link |
which at least in a philosophical sense captures much of the dreams of what's possible in computer
link |
science and the artificial intelligence. First, let me ask what is program synthesis
link |
and can neural networks be used to learn programs from data? So can this be learned,
link |
some aspect of the synthesis can it be learned? So program synthesis is about teaching computers
link |
to write code to program. And I think that's one of our ultimate dreams or goals.
link |
I think Andreessen talked about software eating the world. So I say once we teach computers to
link |
write software to write programs, then I guess computers will be eating the world by
link |
transitivity. Yeah, exactly. And also for me, actually, when I shifted from security to more AI
link |
machine learning, program synthesis is program synthesis and adversarial machine learning.
link |
These are the two fields that I particularly focus on. Like program synthesis is one of the
link |
first questions that I actually started. Just as a question, I guess from the security side,
link |
there's a, you know, you're looking for holes in programs. So at least see small connection. But
link |
why, where was your interest for program synthesis? Because it's such a fascinating, such a big, such
link |
a hard problem in the general case. Why program synthesis? So the reason for that is actually
link |
when I shifted my focus from security into AI machine learning, actually one of my main
link |
motivation at the time is that even though I have been doing a lot of work in security and
link |
privacy, but I have always been fascinated about building intelligent machines. And that was really
link |
my main motivation to spend more time in AI machine learning is that I really want to figure out how
link |
we can build intelligent machines. And to help us towards that goal, program synthesis is really
link |
one of, I would say, the best domain to work on. I actually call it like a program synthesis is
link |
like the perfect playground for building intelligent machines and for artificial
link |
generating intelligence. Well, it's also in that sense, not just a playground, I guess it's the
link |
ultimate test of intelligence because I think if you can generate neural networks can learn
link |
good functions and they can help you out in classification tasks, but to be able to write
link |
programs, that's the epitome from the machine side. That's the same as passing the Turing test
link |
in natural language, but with programs, it's able to express complicated ideas, to reason through
link |
ideas, and yeah, and boil them down to algorithms. Yes, exactly, incredible. So can this be learned?
link |
How far are we? Is there hope? What are the open challenges? Yeah, very good questions. We are
link |
still at an early stage, but already I think we have seen a lot of progress. I mean, definitely we
link |
have, you know, existence proof, just like humans can write programs, so there's no reason why
link |
computers cannot write programs. So I think that's definitely an achievable goal, it's just how long
link |
it takes. And then, and even today, we actually have, you know, the program synthesis community,
link |
especially the program synthesis via learning, how we call it, neural program synthesis community,
link |
is still very small, but the community has been growing and we have seen a lot of progress.
link |
And in limited domains, I think actually program synthesis is ripe for real world
link |
applications. So actually it was quite amazing, I was at, I was giving a talk,
link |
so here is a rework conference. Yeah, rework deep learning summary. I actually, so I give another
link |
talk at the previous rework conference in deep reinforcement learning. And then I actually
link |
met someone from a startup, the CEO of the startup, and when he saw my name, he recognized it, and
link |
he actually said, one of our papers actually had, they have put, had actually become a key
link |
product in their startup. And that was program synthesis in that particular case was natural
link |
language translation, translating natural language description into SQL queries.
link |
Oh, wow, that, that direction. Okay. Right. So, right. So, yeah, so in program synthesis,
link |
in limited domains, in well specified domains, actually already we can see
link |
really great progress and applicability in the real world. So domains like,
link |
I mean, as an example, you said natural language being able to express something through just
link |
normal language and it converts it into a database SQL SQL query. Right. And that's how,
link |
how solves the problem is that, because that seems like a really hard problem.
link |
Again, in limited domains, actually it can work pretty well. And now this is also a very active
link |
domain of research. At the time, I think when he saw our paper at the time, we were the state of the
link |
arts on that task. And since then, actually now there has been more work and with even more
link |
like sophisticated data sets. And so, but I, I think I wouldn't be surprised that more of this
link |
type of technology really gets into the real world. That's exciting. In the near term.
link |
Being able to learn in the space of programs is, is super exciting. I still,
link |
I'm still skeptical because I think it's a really hard problem, but I'd love to see progress.
link |
And also, I think in terms of the, you asked about open challenges, I think the domain is
link |
full of challenges. And in particular, also we want to see how we should measure the progress
link |
in the space. And I would say mainly three main, I would say metrics. So one is the complexity of
link |
the program that we can synthesize. And that will actually have clear measures and just look at,
link |
you know, the past publications. And even like, for example, I was at the recent
link |
New Europe's conference now, there's actually a very sizable like session dedicated to program
link |
synthesis, which is. Oh, even neural programs. So this is. Right. Which is great. And, and we
link |
continue to see the increase. What does sizable mean? I like, I like the word sizable. It's,
link |
it's five people. It's still a small community, but this is growing. And they will all win touring
link |
awards one day. I like it. Right. So, so we can clearly see increase in the complexity of the
link |
programs that these just elaborate synthesize side to is it the complexity of the actual text of
link |
the program or the running time complexity, which complexity over how the complexity of the task
link |
to be synthesized and the complexity of the actual synthesizer programs. So it's right. So the lines
link |
of code even, for example, okay, I got you. But it's not the theoretical. No, no, no, no, the running
link |
time of the algorithm. Okay, got it. Got it. And you can see the complexity decreasing already.
link |
Oh, no, meaning we want to be able to synthesize more and more complex programs,
link |
bigger and bigger programs. So we want to see that we want to increase the complexity. I have to think
link |
through because I thought of complexity is you want to be able to accomplish the same task
link |
with a simpler simpler program. No, we are not doing that. Okay. It's more, it's more about
link |
how complex a task we can synthesize programs for. Got it. Being able to synthesize programs,
link |
learn them for more and more difficult. Right. So for example, initially, our first work in program
link |
synthesis was to translate natural language distribution into really simple programs called
link |
if TTT, if this then that. So given a trigger condition, what is the action you should take.
link |
So that program is super simple. You just identify the trigger conditions and the action.
link |
Yep. And then later on with the SQL queries, it gets more complex. And then also, we started
link |
to synthesize programs with loops and. Oh, no. And if you can synthesize recursion, it's all over.
link |
Right. Actually, one of our works actually is learning recursive programs. But anyway,
link |
anyway, so that's the one is the complexity and the other one is generalization. Like when we
link |
train our own learn a program synthesizer in this case, a neural programs to synthesize programs,
link |
then you wanted to generalize. For any for a large number of inputs. Right. So to be able to
link |
right generalize to previously unseen inputs. Got it. And so, right. So some of the work we did earlier
link |
learning recursive neural programs actually show that recursion actually is important
link |
to learn. And if you have recursion, then for certain set of tasks, we can actually show that
link |
you can actually have perfect generalization. So that's one of the best people were words that
link |
I clear earlier. So that's one example of we want to learn these neural programs that can
link |
generalize better. But that works for certain tasks, certain domains. And there's question how we can
link |
essentially develop more techniques that can have generalization for wider set of domains,
link |
and so on. So that's another area. And then the third challenge I think will, it's not just for
link |
program synthesis is also cutting across other fields in machine learning and also including
link |
like deep reinforcement learning in particular is that this adaptation is that we want to be able
link |
to learn from the past and tasks and training and so on to be able to solve new tasks. So for example,
link |
in program synthesis today, we still are working in the setting where given a particular task,
link |
given a particular task, we train the right the model and to solve this particular task.
link |
But that's not how humans work. The whole point is we train a human and you can then program to
link |
solve new tasks. Exactly. And just like in deep reinforcement learning, we don't want to just
link |
train agent to play a particular game, either it's Atari or it's Go or whatever. We want to train
link |
these agents that can essentially extract knowledge from the past learning experience
link |
to be able to adapt to new tasks and solve new tasks. And I think this is particularly important
link |
for program synthesis. Yeah, that's the whole point. That's the whole dream of programs. This is
link |
your learning a tool that can solve new problems. Right, exactly. And I think that's a particular
link |
domain that as a community, we need to put more emphasis on and I hope that we can make more
link |
progress there as well. Awesome. There's a lot more to talk about. Let me ask that you also had a very
link |
interesting and we talked about rich representations. You had a rich life journey. You did your
link |
bachelors in China and your masters and PhD in the United States, CMU and Berkeley.
link |
Are there interesting differences? I told you I'm Russian. I think there's a lot of
link |
interesting difference between Russia and the United States. Are there in your eyes
link |
interesting differences between the two cultures from the romantic notion of the spirit of the
link |
people to the more practical notion of how research is conducted that you find interesting
link |
or useful in your own work of having experience both? That's a good question. I think, so I
link |
I studied in China for my undergraduate years and that was more than 20 years ago. So it's
link |
been a long time. Is there echoes of that time in you? Actually, it's interesting. I think even
link |
more so maybe something that's even be more different from my experience than a lot of computer
link |
science researchers and practitioners. So for my undergrad, I actually studied physics.
link |
Nice. Very nice. And then I switched to computer science in graduate school.
link |
What happened? Is there another possible universe where you could have
link |
become a theoretical physicist at Caltech or something like that?
link |
That's very possible. Some of my undergrad classmates, then they later on started physics,
link |
got their PhD in physics from these schools from, yeah, from tough physics programs.
link |
So you switched to, I mean, from that experience of doing physics in your bachelors,
link |
what made you decide to switch to computer science and computer science at arguably the best
link |
university, one of the best universities in the world for computer science with Carnegie Mellon,
link |
especially for grad school and so on. So what, second only to MIT, just kidding. Okay.
link |
I had to throw that in there. No, what was the choice like and what was the
link |
move to the United States like? What was that whole transition? And if you remember,
link |
if there's still echoes of some of the spirit of the people of China in you in New York?
link |
Right. That's like three questions. I'm sorry.
link |
No, that's okay. So yes, I guess, okay, the first transition from physics to computer science.
link |
So when I first came to the United States, I was actually in the physics PhD program at Cornell.
link |
I was there for one year and then I switched to computer science and then I was in the PhD
link |
program at Carnegie Mellon. So, okay, so the reasons for switching. So one thing,
link |
so that's why I also mentioned that about this difference in backgrounds about having studied
link |
physics first in my undergrad. I actually really, I really did enjoy my undergrad time and education
link |
in physics. I think that actually really helped me in my future work in computer science.
link |
Actually, even for machine learning, a lot of machine learning stuff,
link |
the core machine methods, many of them actually came from physics.
link |
For honest, most of everything came from physics.
link |
But anyway, so when I started physics, I was, I think I was really attracted to physics.
link |
It was, it's really beautiful. And I actually, physics is the language of nature.
link |
And I actually clearly remember like one moment in my undergrad, like I did my undergrad in
link |
Tsinghua and I used to study in the library. And I clearly remember like one day I was sitting
link |
in the library and I, and I was like writing on my notes and so on. And I got so excited
link |
that I realized that really just from a few simple axioms, a few simple laws, I can derive
link |
so much. It's almost like I can derive the rest of the world.
link |
Yeah, the rest of the universe.
link |
Yes. Yes. So that was like amazing.
link |
Do you think you, have you ever seen or do you think you can rediscover that kind of power and
link |
beauty in computer science in the world that you use?
link |
That's very interesting. So that gets to, you know, the transition from physics to computer
link |
science. It's quite different for physics in, in grad school actually things changed.
link |
So one is, I started to realize that when I started doing research in physics,
link |
at the time I was doing theoretical physics. And a lot of it, you still have the beauty
link |
but it's very different. So I had to actually do a lot of the simulation. So essentially I was
link |
actually writing, in some, in some cases writing fortune code.
link |
Good old fortune, yeah.
link |
To actually write, do like, do simulations and so on. That was not, not exactly what I enjoyed doing.
link |
And also at the time from talking with senior, you know, students in the program,
link |
I realized many of the students actually were going off to like Wall Street and so on.
link |
So, and I've always been interested in computer science and actually essentially taught myself
link |
the C programming, like, program, right, and so on. Of which when? In college. In college
link |
somewhere? In the summer. For fun. Physics major, learning to do C programming, beautiful.
link |
Actually it's interesting, you know, in physics at the time, I think now the program probably has
link |
changed. But at the time, really the only class we had in, in, related to computer science education
link |
was introduction to, I forgot, to computer science or computing and fortune 77.
link |
There's a lot of people that still use Fortran. I'm actually, if you're a programmer out there,
link |
I'm looking for an expert to talk to about Fortran. They seem to, there's not many,
link |
but there's still a lot of people that still use Fortran and still a lot of people use Cobalt.
link |
But anyway, so, so then, then I realized, instead of just doing programming for doing simulations
link |
and so on, that I may as well just change to computer science. And also one thing I really
link |
liked, and that's a key difference between the two is in computer science is so much easier to
link |
realize your ideas. If you have an idea, you write it up, you code it up, and then you can see it's
link |
actually running and you can, you can see it. You can bring it to life quickly. Bring it to life.
link |
Whereas in physics, if you have a good theory, you, you, you have to wait for the experimentalist
link |
to do the experiments and to confirm the theory and things just take so much longer. And, and
link |
also the reason I, in physics, I decided to do theoretical physics was because I had my experience
link |
with experimental physics. First, you have to fix the equipment. You spend most of your time fixing
link |
the equipment first. So, super expensive equipment. So there's a lot of, yeah, you have to collaborate
link |
with a lot of people. It takes a long time. It just takes really much longer. Yeah, it's messy.
link |
So I decided to switch to computer science. And one thing I think maybe people have realized is that
link |
for people who study physics, actually it's very easy for physicists to change, to do something
link |
else. I think physics provides a really good training. And yeah, so actually it was very easy
link |
to switch to computer science. But one thing going back to your earlier question. So one thing I
link |
actually did realize. So there is a big difference between computer science and physics, where physics
link |
you can derive the whole universe from just a few simple laws. And computer science, given that
link |
a lot of it is defined by humans, the systems that define by humans, and it's artificial.
link |
Essentially, you create a lot of these artifacts and so on. It's not quite the same. You don't
link |
derive the computer systems with just a few simple laws. You actually have to see there is historical
link |
reasons why a system is built and designed one way versus the other. There's a lot more complexity,
link |
less elegant simplicity of E equals MC squared that kind of reduces everything down to those
link |
beautiful fundamental equations. But what about the move from China to the United States? Is there
link |
anything that still stays in you that contributes to your work, the fact that you grew up in another
link |
culture? So yes, I think especially back then it's very different from now. So now actually
link |
I see these students coming from China and even undergraduates actually speak fluent English. It
link |
was just amazing. And they have already understood so much of the culture in the U.S. and so on.
link |
It was to you, it was all foreign? It was a very different time. At the time, actually
link |
we didn't even have easy access to email, not to mention about the web. I remember I had to
link |
go to specific privileged server rooms to use email. At the time, we had much less knowledge
link |
about the Western world. And actually at the time, I didn't know actually in the U.S. West Coast
link |
whether it's much better than the East Coast. Things like that actually. It's very interesting.
link |
But now it's so different. At the time, I would say there's also a bigger cultural difference
link |
because there's so much less opportunity for shared information. So it's such a different
link |
time and world. So let me ask maybe a sensitive question. I'm not sure, but I think you and I
link |
are in similar positions as I've been here for already 20 years as well. And looking at Russia
link |
from my perspective and you looking at China, in some ways it's a very distant place because it's
link |
changed a lot, but in some ways you still have echoes, you still have knowledge of that place.
link |
The question is, China is doing a lot of incredible work in AI. Do you see, please tell me there's
link |
an optimistic picture you see where the United States and China can collaborate and sort of
link |
grow together in the development of AI towards, there's different values in terms of the role
link |
of government and so on, of ethical, transparent, secure systems. We see it differently in the
link |
United States a little bit than China, but we're still trying to work it out. Do you see the two
link |
countries being able to successfully collaborate and work in a healthy way without sort of fighting
link |
and making it an AI arms race kind of situation? Yeah, I believe so. I think science has no border
link |
and the advancement of the technology helps everyone, helps the whole world. And so I certainly
link |
hope that the two countries will collaborate and I certainly believe so. Do you have any reason
link |
to believe so except being an optimist? So first again, like I said, science has no borders and
link |
especially in... Science doesn't know borders. Right. And you believe that well, you know,
link |
in the form of sort of union during the Cold War. So that's the other point I was going to mention
link |
is that especially in academic research, everything is public. Like we write papers,
link |
we open source codes and all this is in the public domain. It doesn't matter whether the person is
link |
in the US, in China or some other parts of the world. They can go on archive and look at the
link |
latest research and results. So that openness gives you hope? Yes. Me too. And that's also how
link |
as a world we make progress the best. So I apologize for the romanticized question, but
link |
looking back, what would you say was the most transformative moment in your life that maybe
link |
made you fall in love with computer science? You said physics. You remember there was a moment
link |
where you thought you could derive the entirety of the universe. Was there a moment that you
link |
really fell in love with the work you do now from security to machine learning to program synthesis?
link |
So maybe, as I mentioned, actually in college, I, one summer I just taught myself programming C.
link |
Yes. You just read a book. Don't tell me you fell in love with computer science by programming
link |
in C. Remember I mentioned one of the draws for me to computer science is how easy it is
link |
to realize your ideas. So once I, you know, read a book, start, like tell myself how to
link |
program in C. What did I do? I programmed two games. One is just simple, like it's a go game,
link |
like it's a board, you can move the stones and so on. And the other one actually programmed the game.
link |
That's like a 3D Tetris. It was, it turned out to be a super hard game to play. Because
link |
instead of just the standard 2D Tetris, it's actually a 3D thing. But I realized, wow,
link |
you know, I just had these ideas to try it out and then you can just do it. And so that's when I
link |
realized, wow, this is amazing. Yeah, you can create yourself. Yes, yes, exactly. From nothing
link |
to something that's actually out in the real world. So let me ask, let me ask a silly question,
link |
or maybe the ultimate question. What is to you the meaning of life? What, what gives your life
link |
meaning, purpose, fulfillment, happiness, joy? Okay, these are two different questions.
link |
Very different. Yeah. It's usually that you ask this question. Maybe this question is
link |
probably the question that has followed me and followed my life the most.
link |
Have you discovered anything, any satisfactory answer for yourself?
link |
Is there something, is there something you've arrived at? You know, there's a moment,
link |
I've talked to a few people who have faced, for example, a cancer diagnosis or face their own
link |
mortality. And that seems to change their view of them. It seems to be a catalyst for them removing
link |
most of the crap of seeing that most of what they've been doing is not that important and really
link |
reducing it into saying like, here's actually the few things that really give me, give meaning.
link |
Mortality is a really powerful catalyst for that. It seems like facing mortality,
link |
whether it's your parents dying or somebody close to you dying or facing your own death
link |
for what a reason or cancer and so on. Right. So yeah, so in my own case, I didn't need to face
link |
mortality too. To try to, you know, to ask that question. Yes. And I think there are a couple
link |
things. So one is like, who should be defining the meaning of your life? Right. Is there some kind
link |
of even greater things than you who should define the meaning of your life? So for example, when
link |
people say that the searching, the meaning for your life is, is there some, is there some outside
link |
voice or is there something, you know, outside of you who actually tells you, you know, so people
link |
talk about, oh, you know, this is what you have been born to do. Right. Right. Like, this is your
link |
destiny. So who, right. So that's one question. Like, who gets to define the meaning of your life?
link |
Should, should you be finding some other thing, some other factor to define this for you? Or
link |
is something actually, it's just entirely what you define yourself and it can be very arbitrary.
link |
Yeah. So in an inner voice or an outer voice, whether it's, it could be spiritual, religious,
link |
too, with God or some other components of the environment outside of you, or just your own
link |
voice, do you have an answer there? So, okay, so for that, I have an answer. Yeah. And through,
link |
you know, the long period of time of thinking and searching, even searching through outside,
link |
right, you know, voices or factors outside of me. Yeah. So that I have, and so I've come to
link |
the conclusion and realization that it's you yourself that defines the meaning of life.
link |
Yeah. That's a big burden though, isn't it? Or a guess. Yes and no. Right. So then you have the
link |
freedom to define it. Yes. And, and another question is like, what does it really mean by
link |
the meaning of life? Right. And also, whether the question even makes sense.
link |
Absolutely. And you said it somehow distinct from happiness. So meaning is something much deeper
link |
than just any kind of emotional and any kind of contentment or joy or whatever. It might be much
link |
deeper. And then you have to ask, what is deeper than that? What is, what is there at all? And
link |
then the question starts being silly. Right. And also you can say it's deeper, but you can also
link |
say it's a shallower depending on how people want to define the meaning of their life. So for example,
link |
most people don't even think about this question. Then the meaning of life to them doesn't really
link |
measure that much. And also whether knowing the meaning of life and whether it actually helps
link |
your life to be better or whether it helps your life to be happier. These actually are open questions.
link |
It's not. Of course. Most questions are open. I tend to think that just asking the question,
link |
as you mentioned, as you've done for a long time is the only, that there is no answer.
link |
And asking the question is a really good exercise. I mean, I have this, for me personally, I've had a
link |
kind of feeling that creation is, like for me, has been very fulfilling. And it seems like my
link |
meaning has been to create. And I'm not sure what that is. I don't have, I'm single out of kids.
link |
I'd love to have kids, but I also, sounds creepy, but I also see, sort of, you said,
link |
see programs. I see programs as little creations. I see robots as little creations.
link |
I think those bring, and then ideas, theorems, and our creations. And those somehow intrinsically,
link |
like you said, bring me joy. And I think they do to a lot of, at least scientists, but I think
link |
they do to a lot of people. So that, to me, if I had to force the answer to that, I would say
link |
creating new things yourself. For you. For me. For me. For me. I don't know. But like you said,
link |
it keeps changing. Is there some answer that? And some people, they can, I think they may say,
link |
it's experience, right? Like their meaning of life. They just want to experience to the riches
link |
and fullest they can. And a lot of people do take that path. Yes. Seeing life is actually a collection
link |
of moments and then trying to make the richest possible sets, fill those moments with the richest
link |
possible experiences. Yeah. Right. And for me, I think certainly we do share a lot of similarity
link |
here. So creation is also really important for me, even from, you know, the things I've already
link |
talked about, even like, you know, writing papers and these are our creations as well.
link |
And I have not quite thought whether that is really the meaning of my life. Like, in a sense,
link |
also that maybe like, what kind of things should you create? So there are so many different things
link |
that you could create. And also you can say, another view is maybe growth is, it's related
link |
but different from experience. Growth is also maybe a type of meaning of life. It's just,
link |
you try to grow every day, try to be a better self every day. And also ultimately, we are here,
link |
it's part of the overall evolution, the, right, the world is evolving. And it's funny,
link |
isn't it funny that the growth seems to be the more important thing than the thing you're growing
link |
towards. It's like, it's not the goal, it's the journey to it. Sort of, it's almost, it's almost
link |
when you submit a paper, there's a sort of depressing element to it, not to submit a paper,
link |
but when that whole project is over, I mean, there's a gratitude, there's a celebration and so on,
link |
but you're usually immediately looking for the next thing or the next step, right? It's not,
link |
it's not that satisfied, the end of it is not the satisfaction, it's the
link |
the hardship, the challenge you have to overcome, the growth through the process.
link |
It's somehow, probably deeply within us, the same thing that drives the evolutionary process
link |
is somehow within us, with everything, the way, the way we see the world, since you're
link |
thinking about these, so you're still in search of an answer. I mean, yes and no, in the sense that
link |
I think for people who really dedicate time to search for the answer, to ask the question,
link |
what is the meaning of life? It does not necessarily bring you happiness.
link |
Yeah, it's a question, we can say, right, like whether it's a well defined question and
link |
but on the other hand, given that you get to answer yourself, you can define it yourself,
link |
then sure, I can just give it an answer and in that sense, yes, it can help.
link |
Like we discussed, if you say, oh, then my meaning of life is to create or to grow,
link |
then yes, then I think it can help, but how do you know that that is really the meaning of life
link |
or the meaning of your life? It's like there's no way for you to really answer the question.
link |
Sure, but something about that certainty is liberating, so it might be an illusion,
link |
you might not really know, you might be just convincing yourself falsely, but being sure
link |
that that's the meaning, there's something liberating in that, there's something freeing
link |
and knowing this is your purpose, so you can fully give yourself to that.
link |
You know, for a long time, I thought like, isn't it all relative? Like why,
link |
what's, how do we even know what's good and what's evil? Like isn't everything just relative?
link |
Like how do we know, the question of meaning is ultimately the question of why do anything?
link |
Why is anything good or bad? Why is anything so on?
link |
Exactly. But the moment, then you start to, I think just like you said, I think it's a really
link |
useful question to ask, but if you ask it for too long and too aggressively.
link |
I mean, not be so productive. They have not be productive and not just for traditionally,
link |
societally defined success, but also for happiness. It seems like asking the question
link |
about the meaning of life is like a trap. We're destined to be asking, we're destined to look
link |
up to the stars and ask these big, why questions we'll never be able to answer,
link |
but we shouldn't get lost in them. I think that's probably the, that's at least the lesson I picked
link |
up so far on that topic. Oh, let me just add one more thing. So it's interesting. So actually,
link |
so sometimes, yes, it can help you to focus. So when I, when I shifted my focus more from
link |
security to AI and machine learning, at the time, the, actually one of the main reasons that I,
link |
I did that was because at the time, I thought my meaning, the meaning of my life and the purpose
link |
of my life is to build intelligent machines. And that's, and then your inner voice said that this
link |
is the right, this is the right journey to take to build intelligent machines. And that you actually
link |
fully realized you took a really legitimate big step to become one of the world class researchers
link |
to actually make it, to actually go down that journey. Yeah, that's profound. That's profound.
link |
I don't think there's a better way to end a conversation than talking for, for a while about
link |
the meaning of life. Don is a huge honor to talk to you. Thank you so much for talking today.
link |
Thank you. Thank you. Thanks for listening to this conversation with Don Song and thank you
link |
to our presenting sponsor, Cash App. Please consider supporting the podcast by downloading
link |
Cash App and using code Lex Podcast. If you enjoy this podcast, subscribe on YouTube,
link |
review it with five stars on Apple podcast, support on Patreon or simply connect with me
link |
on Twitter at Lex Freedman. And now let me leave you with some words about hacking from the great
link |
Steve Wozniak. A lot of hacking is playing with other people, you know, getting them to do strange
link |
things. Thank you for listening and hope to see you next time.