back to indexVladimir Vapnik: Statistical Learning | Lex Fridman Podcast #5
link |
The following is a conversation with Vladimir Vapnik.
link |
He's the co inventor of support vector machines,
link |
support vector clustering, VC theory,
link |
and many foundational ideas in statistical learning.
link |
He was born in the Soviet Union
link |
and worked at the Institute of Control Sciences in Moscow.
link |
Then in the United States, he worked at AT&T,
link |
NEC Labs, Facebook Research,
link |
and now is a professor at Columbia University.
link |
His work has been cited over 170,000 times.
link |
He has some very interesting ideas
link |
about artificial intelligence and the nature of learning,
link |
especially on the limits of our current approaches
link |
and the open problems in the field.
link |
This conversation is part of MIT course
link |
on artificial general intelligence
link |
and the artificial intelligence podcast.
link |
If you enjoy it, please subscribe on YouTube
link |
or rate it on iTunes or your podcast provider of choice,
link |
or simply connect with me on Twitter
link |
or other social networks at Lex Friedman spelled F R I D.
link |
And now here's my conversation with Vladimir Vapnik.
link |
Einstein famously said that God doesn't play dice.
link |
You have studied the world through the eyes of statistics.
link |
So let me ask you in terms of the nature of reality,
link |
fundamental nature of reality, does God play dice?
link |
We don't know some factors.
link |
And because we don't know some factors,
link |
which could be important,
link |
it looks like God plays dice.
link |
But we should describe it.
link |
In philosophy, they distinguish between two positions,
link |
positions of instrumentalism,
link |
where you're creating theory for prediction
link |
and position of realism,
link |
where you're trying to understand what God did.
link |
Can you describe instrumentalism and realism a little bit?
link |
For example, if you have some mechanical laws,
link |
Is it law which is true always and everywhere?
link |
Or it is law which allow you to predict
link |
position of moving element?
link |
You believe that it is God's law,
link |
that God created the world,
link |
which obey to this physical law.
link |
Or it is just law for predictions.
link |
And which one is instrumentalism?
link |
If you believe that this is law of God,
link |
and it's always true everywhere,
link |
that means that you're realist.
link |
So you're trying to really understand God's thought.
link |
So the way you see the world is as an instrumentalist?
link |
You know, I'm working for some models,
link |
model of machine learning.
link |
So in this model, we can see setting,
link |
and we try to solve,
link |
resolve the setting to solve the problem.
link |
And you can do in two different way.
link |
From the point of view of instrumentalist,
link |
and that's what everybody does now.
link |
Because they say that goal of machine learning
link |
is to find the rule for classification.
link |
But it is instrument for prediction.
link |
But I can say the goal of machine learning
link |
is to learn about conditional probability.
link |
So how God played use, and if he play,
link |
what is probability for one,
link |
what is probability for another, given situation.
link |
But for prediction, I don't need this.
link |
But for understanding, I need conditional probability.
link |
So let me just step back a little bit first to talk about,
link |
you mentioned, which I read last night,
link |
the parts of the 1960 paper by Eugene Wigner,
link |
Unreasonable Effectiveness of Mathematics
link |
and Natural Sciences.
link |
Such a beautiful paper, by the way.
link |
Made me feel, to be honest,
link |
to confess my own work in the past few years
link |
on deep learning, heavily applied.
link |
Made me feel that I was missing out
link |
on some of the beauty of nature
link |
in the way that math can uncover.
link |
So let me just step away from the poetry of that for a second.
link |
How do you see the role of math in your life?
link |
Is it a tool, is it poetry?
link |
Where does it sit?
link |
And does math for you have limits of what it can describe?
link |
Some people say that math is language which use God.
link |
So I believe that...
link |
Speak to God or use God or...
link |
So I believe that this article
link |
about effectiveness, unreasonable effectiveness of math,
link |
is that if you're looking at mathematical structures,
link |
they know something about reality.
link |
And the most scientists from Natural Science,
link |
they're looking on equation and trying to understand reality.
link |
So the same in machine learning.
link |
If you try very carefully look on all equations
link |
which define conditional probability,
link |
you can understand something about reality
link |
more than from your fantasy.
link |
So math can reveal the simple underlying principles of reality perhaps.
link |
You know what means simple?
link |
It is very hard to discover them.
link |
But then when you discover them and look at them,
link |
you see how beautiful they are.
link |
And it is surprising why people did not see that before.
link |
You're looking on equation and derive it from equations.
link |
For example, I talked yesterday about least square method.
link |
And people had a lot of fantasy how to improve least square method.
link |
But if you're going step by step by solving some equations,
link |
you suddenly will get some term which after thinking,
link |
you understand that it describes position of observation point.
link |
In least square method, we throw out a lot of information.
link |
We don't look in composition of point of observations,
link |
we're looking only on residuals.
link |
But when you understood that, that's very simple idea,
link |
but it's not too simple to understand.
link |
And you can derive this just from equations.
link |
So some simple algebra, a few steps will take you to something surprising
link |
that when you think about, you understand.
link |
And that is proof that human intuition is not too rich and very primitive.
link |
And it does not see very simple situations.
link |
So let me take a step back.
link |
But what about human, as opposed to intuition, ingenuity?
link |
Moments of brilliance.
link |
Do you have to be so hard on human intuition?
link |
Are there moments of brilliance in human intuition?
link |
They can leap ahead of math and then the math will catch up?
link |
I think that the best human intuition, it is putting in axioms.
link |
And then it is technical.
link |
See where the axioms take you.
link |
But if they correctly take axioms.
link |
But it axiom polished during generations of scientists.
link |
And this is integral wisdom.
link |
That is beautifully put.
link |
But if you maybe look at, when you think of Einstein and special relativity,
link |
what is the role of imagination coming first there in the moment of discovery of an idea?
link |
So there is obviously a mix of math and out of the box imagination there.
link |
That I don't know.
link |
Whatever I did, I exclude any imagination.
link |
Because whatever I saw in machine learning that comes from imagination,
link |
like features, like deep learning, they are not relevant to the problem.
link |
When you are looking very carefully from mathematical equations,
link |
you are deriving very simple theory, which goes far beyond theoretically
link |
than whatever people can imagine.
link |
Because it is not good fantasy.
link |
It is just interpretation.
link |
It is just fantasy.
link |
But it is not what you need.
link |
You don't need any imagination to derive the main principle of machine learning.
link |
When you think about learning and intelligence,
link |
maybe thinking about the human brain and trying to describe mathematically
link |
the process of learning, that is something like what happens in the human brain.
link |
Do you think we have the tools currently?
link |
Do you think we will ever have the tools to try to describe that process of learning?
link |
It is not description what is going on.
link |
It is interpretation.
link |
It is your interpretation.
link |
Your vision can be wrong.
link |
You know, one guy invented microscope, Levenhuk, for the first time.
link |
Only he got this instrument and he kept secret about microscope.
link |
But he wrote a report in London Academy of Science.
link |
In his report, when he was looking at the blood,
link |
he looked everywhere, on the water, on the blood, on the sperm.
link |
But he described blood like fight between queen and king.
link |
So, he saw blood cells, red cells, and he imagined that it is army fighting each other.
link |
And it was his interpretation of situation.
link |
And he sent this report in Academy of Science.
link |
They very carefully looked because they believed that he is right.
link |
But he gave wrong interpretation.
link |
And I believe the same can happen with brain.
link |
The most important part.
link |
You know, I believe in human language.
link |
In some proverbs, there is so much wisdom.
link |
For example, people say that it is better than thousand days of diligent studies one day with great teacher.
link |
But if I will ask you what teacher does, nobody knows.
link |
And that is intelligence.
link |
But we know from history and now from math and machine learning that teacher can do a lot.
link |
So, what from a mathematical point of view is the great teacher?
link |
That's an open question.
link |
No, but we can say what teacher can do.
link |
He can introduce some invariants, some predicate for creating invariants.
link |
I don't know because teacher knows reality and can describe from this reality a predicate, invariants.
link |
But he knows that when you're using invariant, you can decrease number of observations hundred times.
link |
So, but maybe try to pull that apart a little bit.
link |
I think you mentioned like a piano teacher saying to the student, play like a butterfly.
link |
I play guitar for a long time.
link |
Yeah, maybe it's romantic, poetic, but it feels like there's a lot of truth in that statement.
link |
Like there is a lot of instruction in that statement.
link |
And so, can you pull that apart?
link |
The language itself may not contain this information.
link |
It is not blah, blah, blah.
link |
It is not blah, blah, blah.
link |
It affects your playing.
link |
Yes, it does, but it's not the laying.
link |
It feels like what is the information being exchanged there?
link |
What is the nature of information?
link |
What is the representation of that information?
link |
I believe that it is sort of predicate, but I don't know.
link |
That is exactly what intelligence and machine learning should be.
link |
Because the rest is just mathematical technique.
link |
I think that what was discovered recently is that there is two mechanism of learning.
link |
One called strong convergence mechanism and weak convergence mechanism.
link |
Before, people use only one convergence.
link |
In weak convergence mechanism, you can use predicate.
link |
That's what play like butterfly and it will immediately affect your playing.
link |
You know, there is English proverb, great.
link |
If it looks like a duck, swims like a duck, and quack like a duck, then it is probably duck.
link |
But this is exact about predicate.
link |
Looks like a duck, what it means.
link |
You saw many ducks that you're training data.
link |
So, you have description of how looks integral looks ducks.
link |
The visual characteristics of a duck.
link |
But you want and you have model for recognition.
link |
So, you would like so that theoretical description from model coincide with empirical description,
link |
which you saw on territory.
link |
So, about looks like a duck, it is general.
link |
But what about swims like a duck?
link |
You should know that duck swims.
link |
You can say it play chess like a duck.
link |
Duck doesn't play chess.
link |
And it is completely legal predicate, but it is useless.
link |
So, half teacher can recognize not useless predicate.
link |
So, up to now, we don't use this predicate in existing machine learning.
link |
So, why we need zillions of data.
link |
But in this English proverb, they use only three predicate.
link |
Looks like a duck, swims like a duck, and quack like a duck.
link |
So, you can't deny the fact that swims like a duck and quacks like a duck has humor in it, has ambiguity.
link |
Let's talk about swim like a duck.
link |
It doesn't say jump like a duck.
link |
It's not relevant.
link |
But that means that you know ducks, you know different birds, you know animals.
link |
And you derive from this that it is relevant to say swim like a duck.
link |
So, underneath, in order for us to understand swims like a duck, it feels like we need to know millions of other little pieces of information.
link |
Which we pick up along the way.
link |
You don't think so.
link |
There doesn't need to be this knowledge base in those statements carries some rich information that helps us understand the essence of duck.
link |
How far are we from integrating predicates?
link |
You know that when you consider complete theory of machine learning.
link |
So, what it does, you have a lot of functions.
link |
And then you're talking it looks like a duck.
link |
You see your training data.
link |
From training data you recognize like expected duck should look.
link |
Then you remove all functions which does not look like you think it should look from training data.
link |
So, you decrease amount of function from which you pick up one.
link |
Then you give a second predicate and again decrease the set of function.
link |
And after that you pick up the best function you can find.
link |
It is standard machine learning.
link |
So, why you need not too many examples?
link |
Because your predicates aren't very good?
link |
That means that predicates are very good because every predicate is invented to decrease admissible set of function.
link |
So, you talk about admissible set of functions and you talk about good functions.
link |
So, what makes a good function?
link |
So, admissible set of function is set of function which has small capacity or small diversity, small VC dimension example.
link |
Which contain good function inside.
link |
So, by the way for people who don't know VC, you're the V in the VC.
link |
So, how would you describe to lay person what VC theory is?
link |
How would you describe VC?
link |
So, when you have a machine.
link |
So, machine capable to pick up one function from the admissible set of function.
link |
But set of admissible function can be big.
link |
So, it contain all continuous functions and it's useless.
link |
You don't have so many examples to pick up function.
link |
But it can be small.
link |
Small, we call it capacity but maybe better called diversity.
link |
So, not very different function in the set.
link |
It's infinite set of function but not very diverse.
link |
So, it is small VC dimension.
link |
When VC dimension is small, you need small amount of training data.
link |
So, the goal is to create admissible set of functions which is have small VC dimension and contain good function.
link |
Then you will be able to pick up the function using small amount of observations.
link |
So, that is the task of learning?
link |
Is creating a set of admissible functions that has a small VC dimension and then you've figure out a clever way of picking up?
link |
No, that is goal of learning which I formulated yesterday.
link |
Statistical learning theory does not involve in creating admissible set of function.
link |
In classical learning theory, everywhere, 100% in textbook, the set of function, admissible set of function is given.
link |
But this is science about nothing because the most difficult problem to create admissible set of functions
link |
given, say, a lot of functions, continuum set of function, create admissible set of functions.
link |
That means that it has finite VC dimension, small VC dimension and contain good function.
link |
So, this was out of consideration.
link |
So, what's the process of doing that?
link |
I mean, it's fascinating.
link |
What is the process of creating this admissible set of functions?
link |
That is invariant.
link |
Yeah, you're looking of properties of training data and properties means that you have some function
link |
and you just count what is value, average value of function on training data.
link |
You have model and what is expectation of this function on the model and they should coincide.
link |
So, the problem is about how to pick up functions.
link |
It can be any function.
link |
In fact, it is true for all functions.
link |
But because when we're talking, say, duck does not jumping, so you don't ask question jump like a duck
link |
because it is trivial.
link |
It does not jumping and doesn't help you to recognize jump.
link |
But you know something, which question to ask and you're asking it seems like a duck,
link |
but looks like a duck at this general situation.
link |
Looks like, say, guy who have this illness, this disease.
link |
So, there is a general type of predicate looks like and special type of predicate,
link |
which related to this specific problem.
link |
And that is intelligence part of all this business and that where teacher is involved.
link |
Incorporating the specialized predicates.
link |
What do you think about deep learning as neural networks, these arbitrary architectures
link |
as helping accomplish some of the tasks you're thinking about?
link |
Their effectiveness or lack thereof?
link |
What are the weaknesses and what are the possible strengths?
link |
You know, I think that this is fantasy, everything which like deep learning, like features.
link |
Let me give you this example.
link |
One of the greatest books is Churchill book about history of Second World War.
link |
And he started this book describing that in old time when war is over, so the great kings,
link |
they gathered together, almost all of them were relatives, and they discussed what should
link |
be done, how to create peace.
link |
And they came to agreement.
link |
And when happened First World War, the general public came in power.
link |
And they were so greedy that robbed Germany.
link |
And it was clear for everybody that it is not peace, that peace will last only 20 years
link |
because they were not professionals.
link |
And the same I see in machine learning.
link |
There are mathematicians who are looking for the problem from a very deep point of view,
link |
mathematical point of view.
link |
And there are computer scientists who mostly does not know mathematics.
link |
They just have interpretation of that.
link |
And they invented a lot of blah, blah, blah interpretations like deep learning.
link |
Why you need deep learning?
link |
Mathematic does not know deep learning.
link |
Mathematic does not know neurons.
link |
It is just function.
link |
If you like to say piecewise linear function, say that and do in class of piecewise linear
link |
But they invent something.
link |
And then they try to prove advantage of that through interpretations, which mostly wrong.
link |
And when it's not enough, they appeal to brain, which they know nothing about that.
link |
Nobody knows what's going on in the brain.
link |
So, I think that more reliable work on math.
link |
This is a mathematical problem.
link |
Do your best to solve this problem.
link |
Try to understand that there is not only one way of convergence, which is strong way of
link |
There is a weak way of convergence, which requires predicate.
link |
And if you will go through all this stuff, you will see that you don't need deep learning.
link |
Even more, I would say one of the theory, which called represented theory.
link |
It says that optimal solution of mathematical problem, which is described learning is on
link |
shadow network, not on deep learning.
link |
And a shallow network.
link |
The ultimate problem is there.
link |
In the end, what you're saying is exactly right.
link |
The question is you have no value for throwing something on the table, playing with it, not
link |
It's like a neural network where you said throwing something in the bucket or the biological
link |
example and looking at kings and queens or the cells or the microscope.
link |
You don't see value in imagining the cells or kings and queens and using that as inspiration
link |
and imagination for where the math will eventually lead you.
link |
You think that interpretation basically deceives you in a way that's not productive.
link |
I think that if you're trying to analyze this business of learning and especially discussion
link |
about deep learning, it is discussion about interpretation, not about things, about what
link |
you can say about things.
link |
But aren't you surprised by the beauty of it?
link |
So not mathematical beauty, but the fact that it works at all or are you criticizing that
link |
very beauty, our human desire to interpret, to find our silly interpretations in these
link |
Let me ask you this.
link |
Are you surprised and does it inspire you?
link |
How do you feel about the success of a system like AlphaGo at beating the game of Go?
link |
Using neural networks to estimate the quality of a board and the quality of the position.
link |
That is your interpretation, quality of the board.
link |
So it's not our interpretation.
link |
The fact is a neural network system, it doesn't matter, a learning system that we don't I
link |
think mathematically understand that well, beats the best human player, does something
link |
that was thought impossible.
link |
That means that it's not a very difficult problem.
link |
So you empirically, we've empirically have discovered that this is not a very difficult
link |
So maybe, can't argue.
link |
So even more I would say that if they use deep learning, it is not the most effective
link |
way of learning theory.
link |
And usually when people use deep learning, they're using zillions of training data.
link |
But you don't need this.
link |
So I describe challenge, can we do some problems which do well deep learning method, this deep
link |
net, using hundred times less training data.
link |
Even more, some problems deep learning cannot solve because it's not necessary they create
link |
admissible set of function.
link |
To create deep architecture means to create admissible set of functions.
link |
You cannot say that you're creating good admissible set of functions.
link |
You just, it's your fantasy.
link |
It does not come from us.
link |
But it is possible to create admissible set of functions because you have your training
link |
That actually for mathematicians, when you consider a variant, you need to use law of
link |
When you're making training in existing algorithm, you need uniform law of large numbers, which
link |
is much more difficult, it requires VC dimension and all this stuff.
link |
But nevertheless, if you use both weak and strong way of convergence, you can decrease
link |
a lot of training data.
link |
You could do the three, the swims like a duck and quacks like a duck.
link |
So let's step back and think about human intelligence in general.
link |
Clearly that has evolved in a non mathematical way.
link |
It wasn't, as far as we know, God or whoever didn't come up with a model and place in our
link |
brain of admissible functions.
link |
It kind of evolved.
link |
I don't know, maybe you have a view on this.
link |
So Alan Turing in the 50s, in his paper, asked and rejected the question, can machines think?
link |
It's not a very useful question, but can you briefly entertain this useful, useless question?
link |
Can machines think?
link |
So talk about intelligence and your view of it.
link |
I don't know that.
link |
I know that Turing described imitation.
link |
If computer can imitate human being, let's call it intelligent.
link |
And he understands that it is not thinking computer.
link |
He completely understands what he's doing.
link |
But he set up problem of imitation.
link |
So now we understand that the problem is not in imitation.
link |
I'm not sure that intelligence is just inside of us.
link |
It may be also outside of us.
link |
I have several observations.
link |
So when I prove some theorem, it's very difficult theorem, in couple of years, in several places,
link |
people prove the same theorem, say, Sawyer Lemma, after us was done, then another guys
link |
proved the same theorem.
link |
In the history of science, it's happened all the time.
link |
For example, geometry, it's happened simultaneously, first it did Lobachevsky and then Gauss and
link |
Boyai and another guys, and it's approximately in 10 times period, 10 years period of time.
link |
And I saw a lot of examples like that.
link |
And many mathematicians think that when they develop something, they develop something
link |
in general which affect everybody.
link |
So maybe our model that intelligence is only inside of us is incorrect.
link |
It's our interpretation.
link |
It might be there exists some connection with world intelligence.
link |
You're almost like plugging in into...
link |
And contributing to this...
link |
Into a big network.
link |
Into a big, maybe in your own network.
link |
On the flip side of that, maybe you can comment on big O complexity and how you see classifying
link |
algorithms by worst case running time in relation to their input.
link |
So that way of thinking about functions, do you think p equals np, do you think that's
link |
an interesting question?
link |
Yeah, it is an interesting question.
link |
But let me talk about complexity in about worst case scenario.
link |
There is a mathematical setting.
link |
When I came to United States in 1990, people did not know, they did not know statistical
link |
So in Russia, it was published to monographs, our monographs, but in America they didn't
link |
Then they learned and somebody told me that it is worst case theory and they will create
link |
real case theory, but till now it did not.
link |
Because it is mathematical too.
link |
You can do only what you can do using mathematics.
link |
And which has a clear understanding and clear description.
link |
And for this reason, we introduce complexity.
link |
And we need this because using, actually it is diversity, I like this one more.
link |
You see the mention, you can prove some theorems.
link |
But we also create theory for case when you know probability measure.
link |
And that is the best case which can happen, it is entropy theory.
link |
So from mathematical point of view, you know the best possible case and the worst possible
link |
You can derive different model in medium, but it's not so interesting.
link |
You think the edges are interesting?
link |
The edges are interesting because it is not so easy to get good bound, exact bound.
link |
It's not many cases where you have the bound is not exact.
link |
But interesting principles which discover the mass.
link |
Do you think it's interesting because it's challenging and reveals interesting principles
link |
that allow you to get those bounds?
link |
Or do you think it's interesting because it's actually very useful for understanding the
link |
essence of a function of an algorithm?
link |
So it's like me judging your life as a human being by the worst thing you did and the best
link |
thing you did versus all the stuff in the middle.
link |
It seems not productive.
link |
I don't think so because you cannot describe situation in the middle.
link |
So it will be not general.
link |
So you can describe edges cases and it is clear it has some model, but you cannot describe
link |
model for every new case.
link |
So you will be never accurate when you're using model.
link |
But from a statistical point of view, the way you've studied functions and the nature
link |
of learning in the world, don't you think that the real world has a very long tail?
link |
That the edge cases are very far away from the mean, the stuff in the middle or no?
link |
I don't know that.
link |
I think that from my point of view, if you will use formal statistic, you need uniform
link |
law of large numbers.
link |
If you will use this invariance business, you will need just law of large numbers.
link |
And there's this huge difference between uniform law of large numbers and large numbers.
link |
Is it useful to describe that a little more or should we just take it to...
link |
For example, when I'm talking about duck, I give three predicates and that was enough.
link |
But if you will try to do formal distinguish, you will need a lot of observations.
link |
So that means that information about looks like a duck contain a lot of bit of information,
link |
formal bits of information.
link |
So we don't know that how much bit of information contain things from artificial and from intelligence.
link |
And that is the subject of analysis.
link |
Till now, all business, I don't like how people consider artificial intelligence.
link |
They consider us some codes which imitate activity of human being.
link |
It is not science, it is applications.
link |
You would like to imitate go ahead, it is very useful and a good problem.
link |
But you need to learn something more.
link |
How people try to do, how people can to develop, say, predicates seems like a duck or play
link |
like butterfly or something like that.
link |
Not the teacher says you, how it came in his mind, how he choose this image.
link |
So that process...
link |
That is problem of intelligence.
link |
That is the problem of intelligence and you see that connected to the problem of learning?
link |
Because you immediately give this predicate like specific predicate seems like a duck
link |
or quack like a duck.
link |
It was chosen somehow.
link |
So what is the line of work, would you say?
link |
If you were to formulate as a set of open problems, that will take us there, to play
link |
We'll get a system to be able to...
link |
Let's separate two stories.
link |
One mathematical story that if you have predicate, you can do something.
link |
And another story how to get predicate.
link |
It is intelligence problem and people even did not start to understand intelligence.
link |
Because to understand intelligence, first of all, try to understand what do teachers.
link |
How teacher teach, why one teacher better than another one.
link |
And so you think we really even haven't started on the journey of generating the predicates?
link |
We don't understand.
link |
We even don't understand that this problem exists.
link |
Because did you hear...
link |
No, I just know name.
link |
I want to understand why one teacher better than another and how affect teacher, student.
link |
It is not because he repeating the problem which is in textbook.
link |
He makes some remarks.
link |
He makes some philosophy of reasoning.
link |
Yeah, that's a beautiful...
link |
So it is a formulation of a question that is the open problem.
link |
Why is one teacher better than another?
link |
What he does better.
link |
How do they get better?
link |
What does it mean to be better?
link |
From whatever model I have, one teacher can give a very good predicate.
link |
One teacher can say swims like a dog and another can say jump like a dog.
link |
And jump like a dog carries zero information.
link |
So what is the most exciting problem in statistical learning you've ever worked on or are working
link |
I just finished this invariant story and I'm happy that...
link |
I believe that it is ultimate learning story.
link |
At least I can show that there are no another mechanism, only two mechanisms.
link |
But they separate statistical part from intelligent part and I know nothing about intelligent
link |
And if you will know this intelligent part, so it will help us a lot in teaching, in learning.
link |
You will know it when we see it?
link |
So for example, in my talk, the last slide was a challenge.
link |
So you have say NIST digit recognition problem and deep learning claims that they did it
link |
very well, say 99.5% of correct answers.
link |
But they use 60,000 observations.
link |
Can you do the same using hundred times less?
link |
But incorporating invariants, what it means, you know, digit one, two, three.
link |
But looking on that, explain to me which invariant I should keep to use hundred examples or say
link |
hundred times less examples to do the same job.
link |
Yeah, that last slide, unfortunately your talk ended quickly, but that last slide was
link |
a powerful open challenge and a formulation of the essence here.
link |
What is the exact problem of intelligence?
link |
Because everybody, when machine learning started and it was developed by mathematicians, they
link |
immediately recognized that we use much more training data than humans needed.
link |
But now again, we came to the same story, have to decrease.
link |
That is the problem of learning.
link |
It is not like in deep learning, they use zillions of training data because maybe zillions
link |
are not enough if you have a good invariants.
link |
Maybe you will never collect some number of observations.
link |
But now it is a question to intelligence, how to do that?
link |
Because statistical part is ready, as soon as you supply us with predicate, we can do
link |
good job with small amount of observations.
link |
And the very first challenge is well known digit recognition.
link |
And you know digits, and please tell me invariants.
link |
I think about that, I can say for digit three, I would introduce concept of horizontal symmetry.
link |
So the digit three has horizontal symmetry, say more than, say, digit two or something
link |
But as soon as I get the idea of horizontal symmetry, I can mathematically invent a lot
link |
of measure of horizontal symmetry, or then vertical symmetry, or diagonal symmetry, whatever,
link |
if I have idea of symmetry.
link |
I think on digit I see that it is meta predicate, which is not shape, it is something like symmetry,
link |
like how dark is whole picture, something like that, which can self rise a predicate.
link |
You think such a predicate could rise out of something that is not general, meaning
link |
it feels like for me to be able to understand the difference between two and three, I would
link |
need to have had a childhood of 10 to 15 years playing with kids, going to school, being
link |
yelled by parents, all of that, walking, jumping, looking at ducks, and then I would be able
link |
to generate the right predicate for telling the difference between two and a three.
link |
Or do you think there's a more efficient way?
link |
I know for sure that you must know something more than digits.
link |
And that's a powerful statement.
link |
But maybe there are several languages of description, these elements of digits.
link |
So I'm talking about symmetry, about some properties of geometry, I'm talking about
link |
something abstract.
link |
I don't know that.
link |
But this is a problem of intelligence.
link |
So in one of our articles, it is trivial to show that every example can carry not more
link |
than one bit of information in real.
link |
Because when you show example and you say this is one, you can remove, say, a function
link |
which does not tell you one, say, is the best strategy.
link |
If you can do it perfectly, it's remove half of the functions.
link |
But when you use one predicate, which looks like a duck, you can remove much more functions
link |
And that means that it contains a lot of bit of information from formal point of view.
link |
But when you have a general picture of what you want to recognize and general picture
link |
of the world, can you invent this predicate?
link |
And that predicate carries a lot of information.
link |
Maybe just me, but in all the math you show, in your work, which is some of the most profound
link |
mathematical work in the field of learning AI and just math in general, I hear a lot
link |
of poetry and philosophy.
link |
You really kind of talk about philosophy of science.
link |
There's a poetry and music to a lot of the work you're doing and the way you're thinking
link |
So do you, where does that come from?
link |
Do you escape to poetry?
link |
Do you escape to music or not?
link |
I think that there exists ground truth.
link |
There exists ground truth?
link |
And that can be seen everywhere.
link |
The smart guy, philosopher, sometimes I'm surprised how they deep see.
link |
Sometimes I see that some of them are completely out of subject.
link |
But the ground truth I see in music.
link |
Music is the ground truth?
link |
And in poetry, many poets, they believe, they take dictation.
link |
So what piece of music as a piece of empirical evidence gave you a sense that they are touching
link |
something in the ground truth?
link |
The structure of the math of music.
link |
Yeah, because when you're listening to Bach, you see the structure.
link |
Very clear, very classic, very simple, and the same in math when you have axioms in geometry,
link |
you have the same feeling.
link |
And in poetry, sometimes you see the same.
link |
And if you look back at your childhood, you grew up in Russia, you maybe were born as
link |
a researcher in Russia, you've developed as a researcher in Russia, you've came to United
link |
States and a few places.
link |
If you look back, what was some of your happiest moments as a researcher, some of the most
link |
profound moments, not in terms of their impact on society, but in terms of their impact on
link |
how damn good you feel that day and you remember that moment?
link |
You know, every time when you found something, it is great in the life, every simple things.
link |
But my general feeling is that most of my time was wrong.
link |
You should go again and again and again and try to be honest in front of yourself, not
link |
to make interpretation, but try to understand that it's related to ground truth, it is not
link |
my blah, blah, blah interpretation and something like that.
link |
But you're allowed to get excited at the possibility of discovery.
link |
You have to double check it.
link |
No, but how it's related to another ground truth, is it just temporary or it is for forever?
link |
You know, you always have a feeling when you found something, how big is that?
link |
So 20 years ago when we discovered statistical learning theory, nobody believed, except for
link |
one guy, Dudley from MIT, and then in 20 years it became fashion, and the same with support
link |
vector machines, that is kernel machines.
link |
So with support vector machines and learning theory, when you were working on it, you had
link |
a sense, you had a sense of the profundity of it, how this seems to be right, this seems
link |
I recognized that it will last forever, and now when I found this invariant story, I have
link |
a feeling that it is complete learning, because I have proof that there are no different mechanisms.
link |
You can have some cosmetic improvement you can do, but in terms of invariants, you need
link |
both invariants and statistical learning, and they should work together.
link |
But also I'm happy that we can formulate what is intelligence from that, and to separate
link |
from technical part, and that is completely different.
link |
Well, Vladimir, thank you so much for talking today.