The following is a conversation with Vladimir Vapnik.

Part two, the second time we spoke on the podcast.

He's the co inventor of support vector machines,

support vector clustering, VC theory,

and many foundational ideas and statistical learning.

He was born in the Soviet Union,

worked at the Institute of Control Sciences in Moscow,

then in the US, worked at AT&T, NEC labs,

Facebook AI research,

and now is a professor at Columbia University.

His work has been cited over 200,000 times.

The first time we spoke on the podcast was just over a year ago,

one of the early episodes.

This time, we spoke after a lecture he gave titled

Complete Statistical Theory of Learning,

as part of the MIT series of lectures on deep learning and AI that I organized.

I'll release the video of the lecture in the next few days.

This podcast and lecture are independent from each other,

so you don't need one to understand the other.

The lecture is quite technical and math heavy,

so if you do watch both,

I recommend listening to this podcast first,

since the podcast is probably a bit more accessible.

This is the Artificial Intelligence Podcast.

If you enjoy it, subscribe on YouTube,

give it five stars on Apple Podcasts,

support it on Patreon,

or simply connect with me on Twitter,

and Lex Friedman spelled F R I D M A N.

As usual, I'll do one or two minutes of ads now,

and never any ads in the middle that can break the flow of the conversation.

I hope that works for you,

and doesn't hurt the listening experience.

This show is presented by Cash App,

the number one finance app in the App Store.

When you get it, use code LEX Podcast.

Cash App lets you send money to friends, buy Bitcoin,

and invest in the stock market with as little as $1.

Brokerage services are provided by Cash App Investing,

a subsidiary of Square, and member SIPC.

Since Cash App allows you to send and receive money digitally peer to peer,

and security in all digital transactions is very important,

let me mention the PCI Data Security Standard,

PCI DSS Level 1, that Cash App is compliant with.

I'm a big fan of standards for safety and security,

and PCI DSS is a good example of that,

where a bunch of competitors got together

and agreed that there needs to be a global standard

around the security of transactions.

Now, we just need to do the same for autonomous vehicles

and AI systems in general.

So again, if you get Cash App from the App Store,

or Google Play, and use the code LEX Podcast,

you get $10, and Cash App will also donate $10 to first,

one of my favorite organizations that is helping to advance

robotics and STEM education for young people around the world.

And now, here's my conversation with Vladimir Vapnick.

You and I talked about Alan Turing yesterday, a little bit,

and that he, as the father of artificial intelligence,

may have instilled in our field an ethic of engineering,

and not science, seeking more to build intelligence

rather than to understand it.

What do you think is the difference between these two paths

of engineering intelligence and the science of intelligence?

It's a completely different story.

Engineering is a mutation of human activity.

You have to make a device which behaves as human behavior,

have all the functions of human.

It does not matter how you do it.

But to understand what is intelligence about is quite a different problem.

So, I think, I believe that it's somehow related to predicate we talked yesterday about,

because look at the Vladimir Props idea.

He just found 31 he predicates.

He called it units, which can explain human behavior,

at least in Russian tales.

He looked at Russian tales and derived from that,

and then people realized that it's more wide than in Russian tales.

It isn't TV, in movie serials, and so on and so on.

You're talking about Vladimir Prop, who in 1928 published a book,

Morphology of the Folktale, describing 31 predicates that have this kind of sequential

structure that a lot of the stories and narratives follow in Russian folklore and other content.

We'll talk about it.

I'd like to talk about predicates in a focused way,

but if you allow me to stay zoomed out on our friend Alan Turing, and he inspired a generation

with the imitation game.

If we can linger on that a little bit longer, do you think learning to imitate

intelligence can get us closer to understanding intelligence?

So, why do you think imitation is so far from understanding?

I think that it is different between you have different goals.

So, your goal is to create something, something useful, and that is great.

And you can see how much things was done, and I believe that it will be done even more.

It's self driving cars and also this business.

It is great, and it was inspired by Turing vision, but understanding is very difficult.

It's more or less philosophical category.

What means understanding the world?

I believe in a scheme which starts from Plato, that there exists a world of ideas.

I believe that intelligence, it is a world of ideas, but it is a world of pure ideas.

And when you combine them with reality things, it creates, as in my case,

invariance, which is very specific, and that's, I believe, the combination of ideas

in way to constructing invariant is intelligence.

But first of all, predicate.

If you know predicate, and hopefully then, not too much predicate exists.

For example, 31 predicate for human behavior, it is not a lot.

Vladimir Prop used 31, you can even call it predicate, 31 predicate to describe stories,

narratives.

So you think human behavior, how much of human behavior, how much of our world, our universe,

all the things that matter in our existence can be summarized in predicates of the kind

that Prop was working with?

I think that we have a lot of form of behavior.

But I think that predicate is much less, because even in these examples, which I gave

you yesterday, you saw that predicate can be, one predicate can construct many different

invariance, depending on your data, they're applying to different data, and they give different

invariance.

So, but pure ideas, maybe not so much.

Not so many.

I don't know about that.

But my guess, I hope, that's why challenge about digital recognition, how much you need.

I think we'll talk about computer vision and 2D images a little bit and your challenge.

That's exactly about intelligence.

That's exactly about, no, that hopes to be exactly about the spirit of intelligence

in the simplest possible way.

Well, there's an open question whether starting at the MNIST digital recognition

is a step towards intelligence or it's an entirely different thing.

I think that to beat records using 100, 200 times less examples, you need intelligence.

You need intelligence.

So, let's, because you use this term and it'll be nice.

I'd like to ask simple, maybe even dumb questions.

Let's start with a predicate.

In terms of terms and how you think about it, what is a predicate?

I don't know. I have a feeling, formally, they exist.

But I believe that predicate for 2D images, one of them is symmetry.

Hold on a second.

Sorry, sorry to interrupt and put you back.

At the simplest level, we're not even, we're not being profound currently.

A predicate is a statement of something that is true.

Yes.

Do you think of predicates as somehow probabilistic in nature or is this binary,

this is truly constraints of logical statements about the world?

In my definition, the simplest predicate is function.

Function and you can use this function to make inner product that is predicate.

What's the input and what's the output of the function?

Input is x, something which is input in reality.

Say, if you consider digit recognition, it's pixel space, input.

But it is function which in pixel space, but it can be any function from pixel space.

And you choose, and I believe that there are several functions,

which is important for understanding of images.

One of them is symmetry.

It's not so simple construction as I described with literarity, with all this stuff.

But another, I believe, I don't know how many, is how well structured is picture.

Structurized?

Yeah.

What do you mean by structurized?

It is formal definition.

Say, something heavy on the left corner, not so heavy in the middle and so on.

You describe in general concept of what you assume.

Concepts, some kind of universal concepts.

Yeah, but I don't know how to formalize this.

Do you?

So this is the thing.

There's a million ways we can talk about this.

I'll keep bringing it up.

But we humans have such concepts when we look at digits.

But it's hard to put them, just like you're saying now, it's hard to put them into words.

You know, that is example.

When critics in music trying to describe music, they use predicate and not too many predicate.

But in different combination, but they have some special words for describing music.

And the same should be for images.

But maybe there are critics who understand essence of what this image is about.

Do you think there exists critics who can summarize the essence of images, human beings?

I hope so, yes, but that explicitly state them on paper.

The fundamental question I'm asking is, do you think there exists a small set of predicates

that will summarize images? It feels to our mind, like it does, that the concept of what

makes a two and a three and a four.

No, no, no.

It's not on this level.

What it should not describe, two, three, four.

It describes some construction which allow you to create invariance.

And invariance, sorry to stick on this, but terminology.

Invariance, it is, it is property of your image.

Say, I can say, looking at my image, it is more or less symmetric and I can give you

value of symmetry, say, level of symmetry using this function which I gave yesterday.

And you can describe that your image has these characteristics.

Exactly in the way how musical critics describe music.

So, but this is invariant applied to specific data, to specific music, to something.

I strongly believe in this plot ideas that there exists world of predicate and world of

reality and predicate and reality is somehow connected and you have to do that.

Let's talk about Plato a little bit. So, you draw a line from Plato to Hegel to Wigner to today.

Yes.

So, Plato has forms, the theory of forms.

There's a world of ideas and a world of things as you talk about and there's a connection.

And presumably the world of ideas is very small and the world of things is arbitrarily big.

But they're all what Plato calls them like, it's a shadow.

The real world is a shadow from the world of form.

Yeah, you have projection.

Projection.

Of world of idea.

Yeah, very poetic.

In reality, you can realize this projection using invariance because it is projection

for own specific examples which create specific features of specific objects.

So, the essence of intelligence is while only being able to observe the world of things,

try to come up with a world of ideas.

Exactly.

Like in this music story, intelligent musical critics knows this all this world and have

a feeling about what.

I feel like that's a contradiction, intelligent music critics.

But I think music is to be enjoyed in all its forms.

The notion of critic, like a food critic.

No, I don't want touching motion.

That's an interesting question.

There's a motion, there's a certain elements of the human psychology of the human experience

which seem to almost contradict intelligence and reason.

Emotion, like emotion, like fear, like love, all of those things, are those not connected

in any way to the space of ideas?

Yes, I don't know.

I just want to be concentrated on very simple story, on digit recognition.

So, you don't think you have to love and fear death in order to recognize digits?

I don't know, because it's so complicated.

It involves a lot of stuff which I never consider.

But I know about digit recognition.

And I know that for digit recognition, to get the records from small number of observations,

you need predicate, but not special predicate for this problem.

But universal predicate, which understand the world of images.

Of visual information.

Visual, yes.

But on the first step, they understand the world of handwritten digits or characters or something simple.

So, like you said, symmetry is an interesting one.

That's what I think one of the predicates related to symmetry, the level of symmetry.

Okay, degree of symmetry.

So, you think symmetry at the bottom is a universal notion and there's

degrees of a single kind of symmetry, or is there many kinds of symmetries?

Many kinds of symmetries.

There is a symmetry, anti symmetry, say letter S.

So, it has vertical anti symmetry.

And it could be diagonal symmetry, vertical symmetry.

So, when you cut vertically the letter S.

Yeah, then the upper part and lower part in different directions.

Yeah, inverted along the y axis.

Yeah.

But that's just like one example of symmetry, right?

Isn't there like...

All right, but there is a degree of symmetry.

If you play all this derivative stuff to do tangent distance,

but whatever I describe, you can have a degree of symmetry.

And that is what describing reason of image.

It is the same as you will describe this image, saying about digits.

It has anti symmetry, digits three, symmetric, more or less look for symmetry.

Do you think such concepts like symmetry, predicates like symmetry,

is it a hierarchical set of concepts?

Or are these independent, distinct predicates that we want to discover some set of?

There is a deal of symmetry.

And you can...

This idea of symmetry make very general, like degree of symmetry.

If degree of symmetry can be zero, no symmetry at all.

Or degree of symmetry, say, more or less symmetrical, but you have one of these descriptions.

And symmetry can be different, as I told, horizontal, vertical, diagonal,

and anti symmetry is also concept of symmetry.

What about shape in general?

I mean, symmetry is a fascinating notion, but...

No, no, I'm talking about digit.

I would like to concentrate on all...

I would like to know predicate for digit recognition.

Yes, but symmetry is not enough for digit recognition, right?

It is not necessarily for digit recognition.

It helps to create invariant, which you can use when you will have examples for digit recognition.

When you have problem of digit recognition, you have examples of the first class or second class.

Plus, you know that there exists concept of symmetry.

And you apply, when you're looking for decision rule, you will apply concept of symmetry,

of this level of symmetry, which you estimate from me.

So let's talk. Everything comes from big convergence.

What is convergence?

What is weak convergence?

What is strong convergence?

I'm sorry, I'm going to do this to you.

What are we converging from and to?

You're converging...

You would like to have a function.

The function, which, say, indicator function, which indicate your digit 5, for example.

A classification task.

Let's talk only about classification.

So classification means you will say, whether this is a 5 or not,

or say which of the 10 digits it is.

All right, all right.

I would like to have these functions.

Then I have some examples.

I can consider property of these examples.

Say symmetry.

And I can measure level of symmetry for every digit.

And then I can take average from my training data.

And I will consider only functions of conditional probability,

which I'm looking for my decision rule, which applying to digits

will give me the same average as I absorb on training data.

So actually, this is different level of description of what you want.

You want not just, you show not one digit, you show this predicate, show general property

of all digits, which you have in mind.

If you have in mind digits 3, it gives you property of digits 3.

And you select as admissible set of function, only function, which keeps this property.

You will not consider other functions.

So you're immediately looking for smaller subset of function.

That's what you mean by admissible functions.

You look at admissible function, exactly.

Which is still a pretty large for the number 3.

It is pretty large, but if you have one predicate.

But according to, there is a strong and weak convergence.

Strong convergence is convergence and function.

You're looking for the function on one function, and you're looking for another function.

And square difference from them should be small.

If you take difference in any points, make a square, make an integral, and it should be small.

That is convergence and function.

Suppose you have some function, any function.

So I would say, I say that some function converge to this function.

If integral from square difference between them is small.

That's the definition of strong convergence.

That definition of strong convergence.

Two functions, the integral of the difference is small.

It is convergence in functions.

But you have different convergence in functionals.

You take any function, you take some function phi,

and take inner product.

This function is f function, f0 function, which you want to find.

And that gives you some value.

So you say that set of functions converge in inner product to this function.

If this value of inner product converge to value f0, that is for one phi.

But phi converges, requires that it converge for any function of Hilbert space.

If it converge for any function of Hilbert space, then you would say that this is weak convergence.

You can think that when you take integral, that is property, integral property of function.

For example, if you will take sine or cosine, it is coefficient of, say, Fourier expansion.

So if it converge for all coefficients of Fourier expansion, so under some condition,

it converge to function you're looking for.

But weak convergence means any property.

Convergence not point wise, but integral property of function.

So weak convergence means integral property of functions.

When I'm talking about predicate, I would like to formulate which integral properties

I would like to have for convergence.

So and if I will take one predicate its function, which I measure property,

if I will use one predicate and say I will consider only function

which give me the same value as this predicate, I selecting set of functions

from functions which is admissible in the sense that function which I looking for

in this set of functions because I checking in training data, it gives the same.

Yeah, so it always has to be connected to the training data in terms of...

Yeah, but property, you can know independent on training data and this guy prop.

Yeah.

So there is formal property, 31 property and...

For fairy tale, Russian fairy tale.

Yeah, but Russian fairy tale is not so interesting.

More interesting that people apply this to movies, to theater, to different things and

the same works, they're universal.

Well, so I would argue that there's a little bit of a difference between

the kinds of things that were applied to which are essentially stories and digit recognition.

It is the same story.

You're saying digits, there's a story within the digit.

Yeah, so but my point is why I hope that it possible to beat record using not 60,000

but say 100 times less because instead you will give predicate and you will select your decision

not from wide set of functions, but from set of function which keeps us predicate.

But predicate is not related just to digit recognition.

Right, so...

Like in Plattus case.

Do you think it's possible to automatically discover the predicates?

So you basically said that the essence of intelligence is the discovery of good predicates.

Yeah, now the natural question is

you know that's what Einstein was good at doing in physics.

Can we make machines do these kinds of discovery of good predicates?

Or is this ultimately a human endeavor?

That's I don't know.

I don't think that machine can do because according to theory about weak convergence any function

from Hilbert space can be predicate.

So you have infinite number of predicate in upper and before you don't know which predicate is good

on which but whatever prop show and why people call it breaks through that there is not too many

predicate which cover most of situation happened in the world.

So there's a sea of predicates and most of the only a small amount are useful for the kinds of

things that happen in the world.

I think that I would say only small part of predicate very useful.

Useful all of them.

Only very few are what we should let's call them good predicates.

Very good predicates.

Very good predicates.

So can we linger on it?

What's your intuition?

Why is it hard for a machine to discover good predicates?

I even in my talk described how to do predicate have to find new predicate.

I'm not sure that it is very good.

What did you propose in your talk?

No, in my talk I gave example for diabetes.

When we achieve some percent so then we're looking for area where some sort of predicate

which I formulate does not keeps invariant.

So if it doesn't keep I retain my data I select only function which keeps this invariant

and when I did it I improve my performance.

I can looking for this predicate.

I know technically have to do that and you can of course do it using machine but I'm not

sure that we will construct the smartest predicate.

Well this is the allow me to linger on it because that's the essence that's the challenge

that is artificial that's that's the human level intelligence that we seek is the discovery of

these good predicates you've talked about deep learning as a way to the predicates they use

and the functions are mediocre.

We can find better ones.

Let's talk about deep learning.

Sure let's do it.

I know only Jan Slikun convolutional network and what else I don't know and it's a very

simple convolution.

There's not much else to know.

Left and right yes I can do it like that one is one predicate it is convolution is a single

predicate it's single it's single predicate yes but you know exactly you take the derivative

for translation and predicate should be kept.

So that's a single predicate but humans discovered that one or at least

not that is a risk not too many predicates and that is big story because Jan did it 25

years ago and nothing so clear was added to deep network and then I don't understand

why we should talk about deep network instead of talking about piecewise linear functions

which keeps us predicated.

Well the you know a counter argument is that maybe the amount of predicates necessary

to solve general intelligence say in space of images doing efficient recognition of

handwritten digits is very small and so we shouldn't be so obsessed about finding

we'll find other good predicates like convolution for example you know there there has been other

advancements like if you look at the work with attention there's intentional mechanisms

in especially used in natural language focusing the the network's ability to

to learn at which part of the input to look at.

The thing is there's other things besides predicates that are important for the actual

engineering mechanism of showing how much you can really do given such these predicates.

I mean that's essentially the work of deep learning is constructing architectures

that are able to be given the training data to be able to converge towards

a function that can approximate can generalize well.

It's an engineering problem.

Yeah I understand but let's talk not on emotional level but on a mathematical level.

You have set of piecewise linear functions.

It is all possible neural networks.

It's just piecewise linear functions.

There's many many pieces.

Large number of piecewise linear functions.

Exactly but very large.

Very large but it's still simpler than say convolution than reproducing

internal Hilbert space which have a Hilbert set of functions.

What's Hilbert space?

It's space with infinite number of coordinates a function for expansion something like that.

So it's much richer so and when I talking about closed form solution I talking about

this set of function not piecewise linear set which is particular case.

It is small part.

So neural networks is a small part of the space your talk of functions you're talking about.

Small small say small set of functions.

But it is fine.

It is fine.

I don't want to to discuss the small or big take advantage.

So you have some set of functions.

So now when you're trying to create architecture you would like to create admissible set of functions

all your tricks to use not all functions but some subset of this set of functions.

Say when you're introducing convolutional net it is way to make this subset useful for you.

But for my point of view convolutional it is something you want to keep some invariance

say translation invariance.

But now if you understand this and you cannot explain on the level of ideas what neural network does

you should agree that it is much better to have a set of functions and they say this set of functions

should be admissible it must keep this invariant this invariant and that invariant.

You know that as soon as you incorporate new invariance set of function because smaller

and smaller and smaller.

But all the invariance are specified by you the human.

Yeah but what I am hope that there is a standard predicate like prop show

that what that's what I want to find for digit recognition if we start it is completely new

area what is intelligence about on the level starting from from Plattus idea what is world of ideas.

So and I believe that it's not too many.

Yeah but you know it is amusing that mathematician doing something in neural network

in general function but people from literature from art they use this all the time that's right

invariance saying say it is great how how people describe music we should learn from that

and something on this level but so why Vladimir prop who was just theoretical who studied

theoretical literature he found that.

You know what let me throw that right back at you because there's a little bit of a

that's less mathematical and more emotional philosophical Vladimir prop I mean he wasn't

doing math no and you just said another emotional statement which is you believe that this

Plato world of ideas is small.

I hope I hope do you do what's your intuition though if we can linger on it.

You know it is not just small or big I know exactly then when I introducing

some predicate I decrease set of functions but my goal to decrease set of function much

by as much as possible by as much as possible good predicate which does this

then I should choose next predicate which does this decrease set as much as possible

so set of good predicate it is such that they decrease this amount of admissible function.

So if each good predicate significantly reduces the set of admissible functions that

there naturally should not be that many yeah predicates.

No but but if you reduce very well the VC dimension of the function of admissible set

of function is small and you need not too much training data to do well.

And VC dimension by the way is some measure of capacity of this set of functions.

Right how roughly speaking how many functions in this set so you're decreasing decreasing

and it makes it easy for you to find function you're looking for.

So the most important part to create good admissible set of functions and it probably

there are many ways but the good predicate is such that that can do that.

So let's for this duck you should know a little bit about duck because what are the

what are the three fundamental laws of ducks?

Looks like a duck swims like a duck and quack like a duck.

You should know something about ducks to be able to.

Not necessarily looks like say horse it's also good.

So it's not it generalizes from ducks.

And talk like and make sound like horse something and run like horse and moves like horse.

It is general it is general predicate that this applied to duck but for duck you can say

play chess like duck.

You cannot say play chess.

Why not?

So you're saying you can but that would not be a good.

No you will not reduce a lot of.

You will not do yeah you would not reduce the set of functions.

So you get the story is formal story mathematical story is that you can use any function you want

like the predicate but some of them are good some of them are not because some of them reduce a

lot of functions to admissible set of some of them.

But the question is I'll probably keep asking this question but how do we find such

what's your intuition?

So my hand written hand written recognition how do we find the the answer to your challenge?

Yeah yeah I understand it like that.

I understand what what to find what it means I knew predicate.

Yeah like guy who understand music can say this word which he described when he listened to music.

He understand music he use not too many different or you can do like prop.

You can make collection what he talking about music about this about that it's not too many

different situations he described.

Because we mentioned Vladimir proper bunch.

Let me just mention there's a there's a sequence of 31 structural notions that are common in stories

and I think he called units units and I think they resonate.

I mean it starts just to give an example of ascension a member of the hero's community

of family leaves the security of the home environment then it goes to the introduction

or forbidding edict or command is passed upon the hero don't go there don't do this the hero's

warrant against some action.

Then step three violet violation of interdiction breaks you know break the rules break out on

your own then reconnaissance the villain makes an effort to attain knowledge needing to fulfill

their plot so on it goes on like this ends ends in a wedding number 31 happily ever after.

No he he he just gave description of all situations he understands this world.

Of folk tales.

Yeah not folk stories and this story is not in just folk tales the stories in in detective

serials as well and probably in our lives we probably live read this and then they they

wrote that this predicate is good for different situation from movie from for movie for theater.

By the way there's also criticism right there's an other way to interpret narratives from

Claude Levy Strauss I think I don't I I'm not in this business I know I know it's

theoretical literature but it's looking at paradise it's always the the the the

discussion yeah yeah but at least there is a unit it's not too many units that can describe

but that's probably gives another unit or another way exactly another another set of units another

set of predicates it does not matter but they exist probably my my question is whether given those

units whether without our human brains to interpret these units they would still hold as much power

as they have meaning are those units enough when we give them to the alien species let me ask you

do you understand digit recognize digit images no I don't understand no no no uh when you can

recognize this digit images it means that you understand yes you understand characters you

understand no no no no I I it's the it's the imitation versus understanding question because

I don't understand the mechanism by which I am not talking about I'm talking about predicates

you understand that it involves symmetry maybe structure maybe something else I cannot formulate

I just was able to find symmetries so I guess symmetries that's really good so this is a good

line I feel like I understand the basic elements of what makes a good hand recognition system

my own like symmetry connects with me it seems like that's a very powerful predicate my question is

is there a lot more going on that we're not able to introspect maybe I need to be able to understand

a huge amount in the world of ideas uh thousands of predicates millions of predicates in order to

do hand recognition I don't think so so you're you're both your hope and your intuition of such

that let me explain you're using digits you're using examples as well theory says that if you

will use all possible functions from hilda space all possible predicate you don't need training data

you just will have admissible set of function which contain one function yes so the tradeoff

is when you're not using all predicates you're only using a few good predicates you need to

have some training data yes the more the the more good predicates you have the less training

data exactly that is intelligent still okay I'm going to keep asking the same dumb question

handwritten recognition to solve the challenge you kind of propose a challenge that says we

should be able to get state of the art amnesty error rates by using very few 60 maybe fewer

examples per digit what kind of predicates do you think that is the challenge so people who will

solve this problem they will answer they will answer do you think they'll be able to answer it

in a human explainable way they just need to write function that's it but so can that function

be written I guess by an automated reasoning system whether we're talking about a neural

network learning a particular function or another mechanism no narrow I'm not against

neural network I'm against admissible set of function which create neural network you did it

by hand you don't you don't do it by invariance by predicate by by by reason but neural networks

can then reverse do the reverse step of helping you find a function just the task of a neural

network is is to find a disentangled representation for example what they call is to find that one

predicate function that's really captures some kind of essence one not the entire essence but

one very useful essence of this particular visual space do you think that's possible like

listen I'm grasping hoping there's an automated way to find good predicates right so the question

is what are the mechanisms of finding good predicates ideas the you think we should pursue

a young grad student listening right now I gave example so find situation where

predicate which you're suggesting don't create invariant

it's like in physics find situation where existing theory cannot explain it

so you're finding contradictions find contradiction and then remove this contradiction

but in my case what means contradiction you find function which if you will use this function

you you're not keeping invariance so really the process of discovering contradictions yeah

it is like in physics find situation where you have contradiction for one of the property

for one of the predicate then include this predicate making invariance and solve again

this problem now you don't have contradiction but it is not

the best way probably I don't know to looking for predicate that's just one way okay that no no

it is brute force way the brute force way what about the ideas of some what big umbrella term

of symbolic AI there's what in 80s with expert systems sort of logic reasoning based systems

is there hope there to find some through sort of deductive reasoning to find good predicates

I don't think so I think that just logic is not enough it's kind of a compelling notion though

you know that when smart people sit in a room and reason through things it seems compelling

and making our machines do the same is also compelling so everything is very simple

when you have infinite number of predicate you can choose the the function you want you have

invariance and you can choose the function you want but you have to have a not too many invariance

to solve the problem so and half from infinite number of function to select finite number

and hopefully small fine number of functions which is good enough to extract small set of admissible

functions so they will be admissible it's for sure because every function just decrease set of

function and leaving it admissible but it will be small but why do you think logic

basic systems don't can't help intuition not because you you should know reality you should

know life this guy like prop he knows something and he tried to to put in invariant his understanding

that's the human yeah but see you're you're putting too much value into

you Vladimir props knowing something no it is my story is that what means you know life

what it means you know common sense no no you know something common sense it is some rules

you think so common sense is simply rules common sense is every it's

it's mortality it's no it's it's fear of death it's love it's spirituality it's a happiness and

sadness all of it is tied up into understanding gravity which is what we think of as common sense

I don't really to discuss so white I want to discuss understand digitally understand

digitally recognition anytime I bring up love and death you you bring it back to digital recognition

I don't like it no you know it is durable because there is a challenge yeah which I see how to solve

it if I will have a student concentrating this work I will suggest something to solve you mean

handwritten recognition yeah it's a beautifully simple elegant and yet I think that I know in

variants which will solve this you do I think so yes but it is not universal it is maybe I want

some universal in variants which are good not only for digital recognition for image understanding

so let me ask how hard do you think is 2d image understanding

so if we we can kind of intuit handwritten recognition how big of a step leap journey is it

from that if I gave you good if I solved your challenge for handwriting recognition how long

would my journey then be from that to understanding more general natural images immediate you will

understand this as soon as you will make a record because it is not for free as soon as you will

create several in variants which will help you to get the same performance that the best neural

net did using 100 times maybe more than 100 times less examples you have to have something smart

to do that and you're saying that it is invariant it is predicate because you should put some idea

how to do that but okay let me just pause maybe it's a trivial point maybe not but handwritten

recognition feels like a 2d two dimensional problem and it seems like how much complicated is the fact

that most images are projection of a three dimensional world onto a 2d plane it feels like

for a three dimensional world we still we need to start understanding common sense in order to

understand an image it's no longer visual shape and symmetry it's having to start to understand

concepts of it understand life yeah yes yes you're you're you're talking that there are

different invariant different predicates yeah and potentially much larger number you know

maybe but let's start from simple okay but you said that you know I cannot think yes about things

which I don't understand this I understand but I'm sure that I don't understand everything there

yeah yeah that's the difference say do as simple as possible but not simpler and that is exact case

with handwritten with handwritten yeah but never that's the difference between you and I I I uh

I welcome and enjoy thinking about things I completely don't understand because to me it's

a natural extension without having solved handwritten recognition to wonder how how difficult is the

the the next step of understanding 2d 3d images because ultimately while the science of intelligence

is fascinating it's also fascinating to see how that maps to the engineering of intelligence

and recognizing handwritten digits is not doesn't help you it might it may not help you with the

problem of general intelligence we don't know it'll help you a little bit we don't know it's unclear

it's unclear yeah it might very much but I would like to make a remark yes I start not from very

primitive problem make a challenge problem I start with very general problem with plateau

so you understand and and it comes from plato to to to digit recognition so so you basically took

play dough and the the world of forms and ideas and mapped and projecting into the

clearest simplest formulation of that big world you know I would say that I did not understand

plato until recently and until I consider weak convergence and then predicate and then oh this

is what plato told me so linger on that like why how do you think about this world of ideas and

world of things in play dough no it is metaphor it is it's the metaphor for sure it's a compelling

it's a poetic and a beautiful yeah but what can you but it is a way how you you you should try to

understand have attack ideas in the world so from my point of view it is very clear but it is lying

all the time people looking for that say plato's and Hegel whatever reasonable it exists whatever

exists it is reasonable I don't know what he have in mind reasonable right this philosophers again

no no no no no no no it is it is next stop of Wigner that what you might understand something

of reality it is the same plato line and then it comes suddenly to Vladimir prop

look 31 ideas 31 units and describes everything there's abstractions ideas that represent

our world and we should always try to reach into that yeah but but you should make a projection

on reality but understanding is it is abstract ideas you have in your mind several abstract ideas

which you can apply to reality and reality in this case sort of if you look at machine learning is

data example data data okay let me let me put put this on you because I'm an emotional creature

I'm not a mathematical creature like you I find compelling the idea forget this the space the sea

of functions there's also a sea of data in the world and I find compelling that there might be

like you said teacher small examples of data that are most useful for discovering good whether it's

predicates or good functions that the selection of data may be a powerful journey a useful

mechanism you know coming up with a mechanism for selecting good data might be useful too

do you find this idea of finding the right data set interesting at all or do you kind of take

the data set as a given I think that it is you know my scheme is very simple you have huge set of

functions if you will apply and you have not too many data right if you will pick up function

which describes this data you will do not very well you will randomly pick up yeah you will

have a fit here yeah it will be overfitting so you should decrease set of function from which

you're picking up one so you should go some half to admissible set of function and this what about

weak conversions so but from another point of view to to make admissible set of function

you need just a deed you just function which you will take in inner product which you will measure

property of your function

and that is how it works no I get it I get I understand it but do you the reality is but let

let this let let's think about examples you have huge set of function if you have several examples

if you just trying to keep take function which satisfies these examples you still will overfit

you need decrease you need admissible set of function absolutely but what say you have

more data than functions so sort of consider the I mean maybe not more data than functions

because that's impossible impossible but what I was trying to be poetic for a second I mean

you have a huge amount of data a huge amount of examples but amount of function can be even

bigger I understand every single there's always there's always a bigger boat full

Hilbert space I got you but okay but you don't you don't find the world of data to be an interesting

optimization space like the the optimization should be in the space of functions

creating admissible set of fun admissible set of function no you know even from the classical

this is sorry from structure risk minimization you should or you should organize function in the way

that they will be useful for you right and that is yeah but the the way you're thinking about

useful is you're given a small set small small set of function which contain function by looking yeah

but as looking for based on the empirical set of small examples yeah but that is another story I

don't touch it because I I believe I believe that this small examples it's not too small

so 60 per class that law of large numbers works I don't need uniform law the story is that in

statistics there are two law law of large numbers uniform law of large numbers so I want to be in

situation where I use law of large numbers no but not uniform law of large numbers right so 60 is

law of large it's like enough I hope no it's still need some evaluation some bounce so that's it

but idea is the following that if you trust that say this average gives you something close to

expectations so you can talk about that about this predicate and that is basis of human intelligence

right good predicates is the discovery of good predicates is the basis of no no it is

discovery of your of your understanding world of your methodology of a distance of understanding

world because you have several functions which you will apply to reality can you say that again so

your you have several functions yeah predicate but they abstract yes then you will apply them to

reality to your data and you will create in this way predicate which is useful for your task

but predicate are not related specifically to your task to this your task it is abstract functions

which being applying applied to many tasks that you might be interested in it may be many tasks I

don't know or different tasks well they should be many tasks right yeah it is like like in probe case

yes it was for free details but it's happened everywhere okay so we talked about images a

little bit but can we talk about Noam Chomsky for a second I don't know him personally well

not personally I don't know his ideas his ideas well let me just say do you think language human

language is essential to expressing ideas as Noam Chomsky believes so like language is at the core

of our formation of predicates the human language for me language and all the story of language

is very complicated I don't understand this and I am not I thought about nobody I'm not ready to

work on that because it's so huge it is not for me and I believe not for our century

the 21st century not for 21st century so you should learn something a lot of stuff

from simple tasks like digit recognition so you think okay you think digital recognition

2d image what how would you more abstractly define it digit recognition it's 2d image

symbol recognition essentially I mean I like I'm trying to get a sense sort of thinking

about it now having worked with MNIST forever how how small of a subset is this of the general

vision recognition problem and the general intelligence problem is it yeah is it a giant

subset is it not and how far away is language you know let me refer to Einstein take the simplest

problem as simple as possible but not simpler and this is challenge is simple problem

but it's simple by idea but not simple to to get it when you will do this you will find

some predicate which helps it but yeah I mean with Einstein you can you look at general relativity

but that doesn't help you with quantum mechanics that's another story you you don't have any

universal instrument yes so I'm trying to wonder if uh which space we're in whether the whether

handwritten recognition is like general relativity and then language is like quantum mechanics so

you're still gonna have to do a lot of mess to to universalize it but uh I'm trying to see

one so what's your intuition why handwritten recognition is easier than language

just I think a lot of people would agree with that but if you could elucidate sort of the

the intuition of why I don't know no I don't think in this direction I just think in

direction that this is problem which if we will solve it well

we will create

some abstract understanding of images maybe not all images I would like to talk to guys who

are doing in real images in Columbia University what kind of images unreal so real images real

images yeah what their idea is there a predicate what can be predicated I still symmetry will play

role in real life images in any real life images 2d images let's talk about 2d images because

that's what we know in neural network was created for 2d images so the people I know in

vision science for example the people study human vision yeah that they usually go to the world of

symbols and like handwritten recognition but not really it's other kinds of symbols to study

our visual perception system as far as I know not much predicate type of thinking is understood

about our vision system they did not think in this direction they don't yeah they but how do

you even begin to think in that direction that says so I would like to discuss with them yeah

because if we will be able to show that it is what working

and theoretical scheme it's not so bad so the the unfortunate so if we compare to language

language has like letters finite set of letters and a finite set of ways you can put together

those letters so it feels more amenable to kind of analysis with natural images there is so many

pixels no no no letter language is much much more complicated it's involved a lot of different stuff

it's not just understanding of very simple class of tasks I would like to see lists of tasks

where language involved yes so there's a there's a lot of nice benchmarks now on

in natural language processing from the very trivial like understanding the elements of a

sentence to question answering to more much more complicated where you talk about open domain dialogue

the natural question is with handwritten recognition is really the first step yeah of

understanding visual information right but not but but even our records show that we go in wrong

direction because we need 60 000 digits so even this first step so forget about talking about the

full journey this first step should be taking in the right direction no no in wrong direction because

60 000 is unacceptable no I'm saying it should be taken in the in the right direction because 60 000

is not acceptable it is you can talk it's great we have half percent of error and hopefully the step

from doing hand recognition using very few examples the step towards what babies do when they crawl and

understand their physical environment I know you don't know about babies if you will do from very

small examples yeah you will find principles that will be different from what we're using now

and theoretically it's more or less clear that means that you will use weak convergence not just

strong convergence do you think these principles are will naturally be human interpretable

oh yeah so like when we will be able to explain them and have a nice presentation to show what

those principles are or are they very going to be very kind of abstract kinds of functions

for example I talked yesterday about symmetry yes and they gave very simple examples the same will

be laying there you gave like a predicate of a basic for for symmetries yes for different symmetries and

you have for a degree of symmetry that is important not just symmetry existence doesn't exist a degree

of symmetry yeah for handwritten recognition no it's not for handwritten it's for any images

but I would like apply to handwritten right it's in theory it's more general okay okay

so a lot of things we've been talking about falls we've been talking about philosophy a little bit

but also about mathematics and statistics a lot of it falls into this idea a universal idea of

statistical theory of learning what is the most beautiful and sort of powerful or essential

idea you've come across even just for yourself personally in in the world of statistics or

a statistic theory of learning probably uniform convergence which we did with Alexei Chilvonakis

can you describe universal convergence you have law of large law of large numbers

so for any function expectation of function average of function conversion expectation

but if you have set of functions for any function it is true but it should converge simultaneously

for all set of functions and for for learning you need uniform convergence just convergence

is not enough because when you pick up one which gives minimum you can pick up one function which

does not converge in and it will give you the best answer for for this function

so you need the uniform convergence to guarantee learning so learning does not

really on trivial law large numbers it really on universal but

but a deal of weak convergence existing statistics for a long time but

it is interesting that as I think about myself how stupid I was 50 years I did not see weak

convergence I work on strong convergence but now I think that most powerful is weak convergence

because it makes admissible set of functions and even in all in proverbs when people try to

understand recognition about dog law looks like a dog and so on they use weak convergence

people in language they understand this but when we're trying to create artificial

intelligence we want event in different way we just consider strong convergence

arguments so reducing a set of admissible functions you think there should be

effort put into understanding the properties of weak convergence

you know in classical mathematics in Gilbert space there are only two way two

form of convergence strong and weak now we can use both that means that we did everything

and it so happened then when we use Hilbert space which is very rich space space of continuous

functions which has an integral in square so we can apply weak and strong convergence for learning

and have closed form solution so for computationally simple for me it is sign that it is right way

because you don't need any heuristic yes whatever you want but now the only what left

it is concept of what is predicate but it is not statistics by the way I like

the fact that you think the heuristics are a mess that should be removed from the system

so closed form solution is the ultimate no it so happened then when you're using

right instrument you have closed form solution

do you think intelligence human level intelligence when we create it will

will have something like a closed form solution you know I now I'm looking on

bounds which I gave bounds for convergence and when I looking for bounds I thinking

what is the most appropriate kernel for this bound would be so you know that in say

all our businesses we use radial basis function but looking on the bound I think that I start

to understand that maybe we need to make corrections to radial basis function to be closer

to work better for these bounds so I'm again trying to understand what type of kernel

have best approximation no approximation best fit to this ball

sure so there's a there's a lot of interesting work that could be done in discovering better

functions and radial basis functions for for yeah but but it still comes from you you're

you're looking to mass and trying to understand what from your own mind looking at the yeah but

I don't know then I trying to understand what what will be good for that yeah but to me there's

still a beauty again maybe I'm a descendant valenturing to heuristics to me ultimately

intelligence will be a mess of heuristics and no that's the engineering answer I guess

absolutely when when you're doing say self driving cars the great guy who will do this

it does not matter what theory behind that who has a better feeling have to apply but by the way

it is the same story about predicates because you cannot create rule for situation is much more

than you have rule for that but maybe you can have more abstract rule than it will be less

this rule it is the same story about ideas and and ideas applied to specific cases

but still you should you cannot avoid this yes of course but you should still reach for the ideas

to understand the science yeah let me kind of ask do you think neural networks or functions

can be made to reason sort of what do you think we've been talking about intelligence but this

idea of reasoning there's a there's an element of sequentially disassembling interpreting

the the images so when you think of handwritten recognition we kind of think that there will

be a single there's an input and output there's not a recurrence yeah what do you think about

sort of the idea of recurrence of going back to memory and thinking through this sort of

sequentially mangling the different representations over and over until you arrive at a conclusion

or is ultimately all that can be wrapped up into a function

no you you're suggesting that let us use this type of algorithm when I starting thinking I

first of all starting to understand what I want can I write down what I want and

then I trying to formalize and when I do that I think I have to solve this problem

and till now I did not see a situation where you need recurrence

but do you observe human beings yeah do you try to it's the imitation question right it seems

that human beings reason this kind of sequentially sort of does that inspire in you a thought that

we need to add that into our intelligent systems you're saying okay I mean you've kind of answered

saying until now I haven't seen a need for it and so because of that you don't see a reason to think

about it you know most of things I don't understand in reasoning in human it is for me too complicated

for me the most difficult part is to ask questions to good questions how it works how

people asking questions I don't know this you said that machine learning is not only about

technical things speaking of questions but it's also about philosophy so what role does philosophy

play in machine learning we talked about Plato but generally thinking in this philosophical way

does it have how does philosophy and math fit together in your mind

first ideas and then their implementation it's like predicate like

say admissible set of functions it comes together everything because

the first iteration of theory was done 50 years ago it all that this is so everything there

if you have data you can and you in your set of function is not has a not have a not big capacity

so low VC dimension you can do that you can make structural risk minimization control capacity

but you was not able to make admissible set of function good no one suddenly realized that

we did not use another idea of convergence which we can

everything comes together but those are mathematical notions philosophy plays a role of simply

saying that we should be swimming in the space of ideas let's talk what is philosophy philosophy

means understanding of life so understanding of life say people like Plato they understand

on very high abstract level of life so and whatever I doing just implementation of my

understanding of life but every new step it is very difficult for example

to find this idea that we need big convergence

was not simple for me

so that required thinking about life a little bit hard to hard to trace but

there was some thought process you know I working guys thinking about the same problem for

50 years somehow and again and again and again I trying to be honest and that is a very important

not to be very enthusiastic yeah but concentrate on whatever we was not able to achieve for example

and understand why and now I understand that because I believe in mass I believe that

in Wigner's idea but now when I see that there are only two way of convergence and we're using both

that means that we must do as well as people doing but now exactly in philosophy and what we

know about predicate what we how we understand life can we describe as a predicate I thought about that

and that is more or less obvious level of symmetry but next I have a feeling it's something about

structures but I don't know how to formulate how to measure measure of structure and all this stuff

and the guy who will solve this challenge problem then when we were looking how he did it

probably just only symmetry is not enough but something like symmetry will be there

that's absolutely symmetry will be there and level of symmetry will be there

and level of symmetry anti symmetry diagonal vertical and I I even don't know how you can

use in different direction idea of symmetry it was very general but it will be there

I think that people very sensitive to idea of symmetry but there are several ideas like symmetry

as I would like to learn but you cannot learn just thinking about that you should do challenging

problems and then analyze them why why it was we were able to solve them and then we will see

it very simple things it's not easy to find

even with talking about this every time about your your I was surprised I I try to understand

is people describe in language strong convergence mechanism for learning

I did not see I don't know but we convergence this dark story and story like that when you

will explain to kid you will use weak convergence argument it looks like it does like it does that

but when you try to formalize you just ignoring this why why 50 years from start of machine

learning and that's the role of philosophy I think I think that maybe I don't know

maybe this is theory also we should blame for that because empirical risk minimization

and all this stuff and if you read now textbooks they just about bound about empirical risk

minimization they don't looking for another problem like admissible set but on the topic of life

perhaps we you could talk in russian for a little bit what's your favorite memory from childhood

what is your favorite memory from childhood music how about can you try to answer in russian music

it was very cool when such music classical music what is your favorite

you were different composers at first it was evaldea in general was surprised that it was possible and then

when I understood Bach I was absolutely shocked by the way, I think that there are predicates

like structures in Bach, but of course, because there is just a sense of structure here and I don't think

that different elements of life are strongly divided in the sense of predicates all the way to the structure

of the structure in human relations to the structure here is how to find these here is a high level of

predicates in Bach and in life now that we're talking about Bach let's switch back to English

because I like Beethoven and Chopin so Chopin it's another music story but Bach if we talk about

predicates Bach probably has the most sort of well defined predicates that underlie you know it is

very interesting to read what critics writing about Bach which words they're using they're trying to

describe predicates and then Chopin it is very different vocabulary very different predicates

and I think that if you will make a collection of that so maybe from this you can describe

predicate for digit recognition well from Bach and Chopin no no no not from Bach and Chopin

from the critic interpretation of the music yeah when they're trying to explain you music

what they use as they use they describe high level ideas of of plateaus ideas what behind

this music that's brilliant so art is not self explanatory in some sense so you have to try to

convert it into ideas it is ill post problems when when you go from ideas to to the representation

it is easy way but when you're trying to go back it is ill post problems but nevertheless

I believe that when you're looking from that even from art you will be able to find predicate for

digit recognition that's such a fascinating and powerful notion do you ponder your own mortality

do you think about it do you fear it do you draw insight from it

about mortality no yeah are you afraid of death

not too much not too much it is pete is it I will not be able to do something which I

I think I have a feeling to do that for example I will be very happy to work with

guys theoretician from music to write this collection of description what what how they

describe music how they use what predicate and from art as well then take what is in common

and try to understand predicate which is absolute for everything and then use that for visual

recognition and see if there is a connection yeah exactly oh there's still time we got time

you got time it it it it takes years and years and years yeah it's a long way well see you've got

the patient mathematic mathematicians mind I think it could be done very quickly and very

beautifully I think it's a really elegant idea yeah but also some of many yeah you know the the

most time it is not to make this collection to understand what is the common to think about

that once again and again and again and again but I think sometimes especially just when you

say this idea now even just putting together the collection and looking at the different

sets of data language trying to interpret music criticize music and images I think there will

be sparks of ideas that will come of course again and again you'll come up with better ideas but

even just that notion is a beautiful notion I even have some example

so I have a friend

who was a specialist in Russian poetry she is a professor of Russian poetry he did not write

poems but she knows a lot of stuff she makes

a book several books and one of them is a collection of Russian poetry

she have images of Russian poetry she collect all images of Russian poetry and I ask you to do

following you have nips digit recognition and we get 100 digits or maybe less than 100 I don't

remember maybe 50 digits and try from a poetical point of view describe every image you see using

only words of images of Russian poetry and she did it and then we tried to

I call it learning using privileged information I call it privileged information

you have on two languages one language is just image of digit in another language

poetic description of this image and this is privileged information

and there is an algorithm when you're working using privileged information you're doing well

better much better so so there's something there something there and there is a

in NEC she unfortunately died the collection of digits in poetic descriptions of these digits

yeah there's some something there in that poetic description but I think that

there is an abstract ideas on the plateau level of ideas yeah that they're there that could be

discovered and music seems to be a good entry point but as soon as we start this is this challenge

problem the challenge from I listen it immediately connected to to all this stuff especially with

your talk and this podcast and I'll do whatever I can to advertise it's such a clean beautiful

Einstein like formulation of the challenge before us right let me ask another absurd question

and we talked about mortality we talked about philosophy of life what do you think is the

meaning of life what's the predicate for mysterious existence here on earth

I don't know it's very interesting how we have in Russia I don't know you know the guy Strugatsky

they are writing pictures they're thinking about hewn what what's going on and they have idea

that there are the developing two type of people common people and very smart people they just

started and these two branches of people will go in different direction very soon so that's what

they're thinking about that so the purpose of life is to create two two paths two human societies

yes simple people and more complicated which do you like best the simple people are the complicated

ones I don't know that he's just his fantasy but you know every week we have a guy who is just

writer and also a theoretic of literature and he explained how he understand

literature and human relationship how he see life and I understood that I'm just small kids

comparing to him he is very smart guy in understanding life he knows this predicate he

knows big blocks of life I am used every time when I listen to him and he just talking about

literature and I think that I was surprised so the managers in big companies most of them are

guys who study English language and English literature so why because they understand life

they understand models and among them maybe many talented critics

did just analyzing this and this is big science like property this is blocks

that's very smart it amazes me that you are and continue to be humbled by the brilliance of

others I'm very modest about myself I see so smart guys around well let me be immodest for you

you're one of the greatest mathematicians statisticians of our time it's truly an honor

thank you for talking and let's talk it is not yeah I know my limits let's let's talk again when

your challenge is taken on and solved by grad student especially when maybe musical be involved

latimer thank you so much thank you very much thanks for listening to this conversation with

latimer of apnick and thank you to our presenting sponsor cash app download it use code lex podcast

you'll get ten dollars and ten dollars will go to first an organization that inspires and educates

young minds to become science and technology innovators of tomorrow if you enjoy this podcast

subscribe on youtube give us five stars an apple podcast support it on page share on

or simply connect with me on twitter at lex freedman and now let me leave you with some words

from latimer of apnick when solving a problem of interest do not solve a more general problem

as an intermediate step thank you for listening I hope to see you next time