The following is a conversation with Melanie Mitchell.

She's a professor of computer science

at Portland State University

and an external professor at Santa Fe Institute.

She has worked on and written about artificial intelligence

from fascinating perspectives,

including adaptive complex systems, genetic algorithms,

and the copycat cognitive architecture,

which places the process of analogy making

at the core of human cognition.

From her doctoral work with her advisors,

Douglas Hofstadter and John Holland to today,

she has contributed a lot of important ideas

to the field of AI, including her recent book,

simply called Artificial Intelligence,

a guide for thinking humans.

This is the Artificial Intelligence podcast.

If you enjoy it, subscribe on YouTube,

give it five stars on Apple Podcast,

support it on Patreon,

or simply connect with me on Twitter

at Lex Freedman, spelled F R I D M A N.

I recently started doing ads

at the end of the introduction.

I'll do one or two minutes after introducing the episode

and never any ads in the middle

that can break the flow of the conversation.

I hope that works for you.

It doesn't hurt the listening experience.

I provide timestamps for the start of the conversation,

but it helps if you listen to the ad

and support this podcast

by trying out the product or service being advertised.

This show is presented by Cash App,

the number one finance app in the App Store.

I personally use Cash App to send money to friends,

but you can also use it to buy, sell,

and deposit Bitcoin in just seconds.

Cash App also has a new investing feature.

You can buy fractions of a stock, say $1 worth,

no matter what the stock price is.

Brokeria services are provided by Cash App investing,

a subsidiary of Square and a member of SIPC.

I'm excited to be working with Cash App

to support one of my favorite organizations called First,

best known for their first robotics and Lego competitions.

They educate and inspire hundreds of thousands of students

in over 110 countries

and have a perfect rating on Charity Navigator,

which means that donated money is used

to maximum effectiveness.

When you get Cash App from the App Store or Google Play

and use code LEX Podcast, you'll get $10

and Cash App will also donate $10 to First,

which again is an organization

that I've personally seen inspire girls and boys

to dream of engineering a better world.

And now here's my conversation with Melanie Mitchell.

The name of your new book is Artificial Intelligence,

subtitle, A Guide for Thinking Humans.

The name of this podcast is Artificial Intelligence.

So let me take a step back and ask the old

Shakespeare question about roses.

What do you think of the term artificial intelligence

for our big and complicated and interesting field?

I'm not crazy about the term.

I think it has a few problems

because it means so many different things

to different people.

And intelligence is one of those words

that isn't very clearly defined either.

There's so many different kinds of intelligence,

degrees of intelligence, approaches to intelligence.

John McCarthy was the one who came up

with the term artificial intelligence.

And from what I read, he called it that

to differentiate it from cybernetics,

which was another related movement at the time.

And he later regretted calling it artificial intelligence.

Herbert Simon was pushing for calling it

complex information processing,

which got mixed, but probably is equally vague, I guess.

Is it the intelligence or the artificial

in terms of words that's most problematic, would you say?

Yeah, I think it's a little of both.

But it has some good size because

I personally was attracted to the field

because I was interested in phenomenon of intelligence.

And if it was called complex information processing,

maybe I'd be doing something wholly different now.

What do you think of, I've heard that term

used cognitive systems, for example.

So using cognitive?

Yeah, I mean cognitive has certain associations with it

and people like to separate things like cognition

and perception, which I don't actually think are separate,

but often people talk about cognition

as being different from sort of other aspects of intelligence.

It's sort of higher level.

So to you, cognition is this broad,

beautiful mess of things that encompasses the whole thing.

Memory, perception. Yeah, I think it's hard

to draw lines like that.

When I was coming out of grad school in 1990,

which is when I graduated,

that was during one of the AI winters.

And I was advised to not put AI,

artificial intelligence on my CV,

but instead call it intelligent systems.

So that was kind of a euphemism, I guess.

What about to stick briefly on terms and words,

the idea of artificial general intelligence?

Or like Jan Lacoon prefers human level intelligence.

Sort of starting to talk about ideas

that achieve higher and higher levels of intelligence

and some how artificial intelligence seems to be

a term used more for the narrow,

very specific applications of AI

and sort of what set of terms appeal to you

to describe the thing that perhaps was strive to create.

People have been struggling with this

for the whole history of the field

and defining exactly what it is that we're talking about.

You know, John Searle had this distinction

between strong AI and weak AI.

And weak AI could be general AI,

but his idea was strong AI was the view

that a machine is actually thinking that

as opposed to simulating thinking

or carrying out processes that we would call intelligent.

At a high level, if you look at the founding

of the field of McCarthy and Searle and so on,

are we closer to having a better sense of that line

between narrow, weak AI and strong AI?

Yes, I think we're closer to having a better idea

of what that line is.

Early on, for example,

a lot of people thought that playing chess would be,

you couldn't play chess if you didn't have

sort of general human level intelligence.

And of course, once computers were able to play chess

better than humans, that revised that view.

And people said, okay, well, maybe now we have to revise

what we think of intelligence as.

And so that's kind of been a theme throughout the history

of the field is that once a machine can do some task,

we then have to look back and say, oh, well,

that changes my understanding of what intelligence is,

because I don't think that machine is intelligent.

At least that's not what I want to call intelligence.

Do you think that line moves forever?

Or will we eventually really feel as a civilization

like we crossed the line if it's possible?

It's hard to predict, but I don't see any reason

why we couldn't, in principle,

create something that we would consider intelligent.

I don't know how we will know for sure.

Maybe our own view of what intelligence is

will be refined more and more

until we finally figure out what we mean

when we talk about it.

But I think eventually we will create machines

in a sense that have intelligence.

They may not be the kinds of machines we have now.

And one of the things that that's going to produce

is making us sort of understand our own machine

like qualities that we, in a sense,

are mechanical in the sense that like cells,

cells are kind of mechanical.

They have algorithms they process information by.

And somehow out of this mass of cells,

we get this emergent property that we call intelligence.

But underlying it is really just cellular processing

and lots and lots and lots of it.

Do you think it's possible to create intelligence

without understanding our own mind?

You said in that process we'll understand more and more.

But do you think it's possible to sort of create

without really fully understanding from a mechanistic perspective,

sort of from a functional perspective,

how our mysterious mind works?

If I had to bet on it, I would say,

no, we do have to understand our own minds,

at least to some significant extent.

But I think that's a really big open question.

I've been very surprised at how far kind of brute force approaches

based on, say, big data and huge networks can take us.

I wouldn't have expected that.

And they have nothing to do with the way our minds work.

So that's been surprising to me, so it could be wrong.

To explore the psychological and the philosophical,

do you think we're okay as a species

with something that's more intelligent than us?

Do you think perhaps the reason we're pushing that line

further and further is we're afraid of acknowledging

that there's something stronger, better, smarter than us humans?

Well, I'm not sure we can define intelligence that way

because, you know, smarter than is with respect to what?

You know, computers are already smarter than us in some areas.

They can multiply much better than we can.

They can figure out driving routes to take much faster

and better than we can.

They have a lot more information to draw on.

They know about, you know, traffic conditions and all that stuff.

So for any given particular task,

sometimes computers are much better than we are,

and we're totally happy with that, right?

I'm totally happy with that.

It doesn't bother me at all.

I guess the question is, you know, which things

about our intelligence would we feel very sad or upset

that machines had been able to recreate?

So in the book, I talk about my former Ph.D.

advisor, Douglas Hofstadter,

who encountered a music generation program,

and that was really the line for him,

that if a machine could create beautiful music,

that would be terrifying for him,

because that is something he feels is really at the core

of what it is to be human,

creating beautiful music, art, literature.

I, you know, I don't think...

He doesn't like the fact that machines can recognize

spoken language really well.

He personally doesn't like using speech recognition,

but I don't think it bothers him to his core,

because it's like, okay, that's not at the core of humanity.

But it may be different for every person.

What really they feel would usurp their humanity.

And I think maybe it's a generational thing also.

Maybe our children or our children's children

will be adapted, they'll adapt to these new devices

that can do all these tasks and say,

yes, this thing is smarter than me in all these areas,

but that's great because it helps me.

Looking at the broad history of our species,

why do you think so many humans have dreamed

of creating artificial life and artificial intelligence

throughout the history of our civilization?

Not just this century or the 20th century,

but really throughout many centuries that preceded it?

That's a really good question,

and I have wondered about that.

Because I myself, you know, was driven by curiosity

about my own thought processes

and thought it would be fantastic to be able to get a computer

to mimic some of my thought processes.

I'm not sure why we're so driven.

I think we want to understand ourselves better,

and we also want machines to do things for us.

But I don't know, there's something more to it

because it's so deep in the kind of mythology

or the ethos of our species.

And I don't think other species have this drive,

so I don't know.

If you were to sort of psychoanalyze yourself

in your own interest in AI,

what excites you about creating intelligence?

You said understanding our own cells?

Yeah, I think that's what drives me particularly.

I'm really interested in human intelligence,

but I'm also interested in the sort of the phenomenon

of intelligence more generally.

And I don't think humans are the only thing with intelligence,

you know, or even animals.

But I think intelligence is a concept

that encompasses a lot of complex systems.

And if you think of things like insect colonies

or cellular processes or the immune system

or all kinds of different biological

or even societal processes have,

as an emergent property,

some aspects of what we would call intelligence.

You know, they have memory, they process information,

they have goals, they accomplish their goals, et cetera.

And to me, the question of what is this thing

we're talking about here

was really fascinating to me

and exploring it using computers

seemed to be a good way to approach the question.

So do you think kind of of intelligence,

do you think of our universe as a kind of hierarchy

of complex systems and intelligence

as just the property of any,

you can look at any level

and every level has some aspect of intelligence.

So we're just like one little speck

in that giant hierarchy of complex systems.

I don't know if I would say any system like that

has intelligence, but I guess what I want to,

I don't have a good enough definition of intelligence

to say that.

So let me do sort of multiple choice, I guess.

So you said ant colonies.

So our ant colonies intelligent are the bacteria

in our body intelligent

and then going to the physics world,

molecules and the behavior at the quantum level

of electrons and so on.

Are those kinds of systems, do they possess intelligence?

Like where is the line that feels compelling to you?

I don't know.

I mean, I think intelligence is a continuum

and I think that the ability to in some sense

have intention, have a goal,

have some kind of self awareness is part of it.

So I'm not sure if, you know,

it's hard to know where to draw that line.

I think that's kind of a mystery,

but I wouldn't say that say that, you know,

the planets orbiting the sun is an intelligent system.

I mean, I would find that maybe not the right term

to describe that.

And this is, you know, there's all this debate in the field

of like, what's the right way to define intelligence?

What's the right way to model intelligence?

Should we think about computation?

Should we think about dynamics?

Should we think about, you know, free energy

and all of that stuff?

And I think that it's a fantastic time to be in the field

because there's so many questions

and so much we don't understand.

There's so much work to do.

So are we the most special kind of intelligence in this kind of,

you said there's a bunch of different elements

and characteristics of intelligence systems and colonies.

Is human intelligence the thing in our brain?

Is that the most interesting kind of intelligence

in this continuum?

Well, it's interesting to us because it is us.

I mean, interesting to me, yes.

And because I'm part of, you know, human.

But to understanding the fundamentals of intelligence

what I'm getting at, is studying the human,

is sort of, if everything we've talked about,

what you talked about in your book,

what just the AI field, this notion,

yes, it's hard to define,

but it's usually talking about something

that's very akin to human intelligence.

Yeah.

To me, it is the most interesting

because it's the most complex, I think.

It's the most self aware.

It's the only system at least that I know of

that reflects on its own intelligence.

And you talk about the history of AI

and us in terms of creating artificial intelligence

being terrible at predicting the future with AI

with tech in general.

So why do you think we're so bad at predicting the future?

Are we hopelessly bad?

So no matter what, whether it's this decade

or the next few decades, every time we make a prediction,

there's just no way of doing it well

or as the field matures, we'll be better and better at it.

I believe as the field matures, we will be better.

And I think the reason that we've had so much trouble

is that we have so little understanding

of our own intelligence.

So there's the famous story about Marvin Minsky

assigning computer vision as a summer project

to his undergrad students.

And I believe that's actually a true story.

Yeah, there's a write up on it, everyone should read.

I think it's like a proposal that describes everything

that should be done in that project.

It's hilarious because, I mean, you can explain it,

but for my recollection, it describes basically

all the fundamental problems of computer vision,

many of which still haven't been solved.

Yeah, and I don't know how far they really expected to get,

but I think that, and they're really, you know,

Marvin Minsky is a super smart guy

and very sophisticated thinker,

but I think that no one really understands

or understood, still doesn't understand

how complicated, how complex the things that we do are

because they're so invisible to us, you know, to us.

Vision, being able to look out at the world

and describe what we see, that's just immediate.

It feels like it's no work at all.

So it didn't seem like it would be that hard,

but there's so much going on unconsciously,

sort of invisible to us that I think we overestimate

how easy it will be to get computers to do it.

And so for me to ask an unfair question,

you've done research, you've thought about

many different branches of AI through this book,

widespread, looking at where AI has been,

where it is today.

If you were to make a prediction,

how many years from now would we as a society

create something that you would say

achieved human level intelligence

or superhuman level intelligence?

That is an unfair question.

A prediction that will most likely be wrong,

but it's just your notion because...

Okay, I'll say more than a hundred years.

More than a hundred years.

And I quoted somebody in my book who said

that human level intelligence is a hundred Nobel prizes away,

which I like because it's a nice way to sort of...

It's a nice unit for prediction.

And it's like that many fantastic discoveries

have to be made.

And of course, there's no Nobel prize in AI,

not yet at least.

If you look at that hundred years,

your sense is really the journey to intelligence

has to go through something more complicated

that's akin to our own cognitive systems,

understanding them, being able to create them

in artificial systems,

as opposed to sort of taking the machine learning approaches

of today and really scaling them and...

Scaling them and scaling them exponentially

with both compute and hardware and data.

That would be my guess.

I think that in the sort of going along in the narrow AI

that the current approaches will get better,

I think there's some fundamental limits

to how far they're going to get.

I might be wrong, but that's what I think.

And there's some fundamental weaknesses that they have

that I talk about in the book

that just comes from this approach of supervised learning,

requiring sort of feedforward networks and so on.

I don't think it's a sustainable approach

to understanding the world.

I'm personally torn on it.

Everything you write about in the book

instead of what we're talking about now,

I agree with you, but I'm more and more,

depending on the day,

first of all, I'm deeply surprised by the success

of machine learning and deep learning in general

from the very beginning.

It's really been my main focus of work.

I'm just surprised how far it gets.

And I also think we're really early on in these efforts

of these narrow AI.

So I think there will be a lot of surprise

of how far it gets.

I think we'll be extremely impressed.

My sense is everything I've seen so far,

and we'll talk about autonomous driving and so on,

I think we can get really far.

But I also have a sense that we will discover,

just like you said,

even though we'll get really far,

in order to create something like our own intelligence,

it's actually much farther than we realize.

I think these methods are a lot more powerful

than people give them credit for, actually.

So, of course, there's the media hype,

but I think there's a lot of researchers in the community,

especially not undergrads,

but people who have been in AI,

they're skeptical about how far deep learning can get,

and I'm more and more thinking that

it can actually get farther than we realize.

It's certainly possible.

One thing that surprised me when I was writing the book

is how far apart different people are in the field are

on their opinion of how far the field has come,

and what has accomplished,

and what's going to happen next.

What's your sense of the different,

who are the different people, groups,

mindsets, thoughts in the community

about where AI is today?

Yeah, they're all over the place.

So, there's kind of the singularity transhumanism group,

I don't know exactly how to characterize that approach,

which is the sort of exponential progress.

We're on the sort of almost at the hugely accelerating part

of the exponential,

and in the next 30 years,

we're going to see super intelligent AI and all that,

and we'll be able to upload our brains and that.

So, there's that kind of extreme view

that I think most people who work in AI don't have.

They disagree with that.

But there are people who are,

maybe aren't singularity people,

but they do think that the current approach

of deep learning is going to scale

and is going to kind of go all the way basically

and take us to true AI or human level AI

or whatever you want to call it.

And there's quite a few of them.

And a lot of them,

like a lot of the people I met who work at big tech companies

in AI groups kind of have this view

that we're really not that far, you know.

Just to link on that point,

if I can take, as an example, like Yann LeCun,

I don't know if you know about his work,

and so it hurts viewpoints on this.

I do.

He believes that there's a bunch of breakthroughs,

like fundamental, like Nobel Prizes that are needed still.

But I think he thinks those breakthroughs

will be built on top of deep learning.

And then there's some people who think

we need to kind of put deep learning to the side a little bit

as just one module that's helpful

in the bigger cognitive framework.

Right.

So I think, so what I understand, Yann LeCun is rightly saying

supervised learning is not sustainable.

We have to figure out how to do unsupervised learning,

that that's going to be the key.

And, you know, I think that's probably true.

I think unsupervised learning is going to be harder than people think.

I mean, the way that we humans do it.

Then there's the opposing view, you know,

that there's the Gary Marcus kind of hybrid view

where deep learning is one part,

but we need to bring back kind of the symbolic approaches

and combine them.

Of course, no one knows how to do that very well.

Which is the more important part to emphasize

and how do they fit together?

What's the foundation?

What's the thing that's on top?

What's the cake?

What's the icing?

Right.

Then there's people pushing different things.

There's the causality people who say, you know,

deep learning as it's formulated today

completely lacks any notion of causality.

And that's dooms it.

And therefore, we have to somehow give it

some kind of notion of causality.

There's a lot of push from the more cognitive science crowd saying

we have to look at developmental learning.

We have to look at how babies learn.

We have to look at intuitive physics.

All these things we know about physics

and as somebody kind of quipped,

we also have to teach machines intuitive metaphysics,

which means like objects exist.

Causality exists.

These things that maybe we're born with, I don't know,

that machines don't have any of that.

They look at a group of pixels

and maybe they get 10 million examples,

but they can't necessarily learn

that there are objects in the world.

So there's just a lot of pieces of the puzzle

that people are promoting

and with different opinions of like how important they are

and how close we are to being able to put them all together

to create general intelligence.

Looking at this broad field, what do you take away from it?

Who is the most impressive?

Is it the cognitive folks?

Is it Gary Marcus camp?

The on camp unsupervised and they're self supervised.

There's the supervisors and then there's the engineers

who are actually building systems.

You have sort of the Andre Karpathy at Tesla

building actual, you know, it's not philosophy,

it's real like systems that operate in the real world.

What do you take away from all this beautiful variety?

I don't know if these different views

are not necessarily mutually exclusive.

And I think people like Jan Lacune

agrees with the developmental psychology,

causality, intuitive physics, et cetera.

But he still thinks that it's learning,

like end to end learning is the way to go.

We'll take this perhaps all the way.

Yeah, and that we don't need, there's no sort of innate stuff

that has to get built in.

This is, you know, it's because it's a hard problem.

I personally, you know, I'm very sympathetic

to the cognitive science side

because that's kind of where I came in to the field.

I've become more and more sort of an embodiment

adherent saying that, you know, without having a body,

it's going to be very hard to learn

what we need to learn about the world.

That's definitely something I'd love to talk about

in a little bit.

To step into the cognitive world,

then if you don't mind,

because you've done so many interesting things,

if you look to Copycat, taking a couple of decades,

step back, you would Douglas Hofstadter

and others have created and developed Copycat

more than 30 years ago.

That's painful to hear.

What is it? What is Copycat?

It's a program that makes analogies

in an idealized domain, idealized world

of letter strings.

So as you say, 30 years ago, wow.

So I started working on it when I started grad school

in 1984.

Wow.

Dates me.

And it's based on Doug Hofstadter's ideas

about that analogy is really a core aspect of thinking.

I remember he has a really nice quote

in the book by himself

and Emmanuel Sander called Surfaces and Essences.

I don't know if you've seen that book,

but it's about analogy.

He says, without concepts, there can be no thought

and without analogies, there can be no concepts.

So the view is that analogy is not just this kind

of reasoning technique where we go, you know,

shoe is to foot as glove is to what?

These kinds of things that we have on IQ tests or whatever.

But that it's much deeper.

It's much more pervasive in everything we do

in our language, our thinking, our perception.

So he had a view that was a very active perception idea.

So the idea was that instead of having kind of a passive network

in which you have input that's being processed

through these feedforward layers

and then there's an output at the end,

that perception is really a dynamic process.

You know, where like our eyes are moving around

getting information and that information is feeding back

to what we look at next, influences what we look at next

and how we look at it.

And so Copycat was trying to do that kind of simulate

that kind of idea where you have these agents.

It's kind of an agent based system and you have these agents

that are picking things to look at and deciding

whether they were interesting or not

because they should be looked at more

and that would influence other agents.

How did they interact?

So they interacted through this global kind of what we call

the workspace.

So it's actually inspired by the old blackboard systems

where you would have agents that post information

on a blackboard, a common blackboard.

This is like very old fashioned AI.

Is that we're talking about like in physical space?

Is this a computer program?

It's a computer program.

Agents posting concepts on a blackboard?

Yeah, we called it a workspace.

And the workspace is a data structure.

The agents are little pieces of code

that you could think of them as little detectors

or little filters that say,

I'm going to pick this place to look

and I'm going to look for a certain thing.

And is this the thing I think is important?

Is it there?

So it's almost like a convolution in a way

except a little bit more general

and then highlighting it in the workspace.

Once it's in the workspace,

how do the things that are highlighted relate to each other?

So there's different kinds of agents

that can build connections between different things.

So just to give you a concrete example,

what Copycat did was it made analogies

between strings of letters.

So here's an example.

ABC changes to ABD.

What does IJK change to?

And the program had some prior knowledge

about the alphabet.

It knew the sequence of the alphabet.

It had a concept of letter or successor of letter.

It had concepts of sameness.

So it has some innate things programmed in.

But then it could do things like say,

discover that ABC is a group of letters

in succession.

And then an agent can mark that.

So the idea that there could be a sequence of letters,

is that a new concept that's formed

or that's a concept that's innate?

That's a concept that's innate.

So can you form new concepts or are all concepts innate?

So in this program,

all the concepts of the program were innate.

Obviously, that limits it quite a bit.

But what we were trying to do is say,

suppose you have some innate concepts,

how do you flexibly apply them to new situations?

And how do you make analogies?

Let's step back for a second.

So I really like that quote that you said,

without concepts, there could be no thought.

And without analogies, there could be no concepts.

In a Santa Fe presentation,

you said that it should be one of the mantras of AI.

Yes.

And that you also yourself said,

how to form and fluidly use concepts

is the most important open problem in AI.

Yes.

How to form and fluidly use concepts

is the most important open problem in AI.

So what is a concept and what is an analogy?

A concept is in some sense a fundamental unit of thought.

So say we have a concept of a dog, okay?

And a concept is embedded in a whole space of concepts

so that there's certain concepts that are closer to it

or farther away from it.

Are these concepts, are they really like fundamental,

like we mentioned innate, almost like axiomatic,

like very basic and then there's other stuff built on top of it?

Or does this include everything?

Are there complicated...

You can certainly form new concepts.

Right, I guess that's the question I'm asking.

Can you form new concepts that are complex combinations

of other concepts?

Yes, absolutely.

And that's kind of what we do in learning.

And then what's the role of analogies in that structure?

So analogy is when you recognize that one situation

is essentially the same as another situation

and essentially is kind of the keyword there

because it's not the same.

So if I say, last week I did a podcast interview

in actually like three days ago in Washington DC

and that situation was very similar to this situation

although it wasn't exactly the same.

It was a different person sitting across from me.

We had different kinds of microphones.

The questions were different.

The building was different.

There's all kinds of different things,

but really it was analogous.

Or I can say, so doing a podcast interview,

that's kind of a concept, it's a new concept.

I never had that concept before this year essentially.

And I can make an analogy with it

like being interviewed for a news article in a newspaper.

And I can say, well, you kind of play the same role

that the newspaper reporter played.

It's not exactly the same

because maybe they actually emailed me

some written questions rather than talking.

And the written questions are analogous

to your spoken questions.

There's just all kinds of similarities.

And this somehow probably connects to conversations

you have over Thanksgiving dinner,

just general conversations.

There's like a thread you can probably take

that just stretches out in all aspects of life

that connect to this podcast.

I mean, conversations between humans.

Sure. And if I go and tell a friend of mine

about this podcast interview,

my friend might say, oh, the same thing happened to me.

Let's say you ask me some really hard question

and I have trouble answering it.

My friend could say, the same thing happened to me,

but it wasn't a podcast interview.

It was a completely different situation.

And yet my friend is seeing essentially the same thing.

You know, we say that very fluidly.

The same thing happened to me.

Essentially the same thing.

But we don't even say that, right?

They would imply it, yes.

Yeah. And the view that kind of went into, say,

Copycat, that whole thing is that act of saying

the same thing happened to me is making an analogy.

And in some sense, that's what underlies

all of our concepts.

Why do you think analogy making that you're describing

is so fundamental to cognition?

It seems like it's the main element action

of what we think of as cognition.

Yeah. So it can be argued that all of this

generalization we do of concepts

and recognizing concepts in different situations

is done by analogy.

Every time I'm recognizing that, say,

you're a person, that's by analogy

because I have this concept of what person is

and I'm applying it to you.

And every time I recognize a new situation,

like one of the things I talked about in the book

was the concept of walking a dog,

that that's actually making an analogy

because all of that, you know, the details are very different.

So reasoning could be reduced

on to sensory analogy making.

So all the things we think of as like,

yeah, like you said, perception.

So what's perception is taking raw sensory input

and it's somehow integrating into our understanding

of the world, updating the understanding

and all of that has just this giant mess of analogies

that are being made.

I think so, yeah.

If you just linger on it a little bit,

what do you think it takes to engineer a process like that

for us in our artificial systems?

We need to understand better, I think,

how we do it, how humans do it.

And it comes down to internal models, I think.

You know, people talk a lot about mental models,

that concepts are mental models,

that I can, in my head,

I can do a simulation of a situation like walking a dog

and that there's some work in psychology

that promotes this idea that all of concepts

are really mental simulations,

that whenever you encounter a concept

or a situation in the world,

or you read about it or whatever,

you do some kind of mental simulation

that allows you to predict what's going to happen

to develop expectations of what's going to happen.

So that's the kind of structure I think we need

is that kind of mental model that,

in our brains, somehow these mental models are very much interconnected.

Again, so a lot of stuff we're talking about

are essentially open problems, right?

So if I ask a question,

I don't mean that you would know the answer,

only just hypothesizing,

but how big do you think is the network,

graph, data structure of concepts that's in our head?

Like if we're trying to build that ourselves,

we take it, that's one of the things we take for granted,

we think, I mean, that's why we take common sense for granted.

We think common sense is trivial,

but how big of a thing of concepts is

that underlies what we think of as common sense, for example?

Yeah, I don't know,

and I don't even know what units to measure it in.

You say how big is it?

It's perfectly put, right?

But it's really hard to know.

We have, what, 100 billion neurons or something,

I don't know,

and they're connected via trillions of synapses,

and there's all this chemical processing going on.

There's just a lot of capacity for stuff,

and their information's encoded in different ways in the brain,

it's encoded in chemical interactions,

it's encoded in electric,

like firing and firing rates,

and nobody really knows how it's encoded,

but it just seems like there's a huge amount of capacity.

So I think it's huge, it's just enormous,

and it's amazing how much stuff we know.

Yeah.

But we know, and not just know, like facts,

but it's all integrated into this thing

that we can make analogies with.

There's a dream of semantic web,

and there's a lot of dreams from expert systems

of building giant knowledge bases.

Do you see a hope for these kinds of approaches

of building, of converting Wikipedia

into something that could be used in analogy making?

Sure.

And I think people have made some progress along those lines.

I mean, people have been working on this for a long time.

But the problem is, and this, I think,

is the problem of common sense,

like people have been trying to get these common sense networks

here at MIT.

There's this concept net project, right?

But the problem is that, as I said,

most of the knowledge that we have is invisible to us.

It's not in Wikipedia.

It's very basic things about, you know,

intuitive physics, intuitive psychology,

intuitive metaphysics, all that stuff.

If you were to create a website that's described

intuitive physics, intuitive psychology,

would it be bigger or smaller than Wikipedia?

What do you think?

I guess described to whom?

I'm sorry, but...

No, it's really good.

Exactly right, yeah.

That's a hard question because, you know,

how do you represent that knowledge is the question, right?

I can certainly write down F equals MA

and Newton's laws and a lot of physics

can be deduced from that.

But that's probably not the best representation

of that knowledge for doing the kinds of reasoning

we want a machine to do.

So, I don't know, it's impossible to say now.

And people, you know, the projects,

like there's a famous psych project, right,

that Douglas Lennart did that was trying...

I think it's still going.

I think it's still going, and the idea was to try

and encode all of common sense knowledge,

including all this invisible knowledge

in some kind of logical representation.

And it just never, I think, could do any of the things

that he was hoping it could do

because that's just the wrong approach.

Of course, that's what they always say, you know,

and then the history books will say,

well, the psych project finally found a breakthrough

in 2058 or something.

So much progress has been made in just a few decades

that who knows what the next breakthroughs will be.

It could be.

It's certainly a compelling notion

what the psych project stands for.

I think Lennart was one of the earliest people to say

common sense is what we need.

Important.

That's what we need.

All this, like, expert system stuff,

that is not going to get you to AI.

You need common sense.

And he basically gave up his whole academic career

to go pursue that.

And I totally admire that,

but I think that the approach itself will not...

What do you think is wrong with the approach?

What kind of approach might be successful?

Well, I knew that.

Again, nobody knows the answer, right?

If I knew that, you know, one of my talks,

one of the people in the audience,

this is a public lecture,

one of the people in the audience said,

what AI companies are you investing in?

Investment advice?

Okay.

I'm a college professor for one thing,

so I don't have a lot of extra funds to invest,

but also, like, no one knows what's going to work in AI, right?

That's the problem.

Let me ask another impossible question

in case you have a sense.

In terms of data structures that will store

this kind of information,

do you think they've been invented yet,

both in hardware and software?

Or is something else needs to be...

Are we totally...

I think something else has to be invented.

That's my guess.

Is the breakthroughs that's most promising?

Would that be in hardware or in software?

Do you think we can get far with the current computers?

Or do we need to do something...

That's what you were saying.

I don't know if turing computation is going to be sufficient.

Probably.

I would guess it will.

I don't see any reason why we need anything else,

but so in that sense,

we have invented the hardware we need,

but we just need to make it faster and bigger.

We need to figure out the right algorithms

and the right architecture.

Turing...

That's a very mathematical notion.

When we have to build intelligence,

it's not an engineering notion

where you throw all that stuff.

I guess it is a question...

People have brought up this question.

When you asked about...

Is our current hardware...

Will our current hardware work?

Well, turing computation says that

our current hardware is, in principle,

a turing machine.

All we have to do is make it faster and bigger.

But there have been people like Roger Penrose,

if you might remember, that he said

turing machines cannot produce intelligence

because intelligence requires continuous valued numbers.

That was my reading of his argument

and quantum mechanics and whatever.

But I don't see any evidence for that,

that we need new computation paradigms.

But I don't think we're going to be able

to scale up our current approaches

to programming these computers.

What is your hope for approaches like Copycat

or other cognitive architectures?

I've talked to the creator of SOAR, for example.

I've used Act R myself.

I don't know if you're familiar with that.

What do you think is...

What's your hope of approaches like that

in helping develop systems of greater and greater intelligence

in the coming decades?

Well, that's what I'm working on now,

is trying to take some of those ideas and extending it.

So I think there are some really promising approaches

that are going on now that have to do with

more active generative models.

So this is the idea of this simulation

in your head of a concept.

If you want to, when you're perceiving a new situation,

you have some simulations in your head.

Those are generative models.

They're generating your expectations.

They're generating predictions.

So that's part of a perception.

You have a method model that generates a prediction,

then you compare it with...

Yeah.

And then the difference...

And you also...

That generative model is telling you where to look

and what to look at and what to pay attention to.

And I think it affects your perception.

It's not that just you compare it with your perception.

It becomes your perception in a way.

It's kind of a mixture of the bottom up information

coming from the world and your top down model

being imposed on the world is what becomes your perception.

So your hope is something like that

can improve perception systems

and that they can understand things better.

Yes.

Understand things.

Yes.

So what's the step?

What's the analogy making step there?

Well, the idea is that you have this pretty complicated

conceptual space.

You can talk about a semantic network or something like that

with these different kinds of concept models

in your brain that are connected.

So let's take the example of walking a dog.

We were talking about that.

Let's say I see someone out on the street walking a cat.

Some people walk their cats, I guess.

It seems like a bad idea, but...

Yeah.

Good thing.

So there's connections between my model of a dog

and model of a cat.

And I can immediately see the analogy

that those are analogous situations.

But I can also see the differences

and that tells me what to expect.

So also, I have a new situation.

So another example with the walking the dog thing is

sometimes I see people riding their bikes

holding a leash and the dog's running alongside.

Okay, so I recognize that as kind of a dog walking situation

even though the person's not walking

and the dog's not walking.

Because I have these models that say,

okay, riding a bike is sort of similar to walking

or it's connected.

It's a means of transportation.

But because they have their dog there,

I assume they're not going to work,

but they're going out for exercise.

And these analogies help me to figure out

kind of what's going on, what's likely.

But sort of these analogies are very human interpretable.

So that's that kind of space.

And then you look at something like

the current deep learning approaches,

they kind of help you to take raw sensory information

and to sort of automatically build up hierarchies

of what you can even call them concepts.

They're just not human interpretable concepts.

What's the link here?

Do you hope it's sort of the hybrid system question?

How do you think that two can start to meet each other?

What's the value of learning in these systems

of forming of analogy making?

The original goal of deep learning

in at least visual perception was that

you would get the system to learn to extract features

at these different levels of complexity.

So maybe edge detection and that would lead

into learning simple combinations of edges

and then more complex shapes and then whole objects

or faces.

And this was based on the ideas of the neuroscientists

Hubel and Weasel who had seen laid out this kind

of structure and brain.

And I think that's right to some extent.

Of course, people have come found that the whole story

is a little more complex than that and the brain

of course always is and there's a lot of feedback.

So I see that as absolutely a good brain inspired approach

to some aspects of perception.

But one thing that it's lacking, for example,

is all of that feedback, which is extremely important.

The interactive element that you mentioned.

The expectation, the conceptual level.

Going back and forth with the expectation

and the perception and just going back and forth.

So that is extremely important.

And one thing about deep neural networks

is that in a given situation, they're trained,

they get these weights and everything.

And then now I give them a new image, let's say.

They treat every part of the image in the same way.

They apply the same filters at each layer

to all parts of the image.

There's no feedback to say like,

oh, this part of the image is irrelevant.

I shouldn't care about this part of the image

or this part of the image is the most important part.

And that's kind of what we humans are able to do

because we have these conceptual expectations.

There's, by the way, a little bit of work in that.

There's certainly a lot more in what's called attention

in natural language processing knowledge.

That's exceptionally powerful.

And it's a very, just as you say, it's a really powerful idea.

But again, in machine learning,

it all operates in an automated way.

That's not human.

It's not dynamic.

In the sense that as a perception of a new example

is being processed, those attention's weights don't change.

There's a kind of notion that there's not a memory.

So you're not aggregating the idea of this mental model.

That seems to be a fundamental idea.

There's not a really powerful...

I mean, there's some stuff with memory,

but there's not a powerful way to represent the world

in some sort of way that's deeper than...

I mean, it's so difficult because neural networks do represent the world.

They do have a mental model, right?

But it just seems to be shallow.

It's hard to criticize them at the fundamental level.

To me, at least.

It's easy to criticize them.

Well, look, like exactly what you're saying,

mental models sort of almost put a psychology hat on,

say, look, these networks are clearly not able to achieve

what we humans do with forming mental models,

the analogy making so on.

But that doesn't mean that they fundamentally cannot do that.

It's very difficult to say that, at least to me.

Do you have a notion that the learning approaches really...

I mean, they're going to...

Not only are they limited today,

but they will forever be limited in being able to construct such mental models.

I think the idea of the dynamic perception is key here,

the idea that moving your eyes around and getting feedback.

And that's something that...

There's been some models like that.

There's certainly recurrent neural networks

that operate over several time steps.

But the problem is that the actual recurrence is...

Basically, the feedback is, at the next time step,

is the entire hidden state of the network,

which is...

And it turns out that that doesn't work very well.

The thing I'm saying is, mathematically speaking,

it has the information in that recurrence to capture everything.

It just doesn't seem to work.

Yeah, right.

It's the same Turing machine question, right?

Yeah, maybe theoretically, computers...

Anything that's a universal Turing machine can be intelligent.

But practically, the architecture might be a very specific kind of architecture

to be able to create it.

I guess to ask almost the same question again is,

how big of a role do you think deep learning will play

or needs to play in this, in perception?

I think that deep learning, as it currently exists,

that kind of thing will play some role.

But I think that there's a lot more going on in perception.

But who knows?

The definition of deep learning, I mean, it's pretty broad.

It's kind of an umbrella for a lot of different...

So what I mean is purely sort of neural networks.

Yeah, and a feed forward neural networks.

Essentially.

Or there could be recurrence, but...

Yeah.

Sometimes it feels like, when I started talking to Gary Marcus,

it feels like the criticism of deep learning is kind of like us birds

criticizing airplanes for not flying well.

Or that they're not really flying.

Do you think deep learning...

Do you think it could go all the way, like Yann LeCloon thinks?

Do you think that, yeah, the brute force learning approach can go all the way?

I don't think so, no.

I mean, I think it's an open question.

But I tend to be on the innateness side that there's some things that...

We've been evolved to be able to learn.

And that learning just can't happen without them.

So one example, here's an example I had in the book that I think is useful to me,

at least, in thinking about this.

So this has to do with the DeepMind Atari game playing program.

And it learned to play these Atari video games just by getting input from the pixels

of the screen.

And it learned to play the game Breakout 1,000% better than Humans.

That was one of their results.

And it was great.

And it learned this thing where it tunneled through the side of the bricks in the breakout game

and the ball could bounce off the ceiling and then just wipe out bricks.

Okay.

So there was a group who did an experiment where they took the paddle that you move

with the joystick and moved it up two pixels or something like that.

And then they looked at a deep Q learning system that had been trained on Breakout and

said, could it now transfer its learning to this new version of the game?

Of course, a human could, but...

And it couldn't.

Maybe that's not surprising, but I guess the point is it hadn't learned the concept

of a paddle.

It hadn't learned the concept of a ball or the concept of tunneling.

It was learning something, we, looking at it, kind of anthropomorphized it and said,

oh, here's what it's doing and the way we describe it.

But it actually didn't learn those concepts.

And so because it didn't learn those concepts, it couldn't make this transfer.

Yeah.

So that's a beautiful statement.

But at the same time, by moving the paddle, we also anthropomorphized flaws to inject

into the system that will then flip how impressed we are by it.

What I mean by that is, to me, the Atari games were, to me, deeply impressive that that was

possible at all.

So I have to first pause on that and people should look at that, just like the Game of

Go, which is fundamentally different to me than what DBlue did.

Even though there's still a tree search, it's just everything a deep mind has done in terms

of learning, however limited it is, is still deeply surprising to me.

Yeah, I'm not trying to say that what they did wasn't impressive.

I think it was incredibly impressive.

To me, it's interesting.

Is moving the board just another thing that needs to be learned?

We've been able to, maybe, maybe, been able to, through the current neural networks, learn

very basic concepts that are not enough to do this general reasoning, and maybe with

more data.

I mean, the data, the interesting thing about the examples that you talk about and beautifully

is, it's often flaws of the data.

Well, that's the question.

I mean, I think that is the key question, whether it's a flaw of the data or not.

Because the reason I brought up this example was because you were asking, do I think that

learning from data could go all the way?

And this was why I brought up the example, because I think, and this is not at all to

take away from the impressive work that they did.

But it's to say that when we look at what these systems learn, do they learn the human, the

things that we humans consider to be the relevant concepts?

And in that example, it didn't.

Sure, if you train it on moving the paddle being in different places, maybe it could deal

with, maybe it would learn that concept.

I'm not totally sure.

But the question is scaling that up to more complicated worlds.

To what extent could a machine that only gets this very raw data learn to divide up the

world into relevant concepts?

And I don't know the answer, but I would bet that without some innate notion that it can't do it.

Yeah, 10 years ago, I 100% agree with you as the most expert in AI system.

But now I have a glimmer of hope.

Okay, that's fair enough.

And I think that's what deep learning did in the community is, no, if I had to bet all my money,

100% deep learning will not take us all the way.

But there's still, I was so personally surprised by the tar games, by Go, by the power of self play,

of just game playing, that I was like many other times just humbled of how little I know about what's possible.

Yeah, I think fair enough, self play is amazingly powerful.

And that goes way back to Arthur Samuel with his checker playing program, which was brilliant and surprising that it did so well.

So just for fun, let me ask you on the topic of autonomous vehicles.

It's the area that that I work, at least these days, most closely on.

And it's also area that I think is a good example that you use is sort of an example of things we as humans don't always realize how hard it is to do.

It's like the constant trend in AI, but the different problems that we think are easy when we first try them.

And then we realize how hard it is.

Okay, so why you've talked about autonomous driving being a difficult problem, more difficult than we realize humans give a credit for.

Why is it so difficult? What are the most difficult parts in your view?

I think it's difficult because of the world is so open ended as to what kinds of things can happen.

So you have sort of what normally happens, which is just you drive along and nothing surprising happens.

And autonomous vehicles can do the ones we have now evidently can do really well on most normal situations as long as the weather is reasonably good and everything.

But if some we have this notion of edge case or things in the tail of the distribution called the long tail problem,

which says that there's so many possible things that can happen that was not in the training data of the machine that it won't be able to handle it because it doesn't have common sense.

Right. It's the old the paddle moved.

Yeah, it's the paddle moved problem. Right.

And so my understanding and you probably are more of an expert than I am on this is that current self driving car vision systems have problems with obstacles,

meaning that they don't know which obstacles, which quote unquote obstacles they should stop for and which ones they shouldn't stop for.

And so a lot of times I read that they tend to slam on the brakes quite a bit.

And the most common accidents with self driving cars are people rear ending them because they were surprised.

They weren't expecting the machine the car to stop.

Yeah, so there's there's a lot of interesting questions there.

Whether because you mentioned kind of two things. So one is the problem of perception of understanding of interpreting the objects that are detected.

Right. Correctly.

And the other one is more like the policy, the action that you take, how you respond to it.

So a lot of the cars breaking is a kind of notion of to clarify.

There's a lot of different kind of things that are people calling autonomous vehicles.

But a lot the L four vehicles with a safety driver are the ones like Waymo and Cruz and those companies, they tend to be very conservative and cautious.

So they tend to be very, very afraid of hurting anything or anyone and getting in any kind of accidents.

So their policies very kind of that results in being exceptionally responsive to anything that could possibly be an obstacle.

Right, which which the human drivers around it.

It's unpredictable. It behaves unpredictably.

Yeah, that's not a very human thing to do caution. That's not the thing we're good at, especially in driving, we're in a hurry, often angry and etc.

Especially in Boston.

So and then there's sort of another and a lot of times that's machine learning is not a huge part of that.

It's becoming more and more unclear to me how much, you know, sort of speaking to public information.

Because a lot of companies say they're doing deep learning and machine learning just attract good candidates.

The reality is, in many cases, it's still not a huge part of the of the perception.

There's this LiDAR and there's other sensors that are much more liable for optical detection.

And then there's Tesla approach, which is vision only.

And there's, I think a few companies doing that, but Tesla most sort of famously pushing that forward.

And that's because the LiDAR is too expensive, right?

Well, I mean, yes, but I would say if you were to free give to every Tesla vehicle,

Elon Musk fundamentally believes that LiDAR is a crutch, right?

He said that that if you want to solve the problem of machine learning, LiDAR is not should not be the primary sensor is the belief.

The camera contains a lot more information.

So if you want to learn, you want that information.

But if you want to not to hit obstacles, you want LiDAR, right?

It's sort of this weird trade off because, yeah, it's sort of what Tesla vehicles have a lot of, which is really the thing.

The primary fallback sensor is LiDAR, which is a very crude version of LiDAR.

It's a good detector of obstacles, except when those things are standing, right?

The stopped vehicle.

Right. That's why it had problems with crashing into stopfire trucks.

Stopfire trucks, right?

So the hope there is that the vision sensor would somehow catch that.

There's a lot of problems with perception.

They are doing actually some incredible stuff in the almost like an active learning space where it's constantly taking edge cases and pulling back in.

There's this data pipeline.

Another aspect that is really important that people are studying now is called multitask learning, which is sort of breaking apart this problem, whatever the problem is.

In this case, driving into dozens or hundreds of little problems that you can turn into learning problems.

So this giant pipeline, it's kind of interesting.

I've been skeptical from the very beginning, but become less and less skeptical over time how much of driving can be learned.

I still think it's much farther than the CEO of that particular company thinks it will be.

But it is constantly surprising that through good engineering and data collection and active selection of data, how you can attack that long tail.

And it's an interesting open question that you're absolutely right.

There's a much longer tail and all these edge cases that we don't think about.

But it's a fascinating question that applies to natural language in all spaces.

How big is that long tail?

And I mean, not to linger on the point, but what's your sense in driving in these practical problems of the human experience?

Can it be learned?

So the current, what are your thoughts of sort of Elon Musk's thought, let's forget the thing that he says it'll be solved in a year, but can it be solved in a reasonable timeline?

Or do fundamentally other methods need to be invented?

I think that ultimately driving, so as a trade off in a way, being able to drive and deal with any situation that comes up does require kind of full human intelligence.

And even in humans aren't intelligent enough to do it because humans, I mean, most human accidents are because the human wasn't paying attention or the humans drunk or whatever.

And not because they weren't intelligent enough?

Not because they weren't intelligent enough, right.

Whereas the accidents with autonomous vehicles is because they weren't intelligent enough.

They're always paying attention.

Yeah, they're always paying attention.

So it's a trade off, you know, and I think that it's a very fair thing to say that autonomous vehicles will be ultimately safer than humans because humans are very unsafe.

It's kind of a low bar.

But just like you said, I think humans got a better rap, right?

Because we're really good at the common sense thing.

Yeah, we're great at the common sense thing.

We're bad at the paying attention thing.

Paying attention thing, right?

Especially when we're, you know, driving is kind of boring and we have these phones to play with and everything.

But I think what's going to happen is that for many reasons, not just AI reasons, but also like legal and other reasons that the definition of self driving is going to change or autonomous is going to change.

It's not going to be just, I'm going to go to sleep in the back and you just drive me anywhere.

It's going to be more certain areas are going to be instrumented to have the sensors and the mapping and all of the stuff you need for that, that the autonomous cars won't have to have full common sense.

And they'll do just fine in those areas as long as pedestrians don't mess with them too much.

That's another question.

That's right.

I don't think we will have fully autonomous self driving in the way that like most the average person thinks of it for a very long time.

And just to reiterate, this is the interesting open question that I think I agree with you on is to solve fully autonomous driving.

You have to be able to engineer in common sense.

Yes.

I think it's an important thing to hear and think about.

I hope that's wrong, but I currently agree with you that unfortunately you do have to have to be more specific sort of these deep understandings of physics and of the way this world works.

And also the human dynamics that you mentioned pedestrians and cyclists are actually that's whatever that nonverbal communication is. Some people call it. There's that dynamic that is also part of this common sense.

Right. And we humans are pretty good at predicting what other humans are going to do.

And how our actions impact the behaviors of so this is weird game theoretic dance that we're good at somehow.

And the funny thing is, because I've watched countless hours of pedestrian video and talked to people, we humans are also really bad at articulating the knowledge we have.

Right.

Which has been a huge challenge.

Yes.

So you've mentioned embodied intelligence. What do you think it takes to build a system of human level intelligence? Does it need to have a body?

I'm not sure, but I, I'm coming around to that more and more.

And what does it mean to be, I don't mean to keep bringing up Yalakoon.

He looms very large.

Well, he certainly has a large personality. Yes.

He thinks that the system needs to be grounded, meaning he needs to sort of be able to interact with reality, but doesn't think it necessarily needs to have a body.

So when you think of.

What's the difference?

I guess I want to ask, when you mean body, do you mean you have to be able to play with the world?

Or do you also mean like there's a body that you, that you have to preserve?

That's a good question.

I haven't really thought about that, but I think both I would guess because it's because I think you, I think intelligence.

It's so hard to separate it from our self, our desire for self preservation, our emotions are all that non rational stuff that kind of gets in the way of logical thinking.

Because we, the way, you know, if we're talking about human intelligence or human level intelligence, whatever that means, a huge part of it is social.

That, you know, we were evolved to be social and to deal with other people.

And that's just so ingrained in us that it's hard to separate intelligence from that.

I think, you know, AI for the last 70 years or however long it's been around, it has largely been separated.

There's this idea that there's like, it's kind of very Cartesian.

There's this, you know, thinking thing that we're trying to create, but we don't care about all this other stuff.

And I think the other stuff is very fundamental.

So there's the idea that things like emotion get in the way of intelligence.

As opposed to being an integral part of it.

Integral part of it.

So, I mean, I'm Russian, so romanticize the notions of emotion and suffering and all that kind of fear of mortality, those kinds of things.

So in AI, especially.

By the way, did you see that there was this recent thing going around the internet of this, some, I think he's a Russian or some Slavic had written this thing, sort of anti the idea of superintelligence.

I forgot, maybe he's Polish.

Anyway, so we had all these arguments and one was the argument from Slavic pessimism.

My favorite.

Do you remember what the argument is?

It's like, nothing ever works. Everything sucks.

So what do you think is the role like that's such a fascinating idea that the what we perceive as sort of the limits of human.

The human mind, which is emotion and fear and all those kinds of things are integral to intelligence.

Could you elaborate on that? Like, what, why is that important, do you think, for human level intelligence?

At least for the way the humans work, it's a big part of how it affects how we perceive the world.

It affects how we make decisions about the world.

It affects how we interact with other people.

It affects our understanding of other people, you know.

For me to understand your, what you're going, what you're likely to do.

I need to have kind of a theory of mind and that's very much a theory of emotion and motivations and goals.

And to understand that, I, you know, we have this whole system of mirror neurons.

You know, I sort of understand your motivations through sort of simulating it myself.

So, you know, it's not something that I can prove that's necessary, but it seems very likely.

So, okay, you've written the op ed in New York Times titled, We Shouldn't Be Scared by Super Intelligent AI, and it criticized a little bit Stuart Russell and Nick Bostrom.

Can you try to summarize that article's key ideas?

So, it was spurred by a earlier New York Times op ed by Stuart Russell, which was summarizing his book called Human Compatible.

And the article was saying, you know, if we, if we have super intelligent AI, we need to have its values aligned with our values and it has to learn about what we really want.

And he gave this example, what if we have a super intelligent AI and we give it the problem of solving climate change and it decides that the best way to lower the carbon in the atmosphere is to kill all the humans.

Okay.

So, to me, that just made no sense at all because a super intelligent AI, first of all, thinking, trying to figure out what super intelligence means.

And it doesn't, it seems that something that's super intelligent can't just be intelligent along this one dimension of, okay, I'm going to figure out all the steps, the best optimal path to solving climate change.

And not be intelligent enough to figure out that humans don't want to be killed, that you could get to one without having the other.

And, you know, Bostrom, in his book talks about the orthogonality hypothesis where he says he thinks that a systems, I can't remember exactly what it is, but like a systems goals and its values don't have to be aligned.

There's some orthogonality there, which didn't make any sense to me.

So you're saying in any system that's sufficiently not even super intelligent, but as opposed to greater and greater intelligence, there's a holistic nature that will sort of attention that will naturally emerge that prevents it from sort of any one dimension running away.

Yeah.

Yeah, exactly.

So, you know, Bostrom had this example of the super intelligent AI that turns the world into paper clips, because its job is to make paper clips or something.

And that just as a thought experiment didn't make any sense to me.

Well, as a thought experiment, there's a thing that could possibly be realized.

Either.

So I think that, you know, what my op ad was trying to do was say that intelligence is more complex than these people are presenting it, that it's not like it's not so separable, the rationality, the values, the emotions, all of that.

That it's the view that you could separate all these dimensions and build the machine that has one of these dimensions and it's super intelligent in one dimension, but it doesn't have any of the other dimensions.

That's what I was trying to criticize that I don't believe that.

So can I read a few sentences from your show, Benjamin, who is always super eloquent.

So he writes, I have the same impression as Melanie that our cognitive biases are linked with our ability to learn to solve many problems.

They may also be a limiting factor for AI.

However, this is a may in quotes.

Things may also turn out differently and there's a lot of uncertainty about the capabilities of future machines.

But more importantly for me, the value alignment problem is a problem well before we reach some hypothetical super intelligence.

It is already posing a problem in the form of super powerful companies whose objective function may not be sufficiently aligned with humanity's general well being, creating all kinds of harmful side effects.

So he goes on to argue that the orthogonality and those kinds of things, the concerns of just aligning values with the capabilities of the system is something that might come long before we reach anything like super intelligence.

So your criticism is kind of really nice to saying this idea of super intelligent systems seem to be dismissing fundamental parts of what intelligence would take.

And then Yoshio kind of says, yes, but if we look at systems that are much less intelligent, there might be these same kinds of problems that emerge.

Sure, but I guess the example that he gives there of these corporations, that's people, right?

Those are people's values.

I mean, we're talking about people, the corporations are, their values are the values of the people who run those corporations.

But the idea is the algorithm, that's right.

So the fundamental person, the fundamental element of what does the bad thing as a human being.

But the algorithm kind of controls the behavior of this mass of human beings.

Which algorithm?

For a company, that's the, for example, if it's advertisement driven company that recommends certain things and encourages engagement.

So it gets money by encouraging engagement.

And therefore, the company more and more, it's like the cycle that builds an algorithm that enforces more engagement and made perhaps more division in the culture and so on, so on.

I guess the question here is sort of who has the agency.

So you might say, for instance, we don't want our algorithms to be racist.

And facial recognition, you know, some people have criticized some facial recognition systems as being racist because they're not as good on darker skin and lighter skin.

Okay, but the agency there, the actual facial recognition algorithm isn't what has the agency.

It's not the racist thing, right?

It's the, I don't know, the combination of the training data, the cameras being used, whatever.

But my understanding of, and I say, I agree with Benjio there that he, you know, I think there are these value issues with our use of algorithms.

But my understanding of what Russell's argument was is more that the algorithm, the machine itself has the agency now.

It's the thing that's making the decisions and it's the thing that has what we would call values.

Yes.

So whether that's just a matter of degree, you know, it's hard to say, right?

But I would say that's sort of qualitatively different than a face recognition neural network.

And to broadly linger on that point, if you look at Elon Musk or Russell or Bostrom, people who are worried about existential risks of AI, however far into the future.

The argument goes is it eventually happens.

We don't know how far, but it eventually happens.

Do you share any of those concerns and what kind of concerns in general do you have about AI that approach anything like existential threat to humanity?

So I would say, yes, it's possible.

But I think there's a lot more closer in existential threats to humanity.

Because you said like a hundred years for, so your time.

More than a hundred years.

More than a hundred years.

Maybe even more than 500 years.

I don't know.

I mean, it's.

So the existential threats are so far out that the future is, I mean, there'll be a million different technologies that we can't even predict now that will fundamentally change the nature of our behavior, reality, society and so on before then.

I think so.

I think so.

And, you know, we have so many other pressing existential threats going on.

Nuclear weapons even.

Nuclear weapons, climate problems, you know.

Poverty, possible pandemics that you can go on and on.

And I think though, you know, worrying about existential threat from AI is.

It is not the best priority for what we should be worried about that.

That's kind of my view because we're so far away.

But, you know, I'm not necessarily criticizing Russell or Bostrom or whoever for worrying about that.

And I think some people should be worried about it.

It's certainly fine.

But I was more sort of getting at their view of what intelligence is.

So I was more focusing on like their view of superintelligence than just the fact of them worrying.

And the title of the article was written by the New York Times editors.

I wouldn't have called it that.

We shouldn't be scared by superintelligence.

No.

If you wrote it, it'd be like we should redefine what you mean by superintelligence.

It actually said something like superintelligence is not a sort of coherent idea.

But that's not like something New York Times would put in.

And the follow up argument that Yoshio makes also, not argument, but a statement.

And I've heard him say it before and I think I agree.

He's kind of has a very friendly way of phrasing it as it's good for a lot of people to believe different things.

He's such a nice guy.

Yeah.

But he's also practically speaking like we shouldn't be like while your article stands like Stuart Russell does amazing work.

Bostrom does a lot of amazing work.

You do amazing work.

And even when you disagree about the definition of superintelligence or the usefulness of even the term.

It's still useful to have people that like use that term.

Right.

And then argue.

I absolutely agree with Benjo there.

And I think it's great that, you know, and it's great that New York Times will publish all this stuff.

That's right.

It's an exciting time to be here.

What do you think is a good test of intelligence?

Is natural language ultimately a test that you find the most compelling like the original or the what, you know, the higher levels of the Turing test kind of?

Yeah.

I still think the original idea of the Turing test is a good test for intelligence.

I mean, I can't think of anything better.

You know, the Turing test, the way that it's been carried out so far has been very impoverished, if you will.

But I think a real Turing test that really goes into depth, like the one that I mentioned, I talked about in the book, I talked about Ray Kurzweil and Mitchell Kapoor have this bet.

Right.

And in 2029, I think is the date there, the machine will pass the Turing test and they have a very specific like how many hours expert judges and all of that.

And, you know, Kurzweil says yes, Kapoor says no.

We can, we only have like nine more years to go to see.

I, you know, if something, a machine could pass that, I would be willing to call it intelligent.

Of course, nobody will.

They will say that's just a language model, right, if it does.

So you would be comfortable.

So language, a long conversation that's, well, yeah, you're, I mean, you're right, because I think probably to carry out that long conversation, you would literally need to have deep common sense understanding of the world.

I think so.

I think so.

And the conversation is enough to reveal that.

I think so.

Perhaps it is.

So another super fun topic of complexity that you have worked on written about.

Let me ask the basic question.

What is complexity?

So complexity is another one of those terms, like intelligence is perhaps overused.

But my book about complexity was about this wide area of complex systems, studying different systems in nature, in technology, in society, in which you have emergence, kind of like I was talking about with intelligence, you know, we have the brain, which has billions of neurons.

And each neuron individually could be said to be not very complex compared to the system as a whole.

But the system, the interactions of those neurons and the dynamics creates these phenomena that we call intelligence or consciousness, you know, that are, we consider to be very complex.

So the field of complexity is trying to find general principles that underlie all these systems that have these kinds of emergent properties.

And the emergence occurs from like underlying the complex system is usually simple fundamental interactions.

Yes.

And the emergence happens when there's just a lot of these things interacting.

Yes.

Sort of what, and then most of science to date, can you talk about what is reductionism?

Well, reductionism is when you try and take a system and divide it up into its elements, whether those be cells or atoms or subatomic particles, whatever your field is.

And then try and understand those elements and then try and build up an understanding of the whole system by looking at sort of the sum of all the elements.

So what's your sense, whether we're talking about intelligence or these kinds of interesting complex systems, is it possible to understand them in a reductionist way?

Which is probably the approach of most of science today, right?

I don't think it's always possible to understand the things we want to understand the most.

So I don't think it's possible to look at single neurons and understand what we call intelligence, you know, to look at sort of summing up.

So sort of the summing up is the issue here that we're, you know, one example is that the human genome, right?

So there was a lot of work on excitement about sequencing the human genome because the idea would be that we'd be able to find genes that underlies diseases.

But it turns out that, and it was a very reductionist idea.

We'd figure out what all the parts are, and then we would be able to figure out which parts cause which things.

But it turns out that the parts don't cause the things that we're interested in.

It's like the interactions, it's the networks of these parts.

And so that kind of reductionist approach didn't yield the explanation that we wanted.

What do you use the most beautiful complex system that you've encountered?

That's beautiful.

That you've been captivated by.

Is it sort of, I mean, for me, is the simplest to be cellular automata?

Oh, yeah.

So I was very captivated by cellular automata and worked on cellular automata for several years.

Is it amazing or is it surprising that such simple systems, such simple rules in cellular automata can create sort of seemingly unlimited complexity?

Yeah, that was very surprising to me.

How do you make sense of it?

How does that make you feel?

Is it just ultimately humbling?

Or is there a hope to somehow leverage this into a deeper understanding and even able to engineer things like intelligence?

It's definitely humbling, how humbling in that, also kind of awe inspiring, that it's that awe inspiring part of mathematics,

that these incredibly simple rules can produce this very beautiful, complex, hard to understand behavior.

And it's mysterious and surprising still.

But exciting because it does give you kind of the hope that you might be able to engineer complexity just from simple rules from the beginning.

Can you briefly say what is the Santa Fe Institute, its history, its culture, its ideas, its future?

So I've never, as I mentioned to you, I've never been, but it's always been this, in my mind, this mystical place where brilliant people study the edge of chaos.

Yeah, exactly.

So the Santa Fe Institute was started in 1984.

And it was created by a group of scientists, a lot of them from Los Alamos National Lab, which is about a 40 minute drive from the Santa Fe Institute.

They were mostly physicists and chemists.

But they were frustrated in their field because they felt that their field wasn't approaching kind of big interdisciplinary questions like the kinds we've been talking about.

And they wanted to have a place where people from different disciplines could work on these big questions without sort of being siloed into physics, chemistry, biology, whatever.

So they started this institute.

And this was people like George Cowan, who was a chemist in the Manhattan Project, and Nicholas Metropolis, who, a mathematician, physicist, Marie Gilman, physicist, so some really big names here,

Ken Arrow, a Nobel Prize winning economist, and they started having these workshops.

And this whole enterprise kind of grew into this research institute that itself has been kind of on the edge of chaos its whole life because it doesn't have a significant endowment.

And it's just been kind of living on whatever funding it can raise through donations and grants and however it can, you know, business associates and so on.