back to indexMatt Botvinick: Neuroscience, Psychology, and AI at DeepMind | Lex Fridman Podcast #106
link |
The following is a conversation with Matt Botmanek,
link |
Director of Neuroscience Research at DeepMind.
link |
He's a brilliant, cross disciplinary mind,
link |
navigating effortlessly between cognitive psychology,
link |
computational neuroscience, and artificial intelligence.
link |
Quick summary of the ads.
link |
Two sponsors, The Jordan Harbinger Show
link |
and Magic Spoon Cereal.
link |
Please consider supporting the podcast
link |
by going to jordanharbinger.com slash lex
link |
and also going to magicspoon.com slash lex
link |
and using code lex at checkout
link |
after you buy all of their cereal.
link |
Click the links, buy the stuff.
link |
It's the best way to support this podcast
link |
and the journey I'm on.
link |
If you enjoy this podcast, subscribe on YouTube,
link |
review it with five stars on Apple Podcast,
link |
follow on Spotify, support on Patreon,
link |
or connect with me on Twitter at lexfriedman,
link |
spelled surprisingly without the E,
link |
just F R I D M A N.
link |
As usual, I'll do a few minutes of ads now
link |
and never any ads in the middle
link |
that can break the flow of the conversation.
link |
This episode is supported by The Jordan Harbinger Show.
link |
Go to jordanharbinger.com slash lex.
link |
It's how he knows I sent you.
link |
On that page, subscribe to his podcast
link |
on Apple Podcast, Spotify, and you know where to look.
link |
I've been binging on his podcast.
link |
Jordan is a great interviewer
link |
and even a better human being.
link |
I recently listened to his conversation with Jack Barsky,
link |
former sleeper agent for the KGB in the 80s
link |
and author of Deep Undercover,
link |
which is a memoir that paints yet another
link |
interesting perspective on the Cold War era.
link |
I've been reading a lot about the Stalin
link |
and then Gorbachev and Putin eras of Russia,
link |
but this conversation made me realize
link |
that I need to do a deep dive into the Cold War era
link |
to get a complete picture of Russia's recent history.
link |
Again, go to jordanharbinger.com slash lex.
link |
Subscribe to his podcast.
link |
It's how he knows I sent you.
link |
It's awesome, you won't regret it.
link |
This episode is also supported by Magic Spoon,
link |
low carb, keto friendly, super amazingly delicious cereal.
link |
I've been on a keto or very low carb diet
link |
for a long time now.
link |
It helps with my mental performance.
link |
It helps with my physical performance,
link |
even during this crazy push up, pull up challenge I'm doing,
link |
including the running, it just feels great.
link |
I used to love cereal.
link |
Obviously, I can't have it now
link |
because most cereals have crazy amounts of sugar,
link |
which is terrible for you, so I quit it years ago.
link |
But Magic Spoon, amazingly, somehow,
link |
is a totally different thing.
link |
Zero sugar, 11 grams of protein,
link |
and only three net grams of carbs.
link |
It tastes delicious.
link |
It has a lot of flavors, two new ones,
link |
including peanut butter.
link |
But if you know what's good for you,
link |
you'll go with cocoa, my favorite flavor,
link |
and the flavor of champions.
link |
Click the magicspoon.com slash lex link in the description
link |
and use code lex at checkout for free shipping
link |
and to let them know I sent you.
link |
They have agreed to sponsor this podcast for a long time.
link |
They're an amazing sponsor and an even better cereal.
link |
I highly recommend it.
link |
It's delicious, it's good for you, you won't regret it.
link |
And now, here's my conversation with Matt Botpenik.
link |
How much of the human brain do you think we understand?
link |
I think we're at a weird moment
link |
in the history of neuroscience in the sense that
link |
I feel like we understand a lot about the brain
link |
at a very high level, but a very coarse level.
link |
When you say high level, what are you thinking?
link |
Are you thinking functional?
link |
Are you thinking structurally?
link |
So in other words, what is the brain for?
link |
What kinds of computation does the brain do?
link |
What kinds of behaviors would we have to explain
link |
if we were gonna look down at the mechanistic level?
link |
And at that level, I feel like we understand
link |
much, much more about the brain
link |
than we did when I was in high school.
link |
But it's almost like we're seeing it through a fog.
link |
It's only at a very coarse level.
link |
We don't really understand what the neuronal mechanisms are
link |
that underlie these computations.
link |
We've gotten better at saying,
link |
what are the functions that the brain is computing
link |
that we would have to understand
link |
if we were gonna get down to the neuronal level?
link |
And at the other end of the spectrum,
link |
in the last few years, incredible progress has been made
link |
in terms of technologies that allow us to see,
link |
actually literally see, in some cases,
link |
what's going on at the single unit level,
link |
even the dendritic level.
link |
And then there's this yawning gap in between.
link |
Well, that's interesting.
link |
So at the high level,
link |
so that's almost a cognitive science level.
link |
And then at the neuronal level,
link |
that's neurobiology and neuroscience,
link |
just studying single neurons,
link |
the synaptic connections and all the dopamine,
link |
all the kind of neurotransmitters.
link |
One blanket statement I should probably make
link |
is that as I've gotten older,
link |
I have become more and more reluctant
link |
to make a distinction between psychology and neuroscience.
link |
To me, the point of neuroscience
link |
is to study what the brain is for.
link |
If you're a nephrologist
link |
and you wanna learn about the kidney,
link |
you start by saying, what is this thing for?
link |
Well, it seems to be for taking blood on one side
link |
that has metabolites in it that shouldn't be there,
link |
sucking them out of the blood
link |
while leaving the good stuff behind,
link |
and then excreting that in the form of urine.
link |
That's what the kidney is for.
link |
It's like obvious.
link |
So the rest of the work is deciding how it does that.
link |
And this, it seems to me,
link |
is the right approach to take to the brain.
link |
You say, well, what is the brain for?
link |
The brain, as far as I can tell, is for producing behavior.
link |
It's for going from perceptual inputs to behavioral outputs,
link |
and the behavioral outputs should be adaptive.
link |
So that's what psychology is about.
link |
It's about understanding the structure of that function.
link |
And then the rest of neuroscience is about figuring out
link |
how those operations are actually carried out
link |
at a mechanistic level.
link |
That's really interesting, but so unlike the kidney,
link |
the brain, the gap between the electrical signal
link |
and behavior, so you truly see neuroscience
link |
as the science that touches behavior,
link |
how the brain generates behavior,
link |
or how the brain converts raw visual information
link |
into understanding.
link |
Like, you basically see cognitive science,
link |
psychology, and neuroscience as all one science.
link |
Yeah, it's a personal statement.
link |
Is that a hopeful or a realistic statement?
link |
So certainly you will be correct in your feeling
link |
in some number of years, but that number of years
link |
could be 200, 300 years from now.
link |
Oh, well, there's a...
link |
Is that aspirational or is that pragmatic engineering
link |
feeling that you have?
link |
It's both in the sense that this is what I hope
link |
and expect will bear fruit over the coming decades,
link |
but it's also pragmatic in the sense that I'm not sure
link |
what we're doing in either psychology or neuroscience
link |
if that's not the framing.
link |
I don't know what it means to understand the brain
link |
if there's no, if part of the enterprise
link |
is not about understanding the behavior
link |
that's being produced.
link |
I mean, yeah, but I would compare it
link |
to maybe astronomers looking at the movement
link |
of the planets and the stars without any interest
link |
of the underlying physics, right?
link |
And I would argue that at least in the early days,
link |
there is some value to just tracing the movement
link |
of the planets and the stars without thinking
link |
about the physics too much because it's such a big leap
link |
to start thinking about the physics
link |
before you even understand even the basic structural
link |
Oh, I agree with that.
link |
But you're saying in the end, the goal should be
link |
to deeply understand.
link |
Well, right, and I think...
link |
So I thought about this a lot when I was in grad school
link |
because a lot of what I studied in grad school
link |
was psychology and I found myself a little bit confused
link |
about what it meant to...
link |
It seems like what we were talking about a lot of the time
link |
were virtual causal mechanisms.
link |
Like, oh, well, you know, attentional selection
link |
then selects some object in the environment
link |
and that is then passed on to the motor, you know,
link |
information about that is passed on to the motor system.
link |
But these are virtual mechanisms.
link |
These are, you know, they're metaphors.
link |
They're, you know, there's no reduction going on
link |
in that conversation to some physical mechanism that,
link |
you know, which is really what it would take
link |
to fully understand, you know, how behavior is rising.
link |
But the causal mechanisms are definitely neurons interacting.
link |
I'm willing to say that at this point in history.
link |
So in psychology, at least for me personally,
link |
there was this strange insecurity about trafficking
link |
in these metaphors, you know,
link |
which were supposed to explain the function of the mind.
link |
If you can't ground them in physical mechanisms,
link |
then what is the explanatory validity of these explanations?
link |
And I managed to soothe my own nerves
link |
by thinking about the history of genetics research.
link |
So I'm very far from being an expert
link |
on the history of this field.
link |
But I know enough to say that, you know,
link |
Mendelian genetics preceded, you know, Watson and Crick.
link |
And so there was a significant period of time
link |
during which people were, you know,
link |
productively investigating the structure of inheritance
link |
using what was essentially a metaphor,
link |
the notion of a gene, you know.
link |
Oh, genes do this and genes do that.
link |
But, you know, where are the genes?
link |
They're sort of an explanatory thing that we made up.
link |
And we ascribed to them these causal properties.
link |
Oh, there's a dominant, there's a recessive,
link |
and then they recombine it.
link |
And then later, there was a kind of blank there
link |
that was filled in with a physical mechanism.
link |
That connection was made.
link |
But it was worth having that metaphor
link |
because that gave us a good sense
link |
of what kind of causal mechanism we were looking for.
link |
And the fundamental metaphor of cognition, you said,
link |
is the interaction of neurons.
link |
Is that, what is the metaphor?
link |
No, no, the metaphor,
link |
the metaphors we use in cognitive psychology
link |
are things like attention, the way that memory works.
link |
I retrieve something from memory, right?
link |
A memory retrieval occurs.
link |
You know, that's not a physical mechanism
link |
that I can examine in its own right.
link |
But it's still worth having, that metaphorical level.
link |
Yeah, so yeah, I misunderstood actually.
link |
So the higher level of abstractions
link |
is the metaphor that's most useful.
link |
But what about, so how does that connect
link |
to the idea that that arises from interaction of neurons?
link |
Well, even, is the interaction of neurons
link |
also not a metaphor to you?
link |
Or is it literally, like that's no longer a metaphor.
link |
That's already the lowest level of abstractions
link |
that could actually be directly studied.
link |
Well, I'm hesitating because I think
link |
what I want to say could end up being controversial.
link |
So what I want to say is, yes,
link |
the interactions of neurons, that's not metaphorical.
link |
That's a physical fact.
link |
That's where the causal interactions actually occur.
link |
Now, I suppose you could say,
link |
well, even that is metaphorical relative
link |
to the quantum events that underlie.
link |
I don't want to go down that rabbit hole.
link |
It's always turtles on top of turtles.
link |
Yeah, there's turtles all the way down.
link |
There's a reduction that you can do.
link |
You can say these psychological phenomena
link |
can be explained through a very different
link |
kind of causal mechanism,
link |
which has to do with neurotransmitter release.
link |
And so what we're really trying to do
link |
in neuroscience writ large, as I say,
link |
which for me includes psychology,
link |
is to take these psychological phenomena
link |
and map them onto neural events.
link |
I think remaining forever at the level of description
link |
that is natural for psychology,
link |
for me personally, would be disappointing.
link |
I want to understand how mental activity
link |
arises from neural activity.
link |
But the converse is also true.
link |
Studying neural activity without any sense
link |
of what you're trying to explain,
link |
to me feels like at best groping around at random.
link |
Now, you've kind of talked about this bridging
link |
of the gap between psychology and neuroscience,
link |
but do you think it's possible,
link |
like my love is, like I fell in love with psychology
link |
and psychiatry in general with Freud
link |
and when I was really young,
link |
and I hoped to understand the mind.
link |
And for me, understanding the mind,
link |
at least at that young age before I discovered AI
link |
and even neuroscience was to, is psychology.
link |
And do you think it's possible to understand the mind
link |
without getting into all the messy details of neuroscience?
link |
Like you kind of mentioned to you it's appealing
link |
to try to understand the mechanisms at the lowest level,
link |
but do you think that's needed,
link |
that's required to understand how the mind works?
link |
That's an important part of the whole picture,
link |
but I would be the last person on earth
link |
to suggest that that reality
link |
renders psychology in its own right unproductive.
link |
I trained as a psychologist.
link |
I am fond of saying that I have learned much more
link |
from psychology than I have from neuroscience.
link |
To me, psychology is a hugely important discipline.
link |
And one thing that warms in my heart is that
link |
ways of investigating behavior
link |
that have been native to cognitive psychology
link |
since it's dawn in the 60s
link |
are starting to become,
link |
they're starting to become interesting to AI researchers
link |
for a variety of reasons.
link |
And that's been exciting for me to see.
link |
Can you maybe talk a little bit about what you see
link |
as beautiful aspects of psychology,
link |
maybe limiting aspects of psychology?
link |
I mean, maybe just start it off as a science, as a field.
link |
To me, it was when I understood what psychology is,
link |
analytical psychology,
link |
like the way it's actually carried out,
link |
it was really disappointing to see two aspects.
link |
One is how small the N is,
link |
how small the number of subject is in the studies.
link |
And two, it was disappointing to see
link |
how controlled the entire,
link |
how much it was in the lab.
link |
It wasn't studying humans in the wild.
link |
There was no mechanism for studying humans in the wild.
link |
So that's where I became a little bit disillusioned
link |
And then the modern world of the internet
link |
is so exciting to me.
link |
The Twitter data or YouTube data,
link |
data of human behavior on the internet becomes exciting
link |
because the N grows and then in the wild grows.
link |
But that's just my narrow sense.
link |
Like, do you have a optimistic or pessimistic
link |
cynical view of psychology?
link |
How do you see the field broadly?
link |
When I was in graduate school,
link |
it was early enough that there was still a thrill
link |
in seeing that there were ways of doing,
link |
there were ways of doing experimental science
link |
that provided insight to the structure of the mind.
link |
One thing that impressed me most when I was at that stage
link |
in my education was neuropsychology,
link |
looking at, analyzing the behavior of populations
link |
who had brain damage of different kinds
link |
and trying to understand what the specific deficits were
link |
that arose from a lesion in a particular part of the brain.
link |
And the kind of experimentation that was done
link |
and that's still being done to get answers in that context
link |
was so creative and it was so deliberate.
link |
It was good science.
link |
An experiment answered one question but raised another
link |
and somebody would do an experiment
link |
that answered that question.
link |
And you really felt like you were narrowing in on
link |
some kind of approximate understanding
link |
of what this part of the brain was for.
link |
Do you have an example from memory
link |
of what kind of aspects of the mind
link |
could be studied in this kind of way?
link |
I mean, the very detailed neuropsychological studies
link |
of language function,
link |
looking at production and reception
link |
and the relationship between visual function,
link |
reading and auditory and semantic.
link |
There were these, and still are, these beautiful models
link |
that came out of that kind of research
link |
that really made you feel like you understood something
link |
that you hadn't understood before
link |
about how language processing is organized in the brain.
link |
But having said all that,
link |
I think you are, I mean, I agree with you
link |
that the cost of doing highly controlled experiments
link |
is that you, by construction, miss out on the richness
link |
and complexity of the real world.
link |
One thing that, so I was drawn into science
link |
by what in those days was called connectionism,
link |
which is, of course, what we now call deep learning.
link |
And at that point in history,
link |
neural networks were primarily being used
link |
in order to model human cognition.
link |
They weren't yet really useful for industrial applications.
link |
So you always found neural networks
link |
in biological form beautiful.
link |
Oh, neural networks were very concretely the thing
link |
that drew me into science.
link |
I was handed, are you familiar with the PDP books
link |
from the 80s when I was in,
link |
I went to medical school before I went into science.
link |
Really, interesting.
link |
I also did a graduate degree in art history,
link |
so I'm kind of exploring.
link |
Well, art history, I understand.
link |
That's just a curious, creative mind.
link |
But medical school, with the dream of what,
link |
if we take that slight tangent?
link |
What, did you want to be a surgeon?
link |
I actually was quite interested in surgery.
link |
I was interested in surgery and psychiatry.
link |
And I thought, I must be the only person on the planet
link |
who was torn between those two fields.
link |
And I said exactly that to my advisor in medical school,
link |
who turned out, I found out later,
link |
to be a famous psychoanalyst.
link |
And he said to me, no, no, it's actually not so uncommon
link |
to be interested in surgery and psychiatry.
link |
And he conjectured that the reason
link |
that people develop these two interests
link |
is that both fields are about going beneath the surface
link |
and kind of getting into the kind of secret.
link |
I mean, maybe you understand this as someone
link |
who was interested in psychoanalysis.
link |
There's sort of a, there's a cliche phrase
link |
that people use now, like in NPR,
link |
the secret life of blankety blank, right?
link |
And that was part of the thrill of surgery,
link |
was seeing the secret activity
link |
that's inside everybody's abdomen and thorax.
link |
That's a very poetic way to connect it to disciplines
link |
that are very, practically speaking,
link |
different from each other.
link |
That's for sure, that's for sure, yes.
link |
So how did we get onto medical school?
link |
So I was in medical school
link |
and I was doing a psychiatry rotation
link |
and my kind of advisor in that rotation
link |
asked me what I was interested in.
link |
And I said, well, maybe psychiatry.
link |
And I said, well, I've always been interested
link |
in how the brain works.
link |
I'm pretty sure that nobody's doing scientific research
link |
that addresses my interests,
link |
which are, I didn't have a word for it then,
link |
but I would have said about cognition.
link |
And he said, well, you know, I'm not sure that's true.
link |
You might be interested in these books.
link |
And he pulled down the PDB books from his shelf
link |
and they were still shrink wrapped.
link |
He hadn't read them, but he handed them to me.
link |
He said, you feel free to borrow these.
link |
And that was, you know, I went back to my dorm room
link |
and I just, you know, read them cover to cover.
link |
Parallel distributed processing,
link |
which was one of the original names for deep learning.
link |
And so I apologize for the romanticized question,
link |
but what idea in the space of neuroscience
link |
and the space of the human brain is to you
link |
the most beautiful, mysterious, surprising?
link |
What had always fascinated me,
link |
even when I was a pretty young kid, I think,
link |
was the paradox that lies in the fact
link |
that the brain is so mysterious
link |
and seems so distant.
link |
But at the same time,
link |
it's responsible for the full transparency
link |
The brain is literally what makes everything obvious
link |
And there's always one in the room with you.
link |
I used to teach, when I taught at Princeton,
link |
I used to teach a cognitive neuroscience course.
link |
And the very last thing I would say to the students was,
link |
you know, people often,
link |
when people think of scientific inspiration,
link |
the metaphor is often, well, look to the stars.
link |
The stars will inspire you to wonder at the universe
link |
and think about your place in it and how things work.
link |
And I'm all for looking at the stars,
link |
but I've always been much more inspired.
link |
And my sense of wonder comes from the,
link |
not from the distant, mysterious stars,
link |
but from the extremely intimately close brain.
link |
There's something just endlessly fascinating
link |
The, like, just like you said,
link |
the one that's close and yet distant
link |
in terms of our understanding of it.
link |
Do you, are you also captivated by the fact
link |
that this very conversation is happening
link |
because two brains are communicating so that?
link |
The, I guess what I mean is the subjective nature
link |
of the experience, if it can take a small attention
link |
into the mystical of it, the consciousness,
link |
or when you were saying you're captivated
link |
by the idea of the brain,
link |
are you talking about specifically
link |
the mechanism of cognition?
link |
Or are you also just, like, at least for me,
link |
it's almost like paralyzing the beauty and the mystery
link |
of the fact that it creates the entirety of the experience,
link |
not just the reasoning capability, but the experience.
link |
Well, I definitely resonate with that latter thought.
link |
And I often find discussions of artificial intelligence
link |
to be disappointingly narrow.
link |
Speaking as someone who has always had an interest in art.
link |
I was just gonna go there
link |
because it sounds like somebody who has an interest in art.
link |
Yeah, I mean, there are many layers
link |
to full bore human experience.
link |
And in some ways it's not enough to say,
link |
oh, well, don't worry, we're talking about cognition,
link |
but we'll add emotion, you know?
link |
There's an incredible scope
link |
to what humans go through in every moment.
link |
And yes, so that's part of what fascinates me,
link |
is that our brains are producing that.
link |
But at the same time, it's so mysterious to us.
link |
Our brains are literally in our heads
link |
producing this experience.
link |
Producing the experience.
link |
And yet it's so mysterious to us.
link |
And so, and the scientific challenge
link |
of getting at the actual explanation for that
link |
is so overwhelming.
link |
That's just, I don't know.
link |
Certain people have fixations on particular questions
link |
and that's always, that's just always been mine.
link |
Yeah, I would say the poetry of that is fascinating.
link |
And I'm really interested in natural language as well.
link |
And when you look at artificial intelligence community,
link |
it always saddens me how much
link |
when you try to create a benchmark
link |
for the community to gather around,
link |
how much of the magic of language is lost
link |
when you create that benchmark.
link |
That there's something, we talk about experience,
link |
the music of the language, the wit,
link |
the something that makes a rich experience,
link |
something that would be required to pass
link |
the spirit of the Turing test is lost in these benchmarks.
link |
And I wonder how to get it back in
link |
because it's very difficult.
link |
The moment you try to do like real good rigorous science,
link |
you lose some of that magic.
link |
When you try to study cognition
link |
in a rigorous scientific way,
link |
it feels like you're losing some of the magic.
link |
The seeing cognition in a mechanistic way
link |
that AI folk at this stage in our history.
link |
Well, I agree with you, but at the same time,
link |
one thing that I found really exciting
link |
about that first wave of deep learning models in cognition
link |
was the fact that the people who were building these models
link |
were focused on the richness and complexity
link |
of human cognition.
link |
So an early debate in cognitive science,
link |
which I sort of witnessed as a grad student
link |
was about something that sounds very dry,
link |
which is the formation of the past tense.
link |
But there were these two camps.
link |
One said, well, the mind encodes certain rules
link |
and it also has a list of exceptions
link |
because of course, the rule is add ED,
link |
but that's not always what you do.
link |
So you have to have a list of exceptions.
link |
And then there were the connectionists
link |
who evolved into the deep learning people who said,
link |
well, if you look carefully at the data,
link |
if you actually look at corpora, like language corpora,
link |
it turns out to be very rich
link |
because yes, there are most verbs
link |
that you just tack on ED, and then there are exceptions,
link |
but there are rules that the exceptions aren't just random.
link |
There are certain clues to which verbs
link |
should be exceptional.
link |
And then there are exceptions to the exceptions.
link |
And there was a word that was kind of deployed
link |
in order to capture this, which was quasi regular.
link |
In other words, there are rules, but it's messy.
link |
And there's either structure even among the exceptions.
link |
And it would be, yeah, you could try to write down,
link |
we could try to write down the structure
link |
in some sort of closed form,
link |
but really the right way to understand
link |
how the brain is handling all this,
link |
and by the way, producing all of this,
link |
is to build a deep neural network
link |
and train it on this data
link |
and see how it ends up representing all of this richness.
link |
So the way that deep learning
link |
was deployed in cognitive psychology
link |
was that was the spirit of it.
link |
It was about that richness.
link |
And that's something that I always found very compelling,
link |
Is there something especially interesting
link |
and profound to you
link |
in terms of our current deep learning neural network,
link |
artificial neural network approaches,
link |
and whatever we do understand
link |
about the biological neural networks in our brain?
link |
Is there, there's quite a few differences.
link |
Are some of them to you,
link |
either interesting or perhaps profound
link |
in terms of the gap we might want to try to close
link |
in trying to create a human level intelligence?
link |
What I would say here is something
link |
that a lot of people are saying,
link |
which is that one seeming limitation
link |
of the systems that we're building now
link |
is that they lack the kind of flexibility,
link |
the readiness to sort of turn on a dime
link |
when the context calls for it
link |
that is so characteristic of human behavior.
link |
So is that connected to you to the,
link |
like which aspect of the neural networks in our brain
link |
is that connected to?
link |
Is that closer to the cognitive science level of,
link |
now again, see like my natural inclination
link |
is to separate into three disciplines of neuroscience,
link |
cognitive science and psychology.
link |
And you've already kind of shut that down
link |
by saying you're kind of see them as separate,
link |
but just to look at those layers,
link |
I guess where is there something about the lowest layer
link |
of the way the neural neurons interact
link |
that is profound to you in terms of this difference
link |
to the artificial neural networks,
link |
or is all the key differences
link |
at a higher level of abstraction?
link |
One thing I often think about is that,
link |
if you take an introductory computer science course
link |
and they are introducing you to the notion
link |
of Turing machines,
link |
one way of articulating
link |
what the significance of a Turing machine is,
link |
is that it's a machine emulator.
link |
It can emulate any other machine.
link |
that way of looking at a Turing machine
link |
really sticks with me.
link |
I think of humans as maybe sharing
link |
in some of that character.
link |
We're capacity limited,
link |
we're not Turing machines obviously,
link |
but we have the ability to adapt behaviors
link |
that are very much unlike anything we've done before,
link |
but there's some basic mechanism
link |
that's implemented in our brain
link |
that allows us to run software.
link |
But just on that point, you mentioned Turing machine,
link |
but nevertheless, it's fundamentally
link |
our brains are just computational devices in your view.
link |
Is that what you're getting at?
link |
It was a little bit unclear to this line you drew.
link |
Is there any magic in there
link |
or is it just basic computation?
link |
I'm happy to think of it as just basic computation,
link |
but mind you, I won't be satisfied
link |
until somebody explains to me
link |
what the basic computations are
link |
that are leading to the full richness of human cognition.
link |
It's not gonna be enough for me
link |
to understand what the computations are
link |
that allow people to do arithmetic or play chess.
link |
I want the whole thing.
link |
And a small tangent,
link |
because you kind of mentioned coronavirus,
link |
there's group behavior.
link |
Is there something interesting
link |
to your search of understanding the human mind
link |
where behavior of large groups
link |
or just behavior of groups is interesting,
link |
seeing that as a collective mind,
link |
as a collective intelligence,
link |
perhaps seeing the groups of people
link |
as a single intelligent organisms,
link |
especially looking at the reinforcement learning work
link |
you've done recently.
link |
Well, yeah, I can't.
link |
I mean, I have the honor of working
link |
with a lot of incredibly smart people
link |
and I wouldn't wanna take any credit
link |
for leading the way on the multiagent work
link |
that's come out of my group or DeepMind lately,
link |
but I do find it fascinating.
link |
And I mean, I think it can't be debated.
link |
You know, human behavior arises within communities.
link |
That just seems to me self evident.
link |
But to me, it is self evident,
link |
but that seems to be a profound aspects
link |
of something that created.
link |
That was like, if you look at like 2001 Space Odyssey
link |
when the monkeys touched the...
link |
That's the magical moment I think Yuval Harari argues
link |
that the ability of our large numbers of humans
link |
to hold an idea, to converge towards idea together,
link |
like you said, shaking hands versus bumping elbows,
link |
somehow converge without being in a room altogether,
link |
just kind of this like distributed convergence
link |
towards an idea over a particular period of time
link |
seems to be fundamental to just every aspect
link |
of our cognition, of our intelligence,
link |
because humans, I will talk about reward,
link |
but it seems like we don't really have
link |
a clear objective function under which we operate,
link |
but we all kind of converge towards one somehow.
link |
And that to me has always been a mystery
link |
that I think is somehow productive
link |
for also understanding AI systems.
link |
But I guess that's the next step.
link |
The first step is try to understand the mind.
link |
Well, I don't know.
link |
I mean, I think there's something to the argument
link |
that that kind of like strictly bottom up approach
link |
In other words, there are basic phenomena,
link |
basic aspects of human intelligence
link |
that can only be understood in the context of groups.
link |
I'm perfectly open to that.
link |
I've never been particularly convinced by the notion
link |
that we should consider intelligence
link |
to inhere at the level of communities.
link |
I don't know why, I'm sort of stuck on the notion
link |
that the basic unit that we want to understand
link |
is individual humans.
link |
And if we have to understand that
link |
in the context of other humans, fine.
link |
But for me, intelligence is just,
link |
I stubbornly define it as something
link |
that is an aspect of an individual human.
link |
That's just my, I don't know if that's a matter of taste.
link |
I'm with you, but that could be the reductionist dream
link |
of a scientist because you can understand a single human.
link |
It also is very possible that intelligence can only arise
link |
when there's multiple intelligences.
link |
When there's multiple sort of, it's a sad thing,
link |
if that's true, because it's very difficult to study.
link |
But if it's just one human,
link |
that one human would not be homosapien,
link |
would not become that intelligent.
link |
That's a possibility.
link |
One thing I will say along these lines
link |
is that I think a serious effort
link |
to understand human intelligence
link |
and maybe to build humanlike intelligence
link |
needs to pay just as much attention
link |
to the structure of the environment
link |
as to the structure of the cognizing system,
link |
whether it's a brain or an AI system.
link |
That's one thing I took away actually
link |
from my early studies with the pioneers
link |
of neural network research,
link |
people like Jay McClelland and John Cohen.
link |
The structure of cognition is really,
link |
it's only partly a function of the architecture of the brain
link |
and the learning algorithms that it implements.
link |
What really shapes it is the interaction of those things
link |
with the structure of the world
link |
in which those things are embedded.
link |
And that's especially important for,
link |
that's made most clear in reinforcement learning
link |
where the simulated environment is,
link |
you can only learn as much as you can simulate.
link |
And that's what DeepMind made very clear
link |
with the other aspect of the environment,
link |
which is the self play mechanism of the other agent,
link |
of the competitive behavior,
link |
which the other agent becomes the environment essentially.
link |
And that's, I mean, one of the most exciting ideas in AI
link |
is the self play mechanism that's able to learn successfully.
link |
There's a thing where competition is essential
link |
for learning, at least in that context.
link |
So if we can step back into another sort of beautiful world,
link |
which is the actual mechanics,
link |
the dirty mess of it of the human brain,
link |
is there something for people who might not know?
link |
Is there something you can comment on
link |
or describe the key parts of the brain
link |
that are important for intelligence or just in general,
link |
what are the different parts of the brain
link |
that you're curious about that you've studied
link |
and that are just good to know about
link |
when you're thinking about cognition?
link |
Well, my area of expertise, if I have one,
link |
is prefrontal cortex.
link |
So, you know. What's that?
link |
It depends on who you ask.
link |
The technical definition is anatomical.
link |
There are parts of your brain
link |
that are responsible for motor behavior
link |
and they're very easy to identify.
link |
And the region of your cerebral cortex,
link |
the sort of outer crust of your brain
link |
that lies in front of those
link |
is defined as the prefrontal cortex.
link |
And when you say anatomical, sorry to interrupt,
link |
so that's referring to sort of the geographic region
link |
as opposed to some kind of functional definition.
link |
Exactly, so this is kind of the coward's way out.
link |
I'm telling you what the prefrontal cortex is
link |
just in terms of what part of the real estate it occupies.
link |
It's the thing in the front of the brain.
link |
And in fact, the early history
link |
of neuroscientific investigation
link |
of what this front part of the brain does
link |
is sort of funny to read
link |
because it was really World War I
link |
that started people down this road
link |
of trying to figure out what different parts of the brain,
link |
the human brain do in the sense
link |
that there were a lot of people with brain damage
link |
who came back from the war with brain damage.
link |
And that provided, as tragic as that was,
link |
it provided an opportunity for scientists
link |
to try to identify the functions of different brain regions.
link |
And that was actually incredibly productive,
link |
but one of the frustrations that neuropsychologists faced
link |
was they couldn't really identify exactly
link |
what the deficit was that arose from damage
link |
to these most kind of frontal parts of the brain.
link |
It was just a very difficult thing to pin down.
link |
There were a couple of neuropsychologists
link |
who identified through a large amount
link |
of clinical experience and close observation,
link |
they started to put their finger on a syndrome
link |
that was associated with frontal damage.
link |
Actually, one of them was a Russian neuropsychologist
link |
named Luria, who students of cognitive psychology still read.
link |
And what he started to figure out was that
link |
the frontal cortex was somehow involved in flexibility,
link |
in guiding behaviors that required someone
link |
to override a habit, or to do something unusual,
link |
or to change what they were doing in a very flexible way
link |
from one moment to another.
link |
So focused on like new experiences.
link |
And so the way your brain processes
link |
and acts in new experiences.
link |
Yeah, what later helped bring this function
link |
into better focus was a distinction
link |
between controlled and automatic behavior,
link |
or in other literatures, this is referred to
link |
as habitual behavior versus goal directed behavior.
link |
So it's very, very clear that the human brain
link |
has pathways that are dedicated to habits,
link |
to things that you do all the time,
link |
and they need to be automatized
link |
so that they don't require you to concentrate too much.
link |
So that leaves your cognitive capacity
link |
free to do other things.
link |
Just think about the difference
link |
between driving when you're learning to drive
link |
versus driving after you're a fairly expert.
link |
There are brain pathways that slowly absorb
link |
those frequently performed behaviors
link |
so that they can be habits, so that they can be automatic.
link |
That's kind of like the purest form of learning.
link |
I guess it's happening there, which is why,
link |
I mean, this is kind of jumping ahead,
link |
which is why that perhaps is the most useful for us
link |
to focusing on and trying to see
link |
how artificial intelligence systems can learn.
link |
Is that the way you think?
link |
I do think about this distinction
link |
between controlled and automatic,
link |
or goal directed and habitual behavior a lot
link |
in thinking about where we are in AI research.
link |
But just to finish the kind of dissertation here,
link |
the role of the prefrontal cortex
link |
is generally understood these days
link |
sort of in contradistinction to that habitual domain.
link |
In other words, the prefrontal cortex
link |
is what helps you override those habits.
link |
It's what allows you to say,
link |
well, what I usually do in this situation is X,
link |
but given the context, I probably should do Y.
link |
I mean, the elbow bump is a great example, right?
link |
Reaching out and shaking hands
link |
is probably a habitual behavior,
link |
and it's the prefrontal cortex that allows us
link |
to bear in mind that there's something unusual
link |
going on right now, and in this situation,
link |
I need to not do the usual thing.
link |
The kind of behaviors that Luria reported,
link |
and he built tests for detecting these kinds of things,
link |
were exactly like this.
link |
So in other words, when I stick out my hand,
link |
I want you instead to present your elbow.
link |
A patient with frontal damage
link |
would have a great deal of trouble with that.
link |
Somebody proffering their hand would elicit a handshake.
link |
The prefrontal cortex is what allows us to say,
link |
hold on, hold on, that's the usual thing,
link |
but I have the ability to bear in mind
link |
even very unusual contexts and to reason about
link |
what behavior is appropriate there.
link |
Just to get a sense, are us humans special
link |
in the presence of the prefrontal cortex?
link |
Do mice have a prefrontal cortex?
link |
Do other mammals that we can study?
link |
If no, then how do they integrate new experiences?
link |
Yeah, that's a really tricky question
link |
and a very timely question
link |
because we have revolutionary new technologies
link |
for monitoring, measuring,
link |
and also causally influencing neural behavior
link |
in mice and fruit flies.
link |
And these techniques are not fully available
link |
even for studying brain function in monkeys,
link |
And so it's a very sort of, for me at least,
link |
a very urgent question whether the kinds of things
link |
that we wanna understand about human intelligence
link |
can be pursued in these other organisms.
link |
And to put it briefly, there's disagreement.
link |
People who study fruit flies will often tell you,
link |
hey, fruit flies are smarter than you think.
link |
And they'll point to experiments where fruit flies
link |
were able to learn new behaviors,
link |
were able to generalize from one stimulus to another
link |
in a way that suggests that they have abstractions
link |
that guide their generalization.
link |
I've had many conversations in which
link |
I will start by observing,
link |
recounting some observation about mouse behavior
link |
where it seemed like mice were taking an awfully long time
link |
to learn a task that for a human would be profoundly trivial.
link |
And I will conclude from that,
link |
that mice really don't have the cognitive flexibility
link |
that we want to explain.
link |
And then a mouse researcher will say to me,
link |
well, hold on, that experiment may not have worked
link |
because you asked a mouse to deal with stimuli
link |
and behaviors that were very unnatural for the mouse.
link |
If instead you kept the logic of the experiment the same,
link |
but presented the information in a way
link |
that aligns with what mice are used to dealing with
link |
in their natural habitats,
link |
you might find that a mouse actually has more intelligence
link |
And then they'll go on to show you videos
link |
of mice doing things in their natural habitat,
link |
which seem strikingly intelligent,
link |
dealing with physical problems.
link |
I have to drag this piece of food back to my lair,
link |
but there's something in my way
link |
and how do I get rid of that thing?
link |
So I think these are open questions
link |
to put it, to sum that up.
link |
And then taking a small step back related to that
link |
is you kind of mentioned we're taking a little shortcut
link |
by saying it's a geographic part of the prefrontal cortex
link |
is a region of the brain.
link |
But if we, what's your sense in a bigger philosophical view,
link |
prefrontal cortex and the brain in general,
link |
do you have a sense that it's a set of subsystems
link |
in the way we've kind of implied
link |
that are pretty distinct or to what degree is it that
link |
or to what degree is it a giant interconnected mess
link |
where everything kind of does everything
link |
and it's impossible to disentangle them?
link |
I think there's overwhelming evidence
link |
that there's functional differentiation,
link |
that it's clearly not the case
link |
that all parts of the brain are doing the same thing.
link |
This follows immediately from the kinds of studies
link |
of brain damage that we were chatting about before.
link |
It's obvious from what you see
link |
if you stick an electrode in the brain
link |
and measure what's going on at the level of neural activity.
link |
Having said that, there are two other things to add,
link |
which kind of, I don't know,
link |
maybe tug in the other direction.
link |
One is that it's when you look carefully
link |
at functional differentiation in the brain,
link |
what you usually end up concluding,
link |
at least this is my observation of the literature,
link |
is that the differences between regions are graded
link |
rather than being discreet.
link |
So it doesn't seem like it's easy
link |
to divide the brain up into true modules
link |
that have clear boundaries and that have
link |
you know, clear channels of communication between them.
link |
And this applies to the prefrontal cortex?
link |
The prefrontal cortex is made up
link |
of a bunch of different subregions,
link |
the functions of which are not clearly defined
link |
and the borders of which seem to be quite vague.
link |
And then there's another thing that's popping up
link |
in very recent research, which, you know, which,
link |
involves application of these new techniques,
link |
which there are a number of studies that suggest that
link |
parts of the brain that we would have previously thought
link |
were quite focused in their function
link |
are actually carrying signals
link |
that we wouldn't have thought would be there.
link |
For example, looking in the primary visual cortex,
link |
which is classically thought of as basically
link |
the first cortical way station
link |
for processing visual information.
link |
Basically what it should care about is, you know,
link |
where are the edges in this scene that I'm viewing?
link |
It turns out that if you have enough data,
link |
you can recover information from primary visual cortex
link |
about all sorts of things.
link |
Like, you know, what behavior the animal is engaged
link |
in right now and how much reward is on offer
link |
in the task that it's pursuing.
link |
So it's clear that even regions whose function
link |
is pretty well defined at a core screen
link |
are nonetheless carrying some information
link |
about information from very different domains.
link |
So, you know, the history of neuroscience
link |
is sort of this oscillation between the two views
link |
that you articulated, you know, the kind of modular view
link |
and then the big, you know, mush view.
link |
And, you know, I think, I guess we're gonna end up
link |
somewhere in the middle.
link |
Which is unfortunate for our understanding
link |
because there's something about our, you know,
link |
conceptual system that finds it's easy to think about
link |
a modularized system and easy to think about
link |
a completely undifferentiated system.
link |
But something that kind of lies in between is confusing.
link |
But we're gonna have to get used to it, I think.
link |
Unless we can understand deeply the lower level mechanism
link |
of neuronal communication.
link |
But on that topic, you kind of mentioned information.
link |
Just to get a sense, I imagine something
link |
that there's still mystery and disagreement on
link |
is how does the brain carry information and signal?
link |
Like what in your sense is the basic mechanism
link |
of communication in the brain?
link |
Well, I guess I'm old fashioned in that I consider
link |
the networks that we use in deep learning research
link |
to be a reasonable approximation to, you know,
link |
the mechanisms that carry information in the brain.
link |
So the usual way of articulating that is to say,
link |
what really matters is a rate code.
link |
What matters is how quickly is an individual neuron spiking?
link |
You know, what's the frequency at which it's spiking?
link |
So the timing of the spike.
link |
Yeah, is it firing fast or slow?
link |
Let's, you know, let's put a number on that.
link |
And that number is enough to capture
link |
what neurons are doing.
link |
There's, you know, there's still uncertainty
link |
about whether that's an adequate description
link |
of how information is transmitted within the brain.
link |
There, you know, there are studies that suggest
link |
that the precise timing of spikes matters.
link |
There are studies that suggest that there are computations
link |
that go on within the dendritic tree, within a neuron,
link |
that are quite rich and structured
link |
and that really don't equate to anything that we're doing
link |
in our artificial neural networks.
link |
Having said that, I feel like we can get,
link |
I feel like we're getting somewhere
link |
by sticking to this high level of abstraction.
link |
Just the rate, and by the way,
link |
we're talking about the electrical signal.
link |
I remember reading some vague paper somewhere recently
link |
where the mechanical signal, like the vibrations
link |
or something of the neurons, also communicates information.
link |
I haven't seen that, but.
link |
There's somebody who was arguing
link |
that the electrical signal, this is in a nature paper,
link |
something like that, where the electrical signal
link |
is actually a side effect of the mechanical signal.
link |
But I don't think that changes the story.
link |
But it's almost an interesting idea
link |
that there could be a deeper, it's always like in physics
link |
with quantum mechanics, there's always a deeper story
link |
that could be underlying the whole thing.
link |
But you think it's basically the rate of spiking
link |
that gets us, that's like the lowest hanging fruit
link |
that can get us really far.
link |
This is a classical view.
link |
I mean, this is not, the only way in which this stance
link |
would be controversial is in the sense
link |
that there are members of the neuroscience community
link |
who are interested in alternatives.
link |
But this is really a very mainstream view.
link |
The way that neurons communicate
link |
is that neurotransmitters arrive,
link |
they wash up on a neuron, the neuron has receptors
link |
for those transmitters, the meeting of the transmitter
link |
with these receptors changes the voltage of the neuron.
link |
And if enough voltage change occurs, then a spike occurs,
link |
one of these like discrete events.
link |
And it's that spike that is conducted down the axon
link |
and leads to neurotransmitter release.
link |
This is just like neuroscience 101.
link |
This is like the way the brain is supposed to work.
link |
Now, what we do when we build artificial neural networks
link |
of the kind that are now popular in the AI community
link |
is that we don't worry about those individual spikes.
link |
We just worry about the frequency
link |
at which those spikes are being generated.
link |
And people talk about that as the activity of a neuron.
link |
And so the activity of units in a deep learning system
link |
is broadly analogous to the spike rate of a neuron.
link |
There are people who believe that there are other forms
link |
of communication in the brain.
link |
In fact, I've been involved in some research recently
link |
that suggests that the voltage fluctuations
link |
that occur in populations of neurons
link |
that are sort of below the level of spike production
link |
may be important for communication.
link |
But I'm still pretty old school in the sense
link |
that I think that the things that we're building
link |
in AI research constitute reasonable models
link |
of how a brain would work.
link |
Let me ask just for fun a crazy question, because I can.
link |
Do you think it's possible we're completely wrong
link |
about the way this basic mechanism
link |
of neuronal communication, that the information
link |
is stored in some very different kind of way in the brain?
link |
I mean, look, I wouldn't be a scientist
link |
if I didn't think there was any chance we were wrong.
link |
But I mean, if you look at the history
link |
of deep learning research as it's been applied
link |
to neuroscience, of course the vast majority
link |
of deep learning research these days isn't about neuroscience.
link |
But if you go back to the 1980s,
link |
there's sort of an unbroken chain of research
link |
in which a particular strategy is taken,
link |
which is, hey, let's train a deep learning system.
link |
Let's train a multi layer neural network
link |
on this task that we trained our rat on,
link |
or our monkey on, or this human being on.
link |
And then let's look at what the units
link |
deep in the system are doing.
link |
And let's ask whether what they're doing
link |
resembles what we know about what neurons
link |
deep in the brain are doing.
link |
And over and over and over and over,
link |
that strategy works in the sense that
link |
the learning algorithms that we have access to,
link |
which typically center on back propagation,
link |
they give rise to patterns of activity,
link |
patterns of response,
link |
patterns of neuronal behavior in these artificial models
link |
that look hauntingly similar to what you see in the brain.
link |
And is that a coincidence?
link |
At a certain point, it starts looking like such coincidence
link |
is unlikely to not be deeply meaningful, yeah.
link |
Yeah, the circumstantial evidence is overwhelming.
link |
But you're always open to total flipping at the table.
link |
So you have coauthored several recent papers
link |
that sort of weave beautifully between the world
link |
of neuroscience and artificial intelligence.
link |
And maybe if we could, can we just try to dance around
link |
and talk about some of them?
link |
Maybe try to pick out interesting ideas
link |
that jump to your mind from memory.
link |
So maybe looking at, we were talking about
link |
the prefrontal cortex, the 2018, I believe, paper
link |
called the Prefrontal Cortex
link |
as a Meta Reinforcement Learning System.
link |
What, is there a key idea
link |
that you can speak to from that paper?
link |
Yeah, I mean, the key idea is about meta learning.
link |
What is meta learning?
link |
Meta learning is, by definition,
link |
a situation in which you have a learning algorithm
link |
and the learning algorithm operates in such a way
link |
that it gives rise to another learning algorithm.
link |
In the earliest applications of this idea,
link |
you had one learning algorithm sort of adjusting
link |
the parameters on another learning algorithm.
link |
But the case that we're interested in this paper
link |
is one where you start with just one learning algorithm
link |
and then another learning algorithm kind of emerges
link |
I can say more about what I mean by that.
link |
I don't mean to be scurrentist,
link |
but that's the idea of meta learning.
link |
It relates to the old idea in psychology
link |
of learning to learn.
link |
Situations where you have experiences
link |
that make you better at learning something new.
link |
A familiar example would be learning a foreign language.
link |
The first time you learn a foreign language,
link |
it may be quite laborious and disorienting
link |
and novel, but let's say you've learned
link |
two foreign languages.
link |
The third foreign language, obviously,
link |
is gonna be much easier to pick up.
link |
Because you've learned how to learn.
link |
You know how this goes.
link |
You know, okay, I'm gonna have to learn how to conjugate.
link |
I'm gonna have to...
link |
That's a simple form of meta learning
link |
in the sense that there's some slow learning mechanism
link |
that's helping you kind of update
link |
your fast learning mechanism.
link |
Does that make sense?
link |
So how from our understanding from the psychology world,
link |
from neuroscience, our understanding
link |
how meta learning might work in the human brain,
link |
what lessons can we draw from that
link |
that we can bring into the artificial intelligence world?
link |
Well, yeah, so the origin of that paper
link |
was in AI work that we were doing in my group.
link |
We were looking at what happens
link |
when you train a recurrent neural network
link |
using standard reinforcement learning algorithms.
link |
But you train that network, not just in one task,
link |
but you train it in a bunch of interrelated tasks.
link |
And then you ask what happens when you give it
link |
yet another task in that sort of line of interrelated tasks.
link |
And what we started to realize is that
link |
a form of meta learning spontaneously happens
link |
in recurrent neural networks.
link |
And the simplest way to explain it is to say
link |
a recurrent neural network has a kind of memory
link |
in its activation patterns.
link |
It's recurrent by definition in the sense
link |
that you have units that connect to other units,
link |
that connect to other units.
link |
So you have sort of loops of connectivity,
link |
which allows activity to stick around
link |
and be updated over time.
link |
In psychology we call, in neuroscience
link |
we call this working memory.
link |
It's like actively holding something in mind.
link |
And so that memory gives
link |
the recurrent neural network a dynamics, right?
link |
The way that the activity pattern evolves over time
link |
is inherent to the connectivity
link |
of the recurrent neural network, okay?
link |
So that's idea number one.
link |
Now, the dynamics of that network are shaped
link |
by the connectivity, by the synaptic weights.
link |
And those synaptic weights are being shaped
link |
by this reinforcement learning algorithm
link |
that you're training the network with.
link |
So the punchline is if you train a recurrent neural network
link |
with a reinforcement learning algorithm
link |
that's adjusting its weights,
link |
and you do that for long enough,
link |
the activation dynamics will become very interesting, right?
link |
So imagine I give you a task
link |
where you have to press one button or another,
link |
left button or right button.
link |
And there's some probability
link |
that I'm gonna give you an M&M
link |
if you press the left button,
link |
and there's some probability I'll give you an M&M
link |
if you press the other button.
link |
And you have to figure out what those probabilities are
link |
just by trying things out.
link |
But as I said before,
link |
instead of just giving you one of these tasks,
link |
I give you a whole sequence.
link |
You know, I give you two buttons
link |
and you figure out which one's best.
link |
And I go, good job, here's a new box.
link |
Two new buttons, you have to figure out which one's best.
link |
Good job, here's a new box.
link |
And every box has its own probabilities
link |
and you have to figure it out.
link |
So if you train a recurrent neural network
link |
on that kind of sequence of tasks,
link |
what happens, it seemed almost magical to us
link |
when we first started kind of realizing what was going on.
link |
The slow learning algorithm that's adjusting
link |
the synaptic weights,
link |
those slow synaptic changes give rise to a network dynamics
link |
that themselves, that, you know,
link |
the dynamics themselves turn into a learning algorithm.
link |
So in other words, you can tell this is happening
link |
by just freezing the synaptic weights saying,
link |
okay, no more learning, you're done.
link |
Here's a new box, figure out which button is best.
link |
And the recurrent neural network will do this just fine.
link |
There's no, like it figures out which button is best.
link |
It kind of transitions from exploring the two buttons
link |
to just pressing the one that it likes best
link |
in a very rational way.
link |
How is that happening?
link |
It's happening because the activity dynamics
link |
of the network have been shaped by the slow learning process
link |
that's occurred over many, many boxes.
link |
And so what's happened is that this slow learning algorithm
link |
that's slowly adjusting the weights
link |
is changing the dynamics of the network,
link |
the activity dynamics into its own learning algorithm.
link |
And as we were kind of realizing that this is a thing,
link |
it just so happened that the group that was working on this
link |
included a bunch of neuroscientists
link |
and it started kind of ringing a bell for us,
link |
which is to say that we thought this sounds a lot
link |
like the distinction between synaptic learning
link |
and activity, synaptic memory
link |
and activity based memory in the brain.
link |
And it also reminded us of recurrent connectivity
link |
that's very characteristic of prefrontal function.
link |
So this is kind of why it's good to have people working
link |
on AI that know a little bit about neuroscience
link |
and vice versa, because we started thinking
link |
about whether we could apply this principle to neuroscience.
link |
And that's where the paper came from.
link |
So the kind of principle of the recurrence
link |
they can see in the prefrontal cortex,
link |
then you start to realize that it's possible
link |
for something like an idea of a learning
link |
to learn emerging from this learning process
link |
as long as you keep varying the environment sufficiently.
link |
Exactly, so the kind of metaphorical transition
link |
we made to neuroscience was to think,
link |
okay, well, we know that the prefrontal cortex
link |
is highly recurrent.
link |
We know that it's an important locus for working memory
link |
for activation based memory.
link |
So maybe the prefrontal cortex
link |
supports reinforcement learning.
link |
In other words, what is reinforcement learning?
link |
You take an action, you see how much reward you got,
link |
you update your policy of behavior.
link |
Maybe the prefrontal cortex is doing that sort of thing
link |
strictly in its activation patterns.
link |
It's keeping around a memory in its activity patterns
link |
of what you did, how much reward you got,
link |
and it's using that activity based memory
link |
as a basis for updating behavior.
link |
But then the question is, well,
link |
how did the prefrontal cortex get so smart?
link |
In other words, where did these activity dynamics come from?
link |
How did that program that's implemented
link |
in the recurrent dynamics of the prefrontal cortex arise?
link |
And one answer that became evident in this work was,
link |
well, maybe the mechanisms that operate
link |
on the synaptic level, which we believe are mediated
link |
by dopamine, are responsible for shaping those dynamics.
link |
So this may be a silly question,
link |
but because this kind of several temporal sort of classes
link |
of learning are happening and the learning to learnism
link |
emerges, can you keep building stacks of learning
link |
to learn to learn, learning to learn to learn
link |
to learn to learn because it keeps,
link |
I mean, basically abstractions of more powerful abilities
link |
to generalize of learning complex rules.
link |
Yeah, that's overstretching this kind of mechanism.
link |
Well, one of the people in AI who started thinking
link |
about meta learning from very early on,
link |
Jürgen Schmidhuber sort of cheekily suggested,
link |
I think it may have been in his PhD thesis,
link |
that we should think about meta, meta, meta,
link |
meta, meta, meta learning.
link |
That's really what's gonna get us to true intelligence.
link |
Certainly there's a poetic aspect to it
link |
and it seems interesting and correct
link |
that that kind of levels of abstraction would be powerful,
link |
but is that something you see in the brain?
link |
This kind of, is it useful to think of learning
link |
in these meta, meta, meta way or is it just meta learning?
link |
Well, one thing that really fascinated me
link |
about this mechanism that we were starting to look at,
link |
and other groups started talking
link |
about very similar things at the same time.
link |
And then a kind of explosion of interest
link |
in meta learning happened in the AI community
link |
shortly after that.
link |
I don't know if we had anything to do with that,
link |
but I was gratified to see that a lot of people
link |
started talking about meta learning.
link |
One of the things that I liked about the kind of flavor
link |
of meta learning that we were studying was that
link |
it didn't require anything special.
link |
It was just, if you took a system that had
link |
some form of memory that the function of which
link |
could be shaped by pick URL algorithm,
link |
then this would just happen, right?
link |
I mean, there are a lot of forms of,
link |
there are a lot of meta learning algorithms
link |
that have been proposed since then
link |
that are fascinating and effective
link |
in their domains of application.
link |
But they're engineered, they're things that somebody
link |
had to say, well, gee, if we wanted meta learning
link |
to happen, how would we do that?
link |
Here's an algorithm that would,
link |
but there's something about the kind of meta learning
link |
that we were studying that seemed to me special
link |
in the sense that it wasn't an algorithm.
link |
It was just something that automatically happened
link |
if you had a system that had memory
link |
and it was trained with a reinforcement learning algorithm.
link |
And in that sense, it can be as meta as it wants to be.
link |
There's no limit on how abstract the meta learning can get
link |
because it's not reliant on a human engineering
link |
a particular meta learning algorithm to get there.
link |
And that's, I also, I don't know,
link |
I guess I hope that that's relevant in the brain.
link |
I think there's a kind of beauty
link |
in the ability of this emergent.
link |
The emergent aspect of it, as opposed to engineered.
link |
Exactly, it's something that just, it just happens
link |
in a sense, in a sense, you can't avoid this happening.
link |
If you have a system that has memory
link |
and the function of that memory is shaped
link |
by reinforcement learning, and this system is trained
link |
in a series of interrelated tasks, this is gonna happen.
link |
You can't stop it.
link |
As long as you have certain properties,
link |
maybe like a recurrent structure to.
link |
You have to have memory.
link |
It actually doesn't have to be a recurrent neural network.
link |
One of, a paper that I was honored to be involved
link |
with even earlier, used a kind of slot based memory.
link |
Do you remember the title?
link |
Just for people to understand.
link |
It was Memory Augmented Neural Networks.
link |
I think it was, I think the title was
link |
Meta Learning in Memory Augmented Neural Networks.
link |
And it was the same exact story.
link |
If you have a system with memory,
link |
here it was a different kind of memory,
link |
but the function of that memory is shaped
link |
by reinforcement learning.
link |
Here it was the reads and writes that occurred
link |
on this slot based memory.
link |
This will just happen.
link |
But this brings us back to something I was saying earlier
link |
about the importance of the environment.
link |
This will happen if the system is being trained
link |
in a setting where there's like a sequence of tasks
link |
that all share some abstract structure.
link |
Sometimes we talk about task distributions.
link |
And that's something that's very obviously true
link |
of the world that humans inhabit.
link |
Like if you just kind of think about what you do every day,
link |
you never do exactly the same thing
link |
that you did the day before.
link |
But everything that you do sort of has a family resemblance.
link |
It shares a structure with something that you did before.
link |
And so the real world is sort of
link |
saturated with this kind of, this property.
link |
It's endless variety with endless redundancy.
link |
And that's the setting in which
link |
this kind of meta learning happens.
link |
And it does seem like we're just so good at finding,
link |
just like in this emergent phenomena you described,
link |
we're really good at finding that redundancy,
link |
finding those similarities, the family resemblance.
link |
Some people call it sort of, what is it?
link |
Melanie Mitchell was talking about analogies.
link |
So we're able to connect concepts together
link |
in this kind of way,
link |
in this same kind of automated emergent way,
link |
which there's so many echoes here
link |
of psychology and neuroscience.
link |
And obviously now with reinforcement learning
link |
with recurrent neural networks at the core.
link |
If we could talk a little bit about dopamine,
link |
you have really, you're a part of coauthoring
link |
really exciting recent paper, very recent,
link |
in terms of release on dopamine
link |
and temporal difference learning.
link |
Can you describe the key ideas of that paper?
link |
I mean, one thing I want to pause to do
link |
is acknowledge my coauthors
link |
on actually both of the papers we're talking about.
link |
So this dopamine paper.
link |
I'll just, I'll certainly post all their names.
link |
Yeah, because I'm sort of abashed
link |
to be the spokesperson for these papers
link |
when I had such amazing collaborators on both.
link |
So it's a comfort to me to know
link |
that you'll acknowledge them.
link |
Yeah, there's an incredible team there, but yeah.
link |
Oh yeah, it's such a, it's so much fun.
link |
And in the case of the dopamine paper,
link |
we also collaborated with Naochit at Harvard,
link |
who, you know, obviously a paper simply
link |
wouldn't have happened without him.
link |
But so you were asking for like a thumbnail sketch of.
link |
Yeah, thumbnail sketch or key ideas or, you know,
link |
things, the insights that are, you know,
link |
continuing on our kind of discussion here
link |
between neuroscience and AI.
link |
Yeah, I mean, this was another,
link |
a lot of the work that we've done so far
link |
is taking ideas that have bubbled up in AI
link |
and, you know, asking the question of whether the brain
link |
might be doing something related,
link |
which I think on the surface sounds like something
link |
that's really mainly of use to neuroscience.
link |
We see it also as a way of validating
link |
what we're doing on the AI side.
link |
If we can gain some evidence that the brain
link |
is using some technique that we've been trying out
link |
in our AI work, that gives us confidence
link |
that, you know, it may be a good idea,
link |
that it'll, you know, scale to rich, complex tasks,
link |
that it'll interface well with other mechanisms.
link |
So you see it as a two way road.
link |
Yeah, for sure. Just because a particular paper
link |
is a little bit focused on from one to the,
link |
from AI, from neural networks to neuroscience.
link |
Ultimately the discussion, the thinking,
link |
the productive longterm aspect of it
link |
is the two way road nature of the whole interaction.
link |
Yeah, I mean, we've talked about the notion
link |
of a virtuous circle between AI and neuroscience.
link |
And, you know, the way I see it,
link |
that's always been there since the two fields,
link |
you know, jointly existed.
link |
There have been some phases in that history
link |
when AI was sort of ahead.
link |
There are some phases when neuroscience was sort of ahead.
link |
I feel like given the burst of innovation
link |
that's happened recently on the AI side,
link |
AI is kind of ahead in the sense that
link |
there are all of these ideas that we, you know,
link |
for which it's exciting to consider
link |
that there might be neural analogs.
link |
And neuroscience, you know,
link |
in a sense has been focusing on approaches
link |
to studying behavior that come from, you know,
link |
that are kind of derived from this earlier era
link |
of cognitive psychology.
link |
And, you know, so in some ways fail to connect
link |
with some of the issues that we're grappling with in AI.
link |
Like how do we deal with, you know,
link |
large, you know, complex environments.
link |
But, you know, I think it's inevitable
link |
that this circle will keep turning
link |
and there will be a moment
link |
in the not too different distant future
link |
when neuroscience is pelting AI researchers
link |
with insights that may change the direction of our work.
link |
Just a quick human question.
link |
Is it, you have parts of your brain,
link |
this is very meta, but they're able to both think
link |
about neuroscience and AI.
link |
You know, I don't often meet people like that.
link |
So do you think, let me ask a meta plasticity question.
link |
Do you think a human being can be both good at AI
link |
It's like what, on the team at DeepMind,
link |
what kind of human can occupy these two realms?
link |
And is that something you see everybody should be doing,
link |
can be doing, or is that a very special few
link |
Just like we talk about art history,
link |
I would think it's a special person
link |
that can major in art history
link |
and also consider being a surgeon.
link |
Otherwise known as a dilettante.
link |
A dilettante, yeah.
link |
Easily distracted.
link |
No, I think it does take a special kind of person
link |
to be truly world class at both AI and neuroscience.
link |
And I am not on that list.
link |
I happen to be someone whose interest in neuroscience
link |
and psychology involved using the kinds
link |
of modeling techniques that are now very central in AI.
link |
And that sort of, I guess, bought me a ticket
link |
to be involved in all of the amazing things
link |
that are going on in AI research right now.
link |
I do know a few people who I would consider
link |
pretty expert on both fronts,
link |
and I won't embarrass them by naming them,
link |
but there are exceptional people out there
link |
who are like this.
link |
The one thing that I find is a barrier
link |
to being truly world class on both fronts
link |
is just the complexity of the technology
link |
that's involved in both disciplines now.
link |
So the engineering expertise that it takes
link |
to do truly frontline, hands on AI research
link |
is really, really considerable.
link |
The learning curve of the tools,
link |
just like the specifics of just whether it's programming
link |
or the kind of tools necessary to collect the data,
link |
to manage the data, to distribute, to compute,
link |
all that kind of stuff.
link |
And on the neuroscience, I guess, side,
link |
there'll be all different sets of tools.
link |
Exactly, especially with the recent explosion
link |
in neuroscience methods.
link |
So having said all that,
link |
I think the best scenario for both neuroscience
link |
and AI is to have people interacting
link |
who live at every point on this spectrum
link |
from exclusively focused on neuroscience
link |
to exclusively focused on the engineering side of AI.
link |
But to have those people inhabiting a community
link |
where they're talking to people who live elsewhere
link |
And I may be someone who's very close to the center
link |
in the sense that I have one foot in the neuroscience world
link |
and one foot in the AI world,
link |
and that central position, I will admit,
link |
prevents me, at least someone
link |
with my limited cognitive capacity,
link |
from having true technical expertise in either domain.
link |
But at the same time, I at least hope
link |
that it's worthwhile having people around
link |
who can kind of see the connections.
link |
Yeah, the community, the emergent intelligence
link |
of the community when it's nicely distributed is useful.
link |
So hopefully that, I mean, I've seen that work,
link |
I've seen that work out well at DeepMind.
link |
There are people who, I mean, even if you just focus
link |
on the AI work that happens at DeepMind,
link |
it's been a good thing to have some people around
link |
doing that kind of work whose PhDs are in neuroscience
link |
Every academic discipline has its kind of blind spots
link |
and kind of unfortunate obsessions and its metaphors
link |
and its reference points,
link |
and having some intellectual diversity is really healthy.
link |
People get each other unstuck, I think.
link |
I see it all the time at DeepMind.
link |
And I like to think that the people
link |
who bring some neuroscience background to the table
link |
are helping with that.
link |
So one of my probably the deepest passion for me,
link |
what I would say, maybe we kind of spoke off mic
link |
a little bit about it, but that I think is a blind spot
link |
for at least robotics and AI folks
link |
is human robot interaction, human agent interaction.
link |
Maybe do you have thoughts about how we reduce the size
link |
of that blind spot?
link |
Do you also share the feeling that not enough folks
link |
are studying this aspect of interaction?
link |
Well, I'm actually pretty intensively interested
link |
in this issue now, and there are people in my group
link |
who've actually pivoted pretty hard over the last few years
link |
from doing more traditional cognitive psychology
link |
and cognitive neuroscience to doing experimental work
link |
on human agent interaction.
link |
And there are a couple of reasons that I'm
link |
pretty passionately interested in this.
link |
One is it's kind of the outcome of having thought
link |
for a few years now about what we're up to.
link |
Like what are we doing?
link |
Like what is this AI research for?
link |
So what does it mean to make the world a better place?
link |
I think I'm pretty sure that means making life better
link |
And so how do you make life better for humans?
link |
That's a proposition that when you look at it carefully
link |
and honestly is rather horrendously complicated,
link |
especially when the AI systems
link |
that you're building are learning systems.
link |
They're not, you're not programming something
link |
that you then introduce to the world
link |
and it just works as programmed,
link |
like Google Maps or something.
link |
We're building systems that learn from experience.
link |
So that typically leads to AI safety questions.
link |
How do we keep these things from getting out of control?
link |
How do we keep them from doing things that harm humans?
link |
And I mean, I hasten to say,
link |
I consider those hugely important issues.
link |
And there are large sectors of the research community
link |
at DeepMind and of course elsewhere
link |
who are dedicated to thinking hard all day,
link |
every day about that.
link |
But there's, I guess I would say a positive side to this too
link |
which is to say, well, what would it mean
link |
to make human life better?
link |
And how can we imagine learning systems doing that?
link |
And in talking to my colleagues about that,
link |
we reached the initial conclusion
link |
that it's not sufficient to philosophize about that.
link |
You actually have to take into account
link |
how humans actually work and what humans want
link |
and the difficulties of knowing what humans want
link |
and the difficulties that arise
link |
when humans want different things.
link |
And so human agent interaction has become,
link |
a quite intensive focus of my group lately.
link |
If for no other reason that,
link |
in order to really address that issue in an adequate way,
link |
you have to, I mean, psychology becomes part of the picture.
link |
Yeah, and so there's a few elements there.
link |
So if you focus on solving like the,
link |
if you focus on the robotics problem,
link |
let's say AGI without humans in the picture
link |
is you're missing fundamentally the final step.
link |
When you do want to help human civilization,
link |
you eventually have to interact with humans.
link |
And when you create a learning system, just as you said,
link |
that will eventually have to interact with humans,
link |
the interaction itself has to be become,
link |
has to become part of the learning process.
link |
So you can't just watch, well, my sense is,
link |
it sounds like your sense is you can't just watch humans
link |
to learn about humans.
link |
You have to also be part of the human world.
link |
You have to interact with humans.
link |
And I mean, then questions arise that start imperceptibly,
link |
but inevitably to slip beyond the realm of engineering.
link |
So questions like, if you have an agent
link |
that can do something that you can't do,
link |
under what conditions do you want that agent to do it?
link |
So if I have a robot that can play Beethoven sonatas
link |
better than any human, in the sense that the sensitivity,
link |
the expression is just beyond what any human,
link |
do I want to listen to that?
link |
Do I want to go to a concert and hear a robot play?
link |
These aren't engineering questions.
link |
These are questions about human preference
link |
and human culture.
link |
Psychology bordering on philosophy.
link |
Yeah, and then you start asking,
link |
well, even if we knew the answer to that,
link |
is it our place as AI engineers
link |
to build that into these agents?
link |
Probably the agents should interact with humans
link |
beyond the population of AI engineers
link |
and figure out what those humans want.
link |
And then when you start,
link |
I referred this the moment ago,
link |
but even that becomes complicated.
link |
Be quote, what if two humans want different things?
link |
And you have only one agent that's able to interact with them
link |
and try to satisfy their preferences.
link |
Then you're into the realm of economics
link |
and social choice theory and even politics.
link |
So there's a sense in which,
link |
if you kind of follow what we're doing
link |
to its logical conclusion,
link |
then it goes beyond questions of engineering and technology
link |
and starts to shade imperceptibly into questions
link |
about what kind of society do you want?
link |
And actually, once that dawned on me,
link |
I don't know what the right word is,
link |
quite refreshed in my involvement in AI research.
link |
It was almost like building this kind of stuff
link |
is gonna lead us back to asking really fundamental questions
link |
about what is this,
link |
what's the good life and who gets to decide
link |
and bringing in viewpoints from multiple sub communities
link |
to help us shape the way that we live.
link |
There's something, it started making me feel like
link |
doing AI research in a fully responsible way, would,
link |
could potentially lead to a kind of like cultural renewal.
link |
Yeah, it's the way to understand human beings
link |
at the individual, at the societal level.
link |
It may become a way to answer all the silly human questions
link |
of the meaning of life and all those kinds of things.
link |
Even if it doesn't give us a way
link |
of answering those questions,
link |
it may force us back to thinking about them.
link |
And it might bring, it might restore a certain,
link |
I don't know, a certain depth to,
link |
or even dare I say spirituality to the way that,
link |
to the world, I don't know.
link |
Maybe that's too grandiose.
link |
Well, I'm with you.
link |
I think it's AI will be the philosophy of the 21st century,
link |
the way which will open the door.
link |
I think a lot of AI researchers are afraid to open that door
link |
of exploring the beautiful richness
link |
of the human agent interaction, human AI interaction.
link |
I'm really happy that somebody like you
link |
have opened that door.
link |
And one thing I often think about is the usual schema
link |
for thinking about human agent interaction
link |
as this kind of dystopian, oh, our robot overlords.
link |
And again, I hasten to say AI safety is hugely important.
link |
And I'm not saying we shouldn't be thinking
link |
about those risks, totally on board for that.
link |
But there's, having said that,
link |
what often follows for me is the thought
link |
that there's another kind of narrative
link |
that might be relevant, which is,
link |
when we think of humans gaining more and more information
link |
about human life, the narrative there is usually
link |
that they gain more and more wisdom
link |
and they get closer to enlightenment
link |
and they become more benevolent.
link |
And the Buddha is like, that's a totally different narrative.
link |
And why isn't it the case that we imagine
link |
that the AI systems that we're creating
link |
are just gonna, like, they're gonna figure out
link |
more and more about the way the world works
link |
and the way that humans interact
link |
and they'll become beneficent.
link |
I'm not saying that will happen.
link |
I don't honestly expect that to happen
link |
without some careful, setting things up very carefully.
link |
But it's another way things could go, right?
link |
And yeah, and I would even push back on that.
link |
I personally believe that the most trajectories,
link |
natural human trajectories will lead us towards progress.
link |
So for me, there is a kind of sense
link |
that most trajectories in AI development
link |
will lead us into trouble.
link |
To me, and we over focus on the worst case.
link |
It's like in computer science,
link |
theoretical computer science has been this focus
link |
on worst case analysis.
link |
There's something appealing to our human mind
link |
at some lowest level to be good.
link |
I mean, we don't wanna be eaten by the tiger, I guess.
link |
So we wanna do the worst case analysis.
link |
But the reality is that shouldn't stop us
link |
from actually building out all the other trajectories
link |
which are potentially leading to all the positive worlds,
link |
all the enlightenment.
link |
There's a book, Enlightenment Now,
link |
with Steven Pinker and so on.
link |
This is looking generally at human progress.
link |
And there's so many ways that human progress
link |
can happen with AI.
link |
And I think you have to do that research.
link |
You have to do that work.
link |
You have to do the, not just the AI safety work
link |
of the one worst case analysis.
link |
How do we prevent that?
link |
But the actual tools and the glue
link |
and the mechanisms of human AI interaction
link |
that would lead to all the positive actions that can go.
link |
It's a super exciting area, right?
link |
Yeah, we should be spending,
link |
we should be spending a lot of our time saying
link |
what can go wrong.
link |
I think it's harder to see that there's work to be done
link |
to bring into focus the question of what it would look like
link |
for things to go right.
link |
That's not obvious.
link |
And we wouldn't be doing this if we didn't have the sense
link |
there was huge potential, right?
link |
We're not doing this for no reason.
link |
We have a sense that AGI would be a major boom to humanity.
link |
But I think it's worth starting now,
link |
even when our technology is quite primitive,
link |
asking exactly what would that mean?
link |
We can start now with applications
link |
that are already gonna make the world a better place,
link |
like solving protein folding.
link |
I think DeepMind has gotten heavy
link |
into science applications lately,
link |
which I think is a wonderful, wonderful move
link |
for us to be making.
link |
But when we think about AGI,
link |
when we think about building fully intelligent
link |
agents that are gonna be able to, in a sense,
link |
do whatever they want,
link |
we should start thinking about
link |
what do we want them to want, right?
link |
What kind of world do we wanna live in?
link |
That's not an easy question.
link |
And I think we just need to start working on it.
link |
And even on the path to,
link |
it doesn't have to be AGI,
link |
but just intelligent agents that interact with us
link |
and help us enrich our own existence on social networks,
link |
for example, on recommender systems of various intelligence.
link |
And there's so much interesting interaction
link |
that's yet to be understood and studied.
link |
And how do you create,
link |
I mean, Twitter is struggling with this very idea,
link |
how do you create AI systems
link |
that increase the quality and the health of a conversation?
link |
That's a beautiful human psychology question.
link |
And how do you do that
link |
without deception being involved,
link |
without manipulation being involved,
link |
maximizing human autonomy?
link |
And how do you make these choices in a democratic way?
link |
How do we face the,
link |
again, I'm speaking for myself here.
link |
How do we face the fact that
link |
it's a small group of people
link |
who have the skillset to build these kinds of systems,
link |
but what it means to make the world a better place
link |
is something that we all have to be talking about.
link |
Yeah, the world that we're trying to make a better place
link |
includes a huge variety of different kinds of people.
link |
Yeah, how do we cope with that?
link |
This is a problem that has been discussed
link |
in gory, extensive detail in social choice theory.
link |
One thing I'm really interested in
link |
and one thing I'm really enjoying
link |
about the recent direction work has taken
link |
in some parts of my team is that,
link |
yeah, we're reading the AI literature,
link |
we're reading the neuroscience literature,
link |
but we've also started reading economics
link |
and, as I mentioned, social choice theory,
link |
even some political theory,
link |
because it turns out that it all becomes relevant.
link |
It all becomes relevant.
link |
But at the same time,
link |
we've been trying not to write philosophy papers,
link |
we've been trying not to write physician papers.
link |
We're trying to figure out ways
link |
of doing actual empirical research
link |
that kind of take the first small steps
link |
to thinking about what it really means
link |
for humans with all of their complexity
link |
and contradiction and paradox
link |
to be brought into contact with these AI systems
link |
in a way that really makes the world a better place.
link |
Often, reinforcement learning frameworks
link |
actually kind of allow you to do that,
link |
machine learning, and so that's the exciting thing about AI
link |
is it allows you to reduce the unsolvable problem,
link |
philosophical problem, into something more concrete
link |
that you can get ahold of.
link |
Yeah, and it allows you to kind of define the problem
link |
in some way that allows for growth in the system
link |
that's sort of, you know,
link |
you're not responsible for the details, right?
link |
You say, this is generally what I want you to do,
link |
and then learning takes care of the rest.
link |
Of course, the safety issues arise in that context,
link |
but I think also some of these positive issues
link |
arise in that context.
link |
What would it mean for an AI system
link |
to really come to understand what humans want?
link |
And with all of the subtleties of that, right?
link |
You know, humans want help with certain things,
link |
but they don't want everything done for them, right?
link |
There is, part of the satisfaction
link |
that humans get from life is in accomplishing things.
link |
So if there were devices around that did everything for,
link |
you know, I often think of the movie WALLI, right?
link |
That's like dystopian in a totally different way.
link |
It's like, the machines are doing everything for us.
link |
That's not what we wanted.
link |
You know, anyway, I find this, you know,
link |
this opens up a whole landscape of research
link |
that feels affirmative and exciting.
link |
To me, it's one of the most exciting, and it's wide open.
link |
We have to, because it's a cool paper,
link |
talk about dopamine.
link |
Oh yeah, okay, so I can.
link |
We were gonna, I was gonna give you a quick summary.
link |
Yeah, a quick summary of, what's the title of the paper?
link |
I think we called it a distributional code for value
link |
in dopamine based reinforcement learning, yes.
link |
So that's another project that grew out of pure AI research.
link |
A number of people at DeepMind and a few other places
link |
had started working on a new version
link |
of reinforcement learning,
link |
which was defined by taking something
link |
in traditional reinforcement learning and just tweaking it.
link |
So the thing that they took
link |
from traditional reinforcement learning was a value signal.
link |
So at the center of reinforcement learning,
link |
at least most algorithms, is some representation
link |
of how well things are going,
link |
your expected cumulative future reward.
link |
And that's usually represented as a single number.
link |
So if you imagine a gambler in a casino
link |
and the gambler's thinking, well, I have this probability
link |
of winning such and such an amount of money,
link |
and I have this probability of losing such and such
link |
an amount of money, that situation would be represented
link |
as a single number, which is like the expected,
link |
the weighted average of all those outcomes.
link |
And this new form of reinforcement learning said,
link |
well, what if we generalize that
link |
to a distributional representation?
link |
So now we think of the gambler as literally thinking,
link |
well, there's this probability
link |
that I'll win this amount of money,
link |
and there's this probability
link |
that I'll lose that amount of money,
link |
and we don't reduce that to a single number.
link |
And it had been observed through experiments,
link |
through just trying this out,
link |
that that kind of distributional representation
link |
really accelerated reinforcement learning
link |
and led to better policies.
link |
What's your intuition about,
link |
so we're talking about rewards.
link |
So what's your intuition why that is, why does it do that?
link |
Well, it's kind of a surprising historical note,
link |
at least surprised me when I learned it,
link |
that this had been proven to be true.
link |
This had been tried out in a kind of heuristic way.
link |
People thought, well, gee, what would happen if we tried?
link |
And then it had this, empirically,
link |
it had this striking effect.
link |
And it was only then that people started thinking,
link |
well, gee, wait, why?
link |
Why is this working?
link |
And that's led to a series of studies
link |
just trying to figure out why it works, which is ongoing.
link |
But one thing that's already clear from that research
link |
is that one reason that it helps
link |
is that it drives richer representation learning.
link |
So if you imagine two situations
link |
that have the same expected value,
link |
the same kind of weighted average value,
link |
standard deep reinforcement learning algorithms
link |
are going to take those two situations
link |
and kind of, in terms of the way
link |
they're represented internally,
link |
they're gonna squeeze them together
link |
because the thing that you're trying to represent,
link |
which is their expected value, is the same.
link |
So all the way through the system,
link |
things are gonna be mushed together.
link |
But what if those two situations
link |
actually have different value distributions?
link |
They have the same average value,
link |
but they have different distributions of value.
link |
In that situation, distributional learning
link |
will maintain the distinction between these two things.
link |
So to make a long story short,
link |
distributional learning can keep things separate
link |
in the internal representation
link |
that might otherwise be conflated or squished together.
link |
And maintaining those distinctions
link |
can be useful when the system is now faced
link |
with some other task where the distinction is important.
link |
If we look at the optimistic
link |
and pessimistic dopamine neurons.
link |
So first of all, what is dopamine?
link |
Why is this at all useful
link |
to think about in the artificial intelligence sense?
link |
But what do we know about dopamine in the human brain?
link |
Why is it interesting?
link |
What does it have to do with the prefrontal cortex
link |
and learning in general?
link |
Yeah, so, well, this is also a case
link |
where there's a huge amount of detail and debate.
link |
But one currently prevailing idea
link |
is that the function of this neurotransmitter dopamine
link |
resembles a particular component
link |
of standard reinforcement learning algorithms,
link |
which is called the reward prediction error.
link |
So I was talking a moment ago
link |
about these value representations.
link |
How do you learn them?
link |
How do you update them based on experience?
link |
Well, if you made some prediction about a future reward
link |
and then you get more reward than you were expecting,
link |
then probably retrospectively,
link |
you want to go back and increase the value representation
link |
that you attached to that earlier situation.
link |
If you got less reward than you were expecting,
link |
you should probably decrement that estimate.
link |
And that's the process of temporal difference.
link |
Exactly, this is the central mechanism
link |
of temporal difference learning,
link |
which is one of several sort of the backbone
link |
of our momentarium in NRL.
link |
And this connection between the reward prediction error
link |
and dopamine was made in the 1990s.
link |
And there's been a huge amount of research
link |
that seems to back it up.
link |
Dopamine may be doing other things,
link |
but this is clearly, at least roughly,
link |
one of the things that it's doing.
link |
But the usual idea was that dopamine
link |
was representing these reward prediction errors,
link |
again, in this like kind of single number way
link |
that representing your surprise with a single number.
link |
And in distributional reinforcement learning,
link |
this kind of new elaboration of the standard approach,
link |
it's not only the value function
link |
that's represented as a single number,
link |
it's also the reward prediction error.
link |
And so what happened was that Will Dabney,
link |
one of my collaborators who was one of the first people
link |
to work on distributional temporal difference learning,
link |
talked to a guy in my group, Zeb Kurt Nelson,
link |
who's a computational neuroscientist,
link |
and said, gee, you know, is it possible
link |
that dopamine might be doing something
link |
like this distributional coding thing?
link |
And they started looking at what was in the literature,
link |
and then they brought me in,
link |
and we started talking to Nao Uchida,
link |
and we came up with some specific predictions
link |
about if the brain is using
link |
this kind of distributional coding,
link |
then in the tasks that Nao has studied,
link |
you should see this, this, this, and this,
link |
and that's where the paper came from.
link |
We kind of enumerated a set of predictions,
link |
all of which ended up being fairly clearly confirmed,
link |
and all of which leads to at least some initial indication
link |
that the brain might be doing something
link |
like this distributional coding,
link |
that dopamine might be representing surprise signals
link |
in a way that is not just collapsing everything
link |
to a single number, but instead is kind of respecting
link |
the variety of future outcomes, if that makes sense.
link |
So yeah, so that's showing, suggesting possibly
link |
that dopamine has a really interesting
link |
representation scheme in the human brain
link |
for its reward signal.
link |
Exactly. That's fascinating.
link |
That's another beautiful example of AI
link |
revealing something nice about neuroscience,
link |
potentially suggesting possibilities.
link |
Well, you never know.
link |
So the minute you publish a paper like that,
link |
the next thing you think is, I hope that replicates.
link |
Like, I hope we see that same thing in other data sets,
link |
but of course, several labs now
link |
are doing the followup experiments, so we'll know soon.
link |
But it has been a lot of fun for us
link |
to take these ideas from AI
link |
and kind of bring them into neuroscience
link |
and see how far we can get.
link |
So we kind of talked about it a little bit,
link |
but where do you see the field of neuroscience
link |
and artificial intelligence heading broadly?
link |
Like, what are the possible exciting areas
link |
that you can see breakthroughs in the next,
link |
let's get crazy, not just three or five years,
link |
but the next 10, 20, 30 years
link |
that would make you excited
link |
and perhaps you'd be part of?
link |
On the neuroscience side,
link |
there's a great deal of interest now
link |
in what's going on in AI.
link |
And at the same time,
link |
I feel like, so neuroscience,
link |
especially the part of neuroscience
link |
that's focused on circuits and systems,
link |
kind of like really mechanism focused,
link |
there's been this explosion in new technology.
link |
And up until recently,
link |
the experiments that have exploited this technology
link |
have not involved a lot of interesting behavior.
link |
And this is for a variety of reasons,
link |
one of which is in order to employ
link |
some of these technologies,
link |
you actually have to, if you're studying a mouse,
link |
you have to head fix the mouse.
link |
In other words, you have to like immobilize the mouse.
link |
And so it's been tricky to come up
link |
with ways of eliciting interesting behavior
link |
from a mouse that's restrained in this way,
link |
but people have begun to create
link |
very interesting solutions to this,
link |
like virtual reality environments
link |
where the animal can kind of move a track ball.
link |
And as people have kind of begun to explore
link |
what you can do with these technologies,
link |
I feel like more and more people are asking,
link |
well, let's try to bring behavior into the picture.
link |
Let's try to like reintroduce behavior,
link |
which was supposed to be what this whole thing was about.
link |
And I'm hoping that those two trends,
link |
the kind of growing interest in behavior
link |
and the widespread interest in what's going on in AI,
link |
will come together to kind of open a new chapter
link |
in neuroscience research where there's a kind of
link |
a rebirth of interest in the structure of behavior
link |
and its underlying substrates,
link |
but that that research is being informed
link |
by computational mechanisms
link |
that we're coming to understand in AI.
link |
If we can do that, then we might be taking a step closer
link |
to this utopian future that we were talking about earlier
link |
where there's really no distinction
link |
between psychology and neuroscience.
link |
Neuroscience is about studying the mechanisms
link |
that underlie whatever it is the brain is for,
link |
and what is the brain for?
link |
What is the brain for? It's for behavior.
link |
I feel like we could maybe take a step toward that now
link |
if people are motivated in the right way.
link |
You also asked about AI.
link |
So that was a neuroscience question.
link |
You said neuroscience, that's right.
link |
And especially places like DeepMind
link |
are interested in both branches.
link |
So what about the engineering of intelligence systems?
link |
I think one of the key challenges
link |
that a lot of people are seeing now in AI
link |
is to build systems that have the kind of flexibility
link |
and the kind of flexibility that humans have in two senses.
link |
One is that humans can be good at many things.
link |
They're not just expert at one thing.
link |
And they're also flexible in the sense
link |
that they can switch between things very easily
link |
and they can pick up new things very quickly
link |
because they very ably see what a new task has in common
link |
with other things that they've done.
link |
And that's something that our AI systems
link |
just blatantly do not have.
link |
There are some people who like to argue
link |
that deep learning and deep RL
link |
are simply wrong for getting that kind of flexibility.
link |
I don't share that belief,
link |
but the simpler fact of the matter
link |
is we're not building things yet
link |
that do have that kind of flexibility.
link |
And I think the attention of a large part
link |
of the AI community is starting to pivot to that question.
link |
How do we get that?
link |
That's gonna lead to a focus on abstraction.
link |
It's gonna lead to a focus on
link |
what in psychology we call cognitive control,
link |
which is the ability to switch between tasks,
link |
the ability to quickly put together a program of behavior
link |
that you've never executed before,
link |
but you know makes sense for a particular set of demands.
link |
It's very closely related to what the prefrontal cortex does
link |
on the neuroscience side.
link |
So I think it's gonna be an interesting new chapter.
link |
So that's the reasoning side and cognition side,
link |
but let me ask the over romanticized question.
link |
Do you think we'll ever engineer an AGI system
link |
that we humans would be able to love
link |
and that would love us back?
link |
So have that level and depth of connection?
link |
I love that question.
link |
And it relates closely to things
link |
that I've been thinking about a lot lately,
link |
in the context of this human AI research.
link |
There's social psychology research
link |
in particular by Susan Fisk at Princeton
link |
the department where I used to work,
link |
where she dissects human attitudes toward other humans
link |
into a sort of two dimensional scheme.
link |
And one dimension is about ability.
link |
How able, how capable is this other person?
link |
But the other dimension is warmth.
link |
So you can imagine another person who's very skilled
link |
and capable, but is very cold.
link |
And you wouldn't really like highly,
link |
you might have some reservations about that other person.
link |
But there's also a kind of reservation
link |
that we might have about another person
link |
who elicits in us or displays a lot of human warmth,
link |
but is not good at getting things done.
link |
We reserve our greatest esteem really
link |
for people who are both highly capable
link |
and also quite warm.
link |
That's like the best of the best.
link |
This isn't a normative statement I'm making.
link |
This is just an empirical statement.
link |
This is what humans seem...
link |
These are the two dimensions that people seem to kind of like
link |
along which people size other people up.
link |
And in AI research,
link |
there's a lot of people who think that humans are
link |
very capable, and in AI research,
link |
we really focus on this capability thing.
link |
We want our agents to be able to do stuff.
link |
This thing can play go at a superhuman level.
link |
But that's only one dimension.
link |
What about the other dimension?
link |
What would it mean for an AI system to be warm?
link |
And I don't know, maybe there are easy solutions here.
link |
Like we can put a face on our AI systems.
link |
It's cute, it has big ears.
link |
I mean, that's probably part of it.
link |
But I think it also has to do with a pattern of behavior.
link |
A pattern of what would it mean for an AI system
link |
to display caring, compassionate behavior
link |
in a way that actually made us feel like it was for real?
link |
That we didn't feel like it was simulated.
link |
We didn't feel like we were being duped.
link |
To me, people talk about the Turing test
link |
or some descendant of it.
link |
I feel like that's the ultimate Turing test.
link |
Is there an AI system that can not only convince us
link |
that it knows how to reason
link |
and it knows how to interpret language,
link |
but that we're comfortable saying,
link |
yeah, that AI system's a good guy.
link |
On the warmth scale, whatever warmth is,
link |
we kind of intuitively understand it,
link |
but we also wanna be able to, yeah,
link |
we don't understand it explicitly enough yet
link |
to be able to engineer it.
link |
And that's an open scientific question.
link |
You kind of alluded it several times
link |
in the human AI interaction.
link |
That's a question that should be studied
link |
and probably one of the most important questions
link |
as we move to AGI.
link |
We humans are so good at it.
link |
It's not just that we're born warm.
link |
I suppose some people are warmer than others
link |
given whatever genes they manage to inherit.
link |
But there are also learned skills involved.
link |
There are ways of communicating to other people
link |
that you care, that they matter to you,
link |
that you're enjoying interacting with them, right?
link |
And we learn these skills from one another.
link |
And it's not out of the question
link |
that we could build engineered systems.
link |
I think it's hopeless, as you say,
link |
that we could somehow hand design
link |
these sorts of behaviors.
link |
But it's not out of the question
link |
that we could build systems that kind of,
link |
we instill in them something that sets them out
link |
in the right direction,
link |
so that they end up learning what it is
link |
to interact with humans
link |
in a way that's gratifying to humans.
link |
I mean, honestly, if that's not where we're headed,
link |
I think it's exciting as a scientific problem,
link |
just as you described.
link |
I honestly don't see a better way to end it
link |
than talking about warmth and love.
link |
And Matt, I don't think I've ever had such a wonderful
link |
conversation where my questions were so bad
link |
and your answers were so beautiful.
link |
So I deeply appreciate it.
link |
I really enjoyed it.
link |
Thanks for talking to me.
link |
Well, it's been very fun.
link |
As you can probably tell,
link |
there's something I like about kind of thinking
link |
outside the box and like,
link |
so it's good having an opportunity to do that.
link |
Thanks so much for doing it.
link |
Thanks for listening to this conversation
link |
with Matt Bopenik.
link |
And thank you to our sponsors,
link |
The Jordan Harbinger Show
link |
and Magic Spoon Low Carb Keto Cereal.
link |
Please consider supporting this podcast
link |
by going to jordanharbinger.com slash lex
link |
and also going to magicspoon.com slash lex
link |
and using code lex at checkout.
link |
Click the links, buy all the stuff.
link |
It's the best way to support this podcast
link |
and the journey I'm on in my research and the startup.
link |
If you enjoy this thing, subscribe on YouTube,
link |
review it with the five stars in Apple Podcasts,
link |
support it on Patreon, follow on Spotify
link |
or connect with me on Twitter at lexfreedman.
link |
Again, spelled miraculously without the E,
link |
just F R I D M A N.
link |
And now let me leave you with some words
link |
from neurologist V.S. Amarachandran.
link |
How can a three pound mass of jelly
link |
that you can hold in your palm imagine angels,
link |
contemplate the meaning of an infinity
link |
and even question its own place in the cosmos?
link |
Especially awe inspiring is the fact that any single brain,
link |
including yours, is made up of atoms
link |
that were forged in the hearts
link |
of countless far flung stars billions of years ago.
link |
These particles drifted for eons and light years
link |
until gravity and change brought them together here now.
link |
These atoms now form a conglomerate, your brain,
link |
that can not only ponder the very stars they gave at birth,
link |
but can also think about its own ability to think
link |
and wonder about its own ability to wander.
link |
With the arrival of humans, it has been said,
link |
the universe has suddenly become conscious of itself.
link |
This truly is the greatest mystery of all.
link |
Thank you for listening and hope to see you next time.