The following is a conversation with Jeff Hawkins.

He's the founder of the Redwood Center

for Theoretical and Neuroscience in 2002

and New Menta in 2005.

In his 2004 book titled On Intelligence

and in the research before and after,

he and his team have worked to reverse engineer

the New York Cortex and propose artificial intelligence

architectures approaches and ideas

that are inspired by the human brain.

These ideas include hierarchical temporal memory,

HTM from 2004, and New Work, The Thousand's Brain's Theory

of Intelligence from 2017, 18, and 19.

Jeff's ideas have been an inspiration

to many who have looked for progress beyond the current

machine learning approaches, but they have also received

criticism for lacking a body of empirical evidence

supporting the models.

This is always a challenge when seeking more than small

incremental steps forward in AI.

Jeff is a brilliant mind and many of the ideas

he has developed and aggregated from neuroscience

are worth understanding and thinking about.

There are limits to deep learning

as it is currently defined.

Forward progress in AI is shrouded in mystery.

My hope is that conversations like this

can help provide an inspiring spark for new ideas.

This is the Artificial Intelligence Podcast.

If you enjoy it, subscribe on YouTube,

iTunes, or simply connect with me on Twitter

at Lex Freedman spelled F R I D.

And now here's my conversation with Jeff Hawkins.

Are you more interested in understanding the human brain

or in creating artificial systems

that have many of the same qualities,

but don't necessarily require that you actually understand

the underpinning workings of our mind?

So there's a clear answer to that question.

My primary interest is understanding the human brain.

No question about it, but I also firmly believe

that we will not be able to create

fully intelligent machines

until we understand how the human brain works.

So I don't see those as separate problems.

I think there's limits to what can be done

with machine intelligence

if you don't understand the principles

by which the brain works.

And so I actually believe that studying the brain

is actually the fastest way to get to machine intelligence.

And within that, let me ask the impossible question.

How do you not define, but at least think

about what it means to be intelligent?

So I didn't try to answer that question first.

We said, let's just talk about how the brain works.

And let's figure out how certain parts of the brain,

mostly the neocortex, but some other parts too,

the parts of the brain most associated with intelligence.

And let's discover the principles by how they work.

Because intelligence isn't just like some mechanism

and it's not just some capabilities.

It's like, okay, we don't even know where to begin

on this stuff.

And so now that we've made a lot of progress on this,

after we've made a lot of progress

on how the neocortex works, and we can talk about that,

I now have a very good idea

what's gonna be required to make intelligent machines.

I can tell you today, some of the things

are gonna be necessary, I believe,

to create intelligent machines.

Well, so we'll get there.

We'll get to the neocortex and some of the theories

of how the whole thing works.

And you're saying, as we understand more and more

about the neocortex, about our own human mind,

we'll be able to start to more specifically define

what it means to be intelligent.

It's not useful to really talk about that until...

I don't know if it's not useful.

Look, there's a long history of AI, as you know.

And there's been different approaches taken to it.

And who knows?

Maybe they're all useful.

So the good old fashioned AI, the expert systems,

the current convolutional neural networks,

they all have their utility.

They all have a value in the world.

But I would think almost everyone agreed

that none of them are really intelligent

in a sort of a deep way that humans are.

And so it's just the question of how do you get

from where those systems were or are today

to where a lot of people think we're gonna go.

And there's a big, big gap there, a huge gap.

And I think the quickest way of bridging that gap

is to figure out how the brain does that.

And then we can sit back and look and say,

oh, what are these principles that the brain works on

are necessary and which ones are not?

Clearly, we don't have to build this in,

and tellage machines aren't gonna be built

out of organic living cells.

But there's a lot of stuff that goes on the brain

that's gonna be necessary.

So let me ask maybe, before we get into the fun details,

let me ask maybe a depressing or a difficult question.

Do you think it's possible that we will never

be able to understand how our brain works,

that maybe there's aspects to the human mind

like we ourselves cannot introspectively get to the core,

that there's a wall you eventually hit?

Yeah, I don't believe that's the case.

I have never believed that's the case.

There's not been a single thing we've ever,

humans have ever put their minds to that we've said,

oh, we reached the wall, we can't go any further.

People keep saying that.

People used to believe that about life, you know,

Alain Vitao, right?

There's like, what's the difference in living matter

and nonliving matter?

Something special you never understand.

We no longer think that.

So there's no historical evidence that suggests this is the case

and I just never even consider that's a possibility.

I would also say today, we understand so much

about the neocortex.

We've made tremendous progress in the last few years

that I no longer think of as an open question.

The answers are very clear to me and the pieces

that we don't know are clear to me,

but the framework is all there and it's like, oh, okay,

we're gonna be able to do this.

This is not a problem anymore.

It just takes time and effort, but there's no mystery,

a big mystery anymore.

So then let's get into it for people like myself

who are not very well versed in the human brain,

except my own.

Can you describe to me at the highest level,

what are the different parts of the human brain

and then zooming in on the neocortex,

the parts of the neocortex and so on, a quick overview.

Yeah, sure.

The human brain, we can divide it roughly into two parts.

There's the old parts, lots of pieces,

and then there's the new part.

The new part is the neocortex.

It's new because it didn't exist before mammals.

The only mammals have a neocortex and in humans

and primates is very large.

In the human brain, the neocortex occupies about 70 to 75%

of the volume of the brain.

It's huge.

And the old parts of the brain are,

there's lots of pieces there.

There's a spinal cord and there's the brainstem

and the cerebellum and the different parts

of the basal ganglion and so on.

In the old parts of the brain,

you have the autonomic regulation,

like breathing and heart rate.

You have basic behaviors.

So like walking and running are controlled

by the old parts of the brain.

All the emotional centers of the brain

are in the old part of the brain.

So when you feel anger or hungry,

lust or things like that,

those are all in the old parts of the brain.

And we associate with the neocortex

all the things we think about

as sort of high level perception.

And cognitive functions, anything from seeing and hearing

and touching things to language, to mathematics

and engineering and science and so on.

Those are all associated with the neocortex.

And they're certainly correlated.

Our abilities in those regards are correlated

with the relative size of our neocortex

compared to other mammals.

So that's like the rough division.

And you obviously can't understand

the neocortex completely isolated,

but you can understand a lot of it

with just a few interfaces

to the old parts of the brain.

And so it gives you a system to study.

The other remarkable thing about the neocortex

compared to the old parts of the brain

is the neocortex is extremely uniform.

It's not visually or anatomically,

or it's very, it's like a,

I always like to say it's like the size

of a dinner napkin, about two and a half millimeters thick.

And it looks remarkably the same everywhere.

Everywhere you look in that two and a half millimeters

is this detailed architecture.

And it looks remarkably the same everywhere.

And that's a cross species,

a mouse versus a cat and a dog and a human.

Where if you look at the old parts of the brain,

there's lots of little pieces do specific things.

So it's like the old parts of a brain evolved,

like this is the part that controls heart rate

and this is the part that controls this

and this is this kind of thing.

And that's this kind of thing.

And these evolve for eons of a long, long time

and they have those specific functions.

And all of a sudden mammals come along

and they got this thing called the neocortex

and it got large by just replicating the same thing

over and over and over again.

This is like, wow, this is incredible.

So all the evidence we have,

and this is an idea that was first articulated

in a very cogent and beautiful argument

by a guy named Vernon Malkassel in 1978, I think it was,

that the neocortex all works on the same principle.

So language, hearing, touch, vision, engineering,

all these things are basically underlying

or all built in the same computational substrate.

They're really all the same problem.

So the low level of the building blocks all look similar.

Yeah, and they're not even that low level.

We're not talking about like neurons.

We're talking about this very complex circuit

that exists throughout the neocortex is remarkably similar.

It is, it's like, yes, you see variations of it here

and they're more of the cell, that's not old and so on.

But what Malkassel argued was,

it says, you know, if you take a section on neocortex,

why is one a visual area and one is a auditory area?

Or why is, and his answer was,

it's because one is connected to eyes

and one is connected to ears.

Literally, you mean just as most closest

in terms of the number of connections to the sensor?

Literally, if you took the optic nerve

and attached it to a different part of the neocortex,

that part would become a visual region.

This actually, this experiment was actually done

by Murgankasur in developing, I think it was lemurs,

I can't remember what it was, it's some animal.

And there's a lot of evidence to this.

You know, if you take a blind person,

a person is born blind at birth,

they're born with a visual neocortex.

It doesn't, may not get any input from the eyes

because of some congenital defect or something.

And that region becomes, does something else.

It picks up another task.

So, and it's, so it's this very complex thing.

It's not like, oh, they're all built on neurons.

No, they're all built in this very complex circuit.

And somehow that circuit underlies everything.

And so this is, it's called the common cortical algorithm,

if you will, some scientists just find it hard to believe.

And they just say, I can't believe that's true.

But the evidence is overwhelming in this case.

And so a large part of what it means

to figure out how the brain creates intelligence

and what is intelligence in the brain

is to understand what that circuit does.

If you can figure out what that circuit does,

as amazing as it is, then you can,

then you understand what all these other cognitive functions are.

So if you were to sort of put neocortex

outside of your book on intelligence,

you look, if you wrote a giant tome,

a textbook on the neocortex,

and you look maybe a couple of centuries from now,

how much of what we know now would still be accurate

two centuries from now.

So how close are we in terms of understanding?

I have to speak from my own particular experience here.

So I run a small research lab here.

It's like any other research lab.

I'm sort of the principal investigator.

There's actually two of us,

and there's a bunch of other people.

And this is what we do.

We started the neocortex,

and we publish our results and so on.

So about three years ago,

we had a real breakthrough in this field.

Just tremendous breakthrough.

We started, we now publish, I think, three papers on it.

And so I have a pretty good understanding

of all the pieces and what we're missing.

I would say that almost all the empirical data

we've collected about the brain, which is enormous.

If you don't know the neuroscience literature,

it's just incredibly big.

And it's, for the most part, all correct.

It's facts and experimental results

and measurements and all kinds of stuff.

But none of that has been really assimilated

into a theoretical framework.

It's data without, in the language of Thomas Kuhn,

the historian, it would be sort of a preparadigm science.

Lots of data, but no way to fit it in together.

I think almost all of that's correct.

There's gonna be some mistakes in there.

And for the most part,

there aren't really good cogent theories

about how to put it together.

It's not like we have two or three competing good theories,

which ones are right and which ones are wrong.

It's like, yeah, people just scratching their heads

throwing things, you know, some people giving up

on trying to figure out what the whole thing does.

In fact, there's very, very few labs that we do

that focus really on theory

and all this unassimilated data and trying to explain it.

So it's not like we've got it wrong.

It's just that we haven't got it at all.

So it's really, I would say, pretty early days

in terms of understanding the fundamental theories,

forces of the way our mind works.

I don't think so.

I would have said that's true five years ago.

So as I said, we had some really big breakthroughs

on this recently and we started publishing papers on this.

So you can get to that.

But so I don't think it's, you know, I'm an optimist

and from where I sit today,

most people would disagree with this,

but from where I sit today, from what I know,

it's not super early days anymore.

We are, you know, the way these things go

is it's not a linear path, right?

You don't just start accumulating

and get better and better and better.

No, you got all the stuff you've collected.

None of it makes sense.

All these different things are just sort of around.

And then you're going to have some breaking points

all of a sudden, oh my God, now we got it right.

That's how it goes in science.

And I personally feel like we passed that little thing

about a couple of years ago.

All that big thing a couple of years ago.

So we can talk about that.

Time will tell if I'm right,

but I feel very confident about it.

That's when we'll just say it on tape like this.

At least very optimistic.

So let's, before those few years ago,

let's take a step back to HTM,

the hierarchical temporal memory theory,

which you first proposed on intelligence

and went through a few different generations.

Can you describe what it is,

how it would evolve through the three generations

since you first put it on paper?

Yeah, so one of the things that neuroscientists

just sort of missed for many, many years.

And especially people were thinking about theory

was the nature of time in the brain.

Brain's process, information through time,

the information coming into the brain is constantly changing.

The patterns from my speech right now,

if you're listening to it at normal speed,

would be changing on your ears

about every 10 milliseconds or so, you'd have a change.

This constant flow, when you look at the world,

your eyes are moving constantly,

three to five times a second,

and the input's completely, completely.

If I were to touch something like a coffee cup

as I move my fingers, the input changes.

So this idea that the brain works on time

changing patterns is almost completely,

or was almost completely missing

from a lot of the basic theories like fears of vision

and so on.

It's like, oh no, we're gonna put this image in front of you

and flash it and say, what is it?

A convolutional neural network's worked that way today, right?

Classified this picture.

But that's not what vision is like.

Vision is this sort of crazy time based pattern

that's going all over the place,

and so is touch and so is hearing.

So the first part of a hierarchical temporal memory

was the temporal part.

It's to say, you won't understand the brain,

nor will you understand intelligent machines

unless you're dealing with time based patterns.

The second thing was, the memory component of it was,

is to say that we aren't just processing input,

we learn a model of the world.

And the memory stands for that model.

The point of the brain, part of the neocortex,

it learns a model of the world.

We have to store things that are experiences

in a form that leads to a model of the world.

So we can move around the world,

we can pick things up and do things

and navigate and know how it's going on.

So that's what the memory referred to.

And many people just, they were thinking about like,

certain processes without memory at all.

They're just like processing things.

And then finally, the hierarchical component

was a reflection to that the neocortex,

although it's just a uniform sheet of cells,

different parts of it project to other parts,

which project to other parts.

And there is a sort of rough hierarchy in terms of that.

So the hierarchical temporal memory is just saying,

look, we should be thinking about the brain

as time based, model memory based and hierarchical processing.

And that was a placeholder for a bunch of components

that we would then plug into that.

We still believe all those things I just said,

but we now know so much more that I'm stopping to use

the word hierarchical temporal memory yet

because it's insufficient to capture the stuff we know.

So again, it's not incorrect,

but I now know more and I would rather describe it

more accurately.

Yeah, so you're basically, we could think of HTM

as emphasizing that there's three aspects of intelligence

that are important to think about

whatever the eventual theory converges to.

So in terms of time, how do you think of nature of time

across different time scales?

So you mentioned things changing,

sensory inputs changing every 10, 20 minutes.

What about every few minutes?

Every few months and years?

Well, if you think about a neuroscience problem,

the brain problem, neurons themselves can stay active

for certain periods of time.

They're parts of the brain where they stay active

for minutes, so you could hold a certain perception

or an activity for a certain period of time,

but not most of them don't last that long.

And so if you think about your thoughts

or the activity neurons,

if you're gonna wanna involve something

that happened a long time ago,

even just this morning, for example,

the neurons haven't been active throughout that time.

So you have to store that.

So by I ask you, what did you have for breakfast today?

That is memory.

That is, you've built it into your model of the world now.

You remember that and that memory is in the synapses,

it's basically in the formation of synapses.

And so you're sliding into what used to different time scales.

There's time scales of which we are

like understanding my language and moving about

and seeing things rapidly and over time.

That's the time scales of activities of neurons.

But if you wanna get in longer time scales,

then it's more memory and we have to invoke those memories

to say, oh, yes, well, now I can remember

what I had for breakfast because I stored that someplace.

I may forget it tomorrow, but I'd store it for now.

So does memory also need to have,

so the hierarchical aspect of reality

is not just about concepts,

it's also about time.

Do you think of it that way?

Yeah, time is infused in everything.

It's like, you really can't separate it out.

If I ask you, what is your, how's the brain

learn a model of this coffee cup here?

I have a coffee cup, then I met the coffee cup.

I said, well, time is not an inherent property

of the model I have of this cup,

whether it's a visual model or tactile model.

I can sense it through time,

but the model itself doesn't really have much time.

If I asked you, if I say, well,

what is the model of my cell phone?

My brain has learned a model of the cell phones.

If you have a smartphone like this,

and I said, well, this has time aspects to it.

I have expectations when I turn it on,

what's gonna happen, what water,

how long it's gonna take to do certain things,

if I bring up an app, what sequences,

and so I have instant, it's like melodies in the world,

you know, melody has a sense of time.

So many things in the world move and act,

and there's a sense of time related to them.

Some don't, but most things do actually.

So it's sort of infused throughout the models of the world.

You build a model of the world,

you're learning the structure of the objects in the world,

and you're also learning

how those things change through time.

Okay, so it really is just a fourth dimension

that's infused deeply,

and you have to make sure

that your models of intelligence incorporate it.

So, like you mentioned, the state of neuroscience

is deeply empirical.

A lot of data collection, it's, you know,

that's where it is, you mentioned Thomas Kuhn, right?

Yeah.

And then you're proposing a theory of intelligence,

and which is really the next step,

the really important step to take,

but why is HTM, or what we'll talk about soon,

the right theory?

So is it more in this, is it backed by intuition,

is it backed by evidence, is it backed by a mixture of both?

Is it kind of closer to where string theory is in physics,

where there's mathematical components

which show that, you know what,

it seems that this,

it fits together too well for it not to be true,

which is where string theory is.

Is that where you're kind of thinking?

It's a mixture of all those things,

although definitely where we are right now,

it's definitely much more on the empirical side

than, let's say, string theory.

The way this goes about, we're theorists, right?

So we look at all this data,

and we're trying to come up with some sort of model

that explains it, basically,

and there's, unlike string theory,

there's vast more amounts of empirical data here

than I think that most physicists deal with.

And so our challenge is to sort through that

and figure out what kind of constructs would explain this.

And when we have an idea,

you come up with a theory of some sort,

you have lots of ways of testing it.

First of all, I am, you know,

there are 100 years of assimilated,

und assimilated empirical data from neuroscience.

So we go back and repapers, and we say,

oh, did someone find this already?

We can predict X, Y, and Z,

and maybe no one's even talked about it

since 1972 or something,

but we go back and find that, and we say,

oh, either it can support the theory

or it can invalidate the theory.

And then we say, okay, we have to start over again.

Oh, no, it's support. Let's keep going with that one.

So the way I kind of view it,

when we do our work, we come up,

we look at all this empirical data,

and it's what I call it is a set of constraints.

We're not interested in something that's biologically inspired.

We're trying to figure out how the actual brain works.

So every piece of empirical data is a constraint on a theory.

If you have the correct theory,

it needs to explain every pin, right?

So we have this huge number of constraints on the problem,

which initially makes it very, very difficult.

If you don't have many constraints,

you can make up stuff all the day.

You can say, oh, here's an answer to how you can do this,

you can do that, you can do this.

But if you consider all biology as a set of constraints,

all neuroscience as a set of constraints,

and even if you're working in one little part of the Neocortex,

for example, there are hundreds and hundreds of constraints.

There are a lot of empirical constraints

that it's very, very difficult initially

to come up with a theoretical framework for that.

But when you do, and it solves all those constraints at once,

you have a high confidence

that you got something close to correct.

It's just mathematically almost impossible not to be.

So that's the curse and the advantage of what we have.

The curse is we have to meet all these constraints,

which is really hard.

But when you do meet them,

then you have a great confidence

that you've discovered something.

In addition, then we work with scientific labs.

So we'll say, oh, there's something we can't find,

we can predict something,

but we can't find it anywhere in the literature.

So we will then, we have people we collaborated with,

we'll say, sometimes they'll say, you know what,

I have some collected data, which I didn't publish,

but we can go back and look at it

and see if we can find that,

which is much easier than designing a new experiment.

You know, neuroscience experiments take a long time, years.

So although some people are doing that now too.

So, but between all of these things,

I think it's a reasonable,

it's actually a very, very good approach.

We are blessed with the fact that we can test our theories

out to yin and yang here,

because there's so much on a similar data,

and we can also falsify our theories very easily,

which we do often.

So it's kind of reminiscent to whenever that was with Copernicus,

you know, when you figure out that the sun is at the center,

the solar system as opposed to Earth,

the pieces just fall into place.

Yeah, I think that's the general nature of the Ha moments,

is in Copernicus, it could be,

you could say the same thing about Darwin,

you could say the same thing about, you know,

about the double helix,

that people have been working on a problem for so long,

and have all this data,

and they can't make sense of it, they can't make sense of it.

But when the answer comes to you,

and everything falls into place,

it's like, oh my gosh, that's it.

That's got to be right.

I asked both Jim Watson and Francis Crick about this.

I asked them, you know,

when you were working on trying to discover the structure

of the double helix,

and when you came up with the sort of,

the structure that ended up being correct,

but it was sort of a guess, you know,

it wasn't really verified yet.

I said, did you know that it was right?

And they both said, absolutely.

We absolutely knew it was right.

And it doesn't matter if other people didn't believe it or not,

we knew it was right, they'd get around to thinking it

and agree with it eventually anyway.

And that's the kind of thing you hear a lot with scientists

who really are studying a difficult problem,

and I feel that way too, about our work.

Have you talked to Crick or Watson about the problem

you're trying to solve, the, of finding the DNA of the brain?

Yeah.

In fact, Francis Crick was very interested in this,

in the latter part of his life.

And in fact, I got interested in brains

by reading an essay he wrote in 1979

called Thinking About the Brain.

And that was when I decided

I'm going to leave my profession of computers and engineering

and become a neuroscientist.

Just reading that one essay from Francis Crick.

I got to meet him later in life.

I got to, I spoke at the Salk Institute

and he was in the audience

and then I had a tea with him afterwards.

You know, he was interested in a different problem.

He was focused on consciousness.

The easy problem, right?

Well, I think it's the red herring

and so we weren't really overlapping a lot there.

Jim Watson, who's still alive,

is also interested in this problem

and when he was director of the Coltsman Harbor Laboratories,

he was really sort of behind

moving in the direction of neuroscience there.

And so he had a personal interest in this field

and I have met with him numerous times.

And in fact, the last time,

a little bit over a year ago,

I gave a talk at Coltsman Harbor Labs

about the progress we were making in our work.

And it was a lot of fun because he said,

well, you wouldn't be coming here

unless you had something important to say,

so I'm going to go attend your talk.

So he sat in the very front row.

Next to him was the director of the lab, Bruce Stillman.

So these guys were in the front row of this auditorium, right?

So nobody else in the auditorium wants to sit in the front row

because there's Jim Watson there as the director.

And I gave a talk and then I had dinner with Jim afterwards.

But there's a great picture of my colleague,

Subitai Amantik, where I'm up there

sort of expiring the basics of this new framework we have.

And Jim Watson's on the edge of his chair.

He's literally on the edge of his chair,

like, internally staring up at the screen.

And when he discovered the structure of DNA,

the first public talk he gave was at Coltsman Harbor Labs.

And there's a picture, there's a famous picture

of Jim Watson standing at the whiteboard

with an overhead thing pointing at something,

pointing at the double helix at this pointer.

And it actually looks a lot like the picture of me.

So there was a sort of funny, there's an area talking about the brain

and there's Jim Watson staring up at the tent.

And of course, there was, you know, whatever,

60 years earlier he was standing pointing at the double helix.

It's one of the great discoveries in all of, you know,

whatever, by all the science, all science and DNA.

So it's the funny that there's echoes of that in your presentation.

Do you think in terms of evolutionary timeline and history,

the development of the neocortex was a big leap?

Or is it just a small step?

So, like, if we ran the whole thing over again,

from the birth of life on Earth,

how likely would we develop the mechanism of the neocortex?

Okay, well, those are two separate questions.

One, was it a big leap?

And one was how likely it is, okay?

They're not necessarily related.

Maybe correlated.

And we don't really have enough data to make a judgment about that.

I would say definitely it was a big leap.

And I can tell you why.

I don't think it was just another incremental step.

I'll get that in a moment.

I don't really have any idea how likely it is.

If we look at evolution, we have one data point,

which is Earth, right?

Life formed on Earth billions of years ago,

whether it was introduced here or it created it here

or someone introduced it we don't really know,

but it was here early.

It took a long, long time to get to multicellular life.

And then from multicellular life,

it took a long, long time to get the neocortex.

And we've only had the neocortex for a few hundred thousand years.

So that's like nothing.

Okay, so is it likely?

Well, certainly it isn't something that happened right away on Earth.

And there were multiple steps to get there.

So I would say it's probably not going to something that would happen

instantaneously on other planets that might have life.

It might take several billion years on average.

Is it likely?

I don't know.

But you'd have to survive for several billion years to find out.

Probably.

Is it a big leap?

Yeah, I think it is a qualitative difference

in all other evolutionary steps.

I can try to describe that if you'd like.

Sure, in which way?

Yeah, I can tell you how.

Pretty much, let's start with a little preface.

Maybe the things that humans are able to do do not have obvious

survival advantages precedent.

We create music.

Is there a really survival advantage to that?

Maybe, maybe not.

What about mathematics?

Is there a real survival advantage to mathematics?

You can stretch it.

You can try to figure these things out, right?

But most of evolutionary history, everything had immediate survival

advantages to it.

I'll tell you a story, which I like.

It may not be true.

But the story goes as follows.

Organisms have been evolving since the beginning of life here on Earth.

Adding this sort of complexity onto that and this sort of complexity onto that.

And the brain itself is evolved this way.

There's an old part, an older part, an older, older part to the brain that kind of just

keeps calming on new things and we keep adding capabilities.

When we got to the neocortex, initially it had a very clear survival advantage

in that it produced better vision and better hearing and better touch and maybe

a new place and so on.

But what I think happens is that evolution took a mechanism, and this is in our

recent theory, but it took a mechanism that evolved a long time ago for

navigating in the world, for knowing where you are.

These are the so called grid cells and place cells of that old part of the brain.

And it took that mechanism for building maps of the world and knowing where you are

on those maps and how to navigate those maps and turns it into a sort of a slim

down idealized version of it.

And that idealized version could now apply to building maps of other things,

maps of coffee cups and maps of phones, maps of mathematics.

Concepts, yes, and not just almost, exactly.

And it just started replicating this stuff.

You just think more and more and more.

So we went from being sort of dedicated purpose neural hardware to solve certain

problems that are important to survival to a general purpose neural hardware

that could be applied to all problems and now it's escaped the orbit of survival.

It's, we are now able to apply it to things which we find enjoyment, you know,

but aren't really clearly survival characteristics.

And that it seems to only have happened in humans to the large extent.

And so that's what's going on where we sort of have, we've sort of escaped the

gravity of evolutionary pressure in some sense in the near cortex.

And it now does things which are not, that are really interesting,

discovering models of the universe, which may not really help us.

It doesn't matter.

How does it help us surviving knowing that there might be multiple verses or that

there might be, you know, the age of the universe or how do, you know,

various stellar things occur?

It doesn't really help us survive at all.

But we enjoy it and that's what happened.

Or at least not in the obvious way, perhaps.

It is required, if you look at the entire universe in an evolutionary way,

it's required for us to do interplanetary travel and therefore survive past our own fun.

But you know, let's not get too quick.

Yeah, but, you know, evolution works at one time frame.

It's survival, if you think of survival of the phenotype,

survival of the individual.

What you're talking about there is spans well beyond that.

So there's no genetic, I'm not transferring any genetic traits to my children.

That are going to help them survive better on Mars.

Totally different mechanism.

So let's get into the new, as you've mentioned, this idea,

I don't know if you have a nice name, thousand.

We call it the thousand brain theory of intelligence.

I like it.

So can you talk about this idea of spatial view of concepts and so on?

Yeah.

So can I just describe sort of the, there's an underlying core discovery,

which then everything comes from that.

That's a very simple, this is really what happened.

We were deep into problems about understanding how we build models of stuff in the world

and how we make predictions about things.

And I was holding a coffee cup just like this in my hand.

And I had my finger was touching the side, my index finger.

And then I moved it to the top and I was going to feel the rim at the top of the cup.

And I asked myself a very simple question.

I said, well, first of all, let's say I know that my brain predicts what it's going to feel

before it touches it.

You can just think about it and imagine it.

And so we know that the brain's making predictions all the time.

So the question is, what does it take to predict that?

And there's a very interesting answer.

First of all, it says the brain has to know it's touching a coffee cup.

It has to have a model of a coffee cup.

It needs to know where the finger currently is on the cup, relative to the cup.

Because when I make a movement, it needs to know where it's going to be on the cup

after the movement is completed, relative to the cup.

And then it can make a prediction about what it's going to sense.

So this told me that the neocortex, which is making this prediction,

needs to know that it's sensing it's touching a cup.

And it needs to know the location of my finger relative to that cup

in a reference frame of the cup.

It doesn't matter where the cup is relative to my body.

It doesn't matter its orientation.

None of that matters.

It's where my finger is relative to the cup, which tells me then that the neocortex

has a reference frame that's anchored to the cup.

Because otherwise, I wouldn't be able to say the location

and I wouldn't be able to predict my new location.

And then we quickly, very instantly, you can say,

well, every part of my skin could touch this cup

and therefore every part of my skin is making predictions

and every part of my skin must have a reference frame

that it's using to make predictions.

So the big idea is that throughout the neocortex,

there are, everything is being stored and referenced in reference frames.

You can think of them like XYZ reference frames,

but they're not like that.

We know a lot about the neural mechanisms for this.

But the brain thinks in reference frames.

And as an engineer, if you're an engineer, this is not surprising.

You'd say, if I were to build a CAD model of the coffee cup,

well, I would bring it up in some CAD software

and I would assign some reference frame and say,

this features at this location and so on.

But the fact that this, the idea that this is occurring

throughout the neocortex everywhere, it was a novel idea.

And then a zillion things fell into place after that, a zillion.

So now we think about the neocortex as processing information

quite differently than we used to do it.

We used to think about the neocortex as processing sensory data

and extracting features from that sensory data

and then extracting features from the features

very much like a deep learning network does today.

But that's not how the brain works at all.

The brain works by assigning everything,

every input, everything to reference frames,

and there are thousands, hundreds of thousands of them

active at once in your neocortex.

It's a surprising thing to think about,

but once you sort of internalize this,

you understand that it explains almost every,

almost all the mysteries we've had about this structure.

So one of the consequences of that is that

every small part of the neocortex, say a millimeter square,

and there's 150,000 of those.

So it's about 150,000 square millimeters.

If you take every little square millimeter of the cortex,

it's got some input coming into it,

and it's going to have reference frames

where it's assigning that input to.

And each square millimeter can learn complete models of objects.

So what do I mean by that?

If I'm touching the coffee cup,

well, if I just touch it in one place,

I can't learn what this coffee cup is

because I'm just feeling one part.

But if I move it around the cup and touch it in different areas,

I can build up a complete model of the cup

because I'm now filling in that three dimensional map,

which is the coffee cup.

I can say, oh, what am I feeling in all these different locations?

That's the basic idea.

It's more complicated than that.

But so through time, and we talked about time earlier,

through time, even a single column,

which is only looking at, or a single part of the cortex,

which is only looking at a small part of the world,

can build up a complete model of an object.

And so if you think about the part of the brain,

which is getting input from all my fingers,

so they're spread across the top of your head here.

This is the somatosensory cortex.

There's columns associated with all the different areas of my skin.

And what we believe is happening is that

all of them are building models of this cup,

every one of them, or things.

Not every column or every part of the cortex

builds models of everything,

but they're all building models of something.

And so when I touch this cup with my hand,

there are multiple models of the cup being invoked.

If I look at it with my eyes,

there are again many models of the cup being invoked,

because each part of the visual system,

the brain doesn't process an image.

That's a misleading idea.

It's just like your fingers touching the cup,

so different parts of my retina are looking at different parts of the cup.

And thousands and thousands of models of the cup

are being invoked at once.

And they're all voting with each other,

trying to figure out what's going on.

So that's why we call it the thousand brains theory of intelligence,

because there isn't one model of a cup.

There are thousands of models of this cup.

There are thousands of models of your cell phone,

and about cameras and microphones and so on.

It's a distributed modeling system,

which is very different than what people have thought about it.

So that's a really compelling and interesting idea.

I have two first questions.

So one, on the ensemble part of everything coming together,

you have these thousand brains.

How do you know which one has done the best job of forming the cup?

Great question. Let me try to explain.

There's a problem that's known in neuroscience

called the sensor fusion problem.

Yes.

And so the idea is something like,

oh, the image comes from the eye.

There's a picture on the retina.

And it gets projected to the neocortex.

Oh, by now it's all sped out all over the place,

and it's kind of squirrely and distorted,

and pieces are all over the, you know,

it doesn't look like a picture anymore.

When does it all come back together again?

Right?

Or you might say, well, yes, but I also,

I also have sounds or touches associated with the cup.

So I'm seeing the cup and touching the cup.

How do they get combined together again?

So this is called the sensor fusion problem.

As if all these disparate parts have to be brought together

into one model someplace.

That's the wrong idea.

The right idea is that you get all these guys voting.

There's auditory models of the cup,

there's visual models of the cup,

there's tactile models of the cup.

In the vision system, there might be ones

that are more focused on black and white,

ones versioned on color.

It doesn't really matter.

There's just thousands and thousands of models of this cup.

And they vote.

They don't actually come together in one spot.

Just literally think of it this way.

Imagine you have, each columns are like about the size

of a little piece of spaghetti.

Okay?

Like a two and a half millimeters tall

and about a millimeter in white.

They're not physical like, but you can think of them that way.

And each one's trying to guess what this thing is or touching.

Now they can, they can do a pretty good job

if they're allowed to move over time.

So I can reach my hand into a black box and move my finger

around an object and if I touch enough space,

it's like, okay, I know what it is.

But often we don't do that.

Often I can just reach and grab something with my hand

all at once and I get it.

Or if I had to look through the world through a straw,

so I'm only invoking one little column,

I can only see part of something because I have to move

the straw around.

But if I open my eyes to see the whole thing at once.

So what we think is going on is all these little pieces

of spaghetti, all these little columns in the cortex

are all trying to guess what it is that they're sensing.

They'll do a better guess if they have time

and can move over time.

So if I move my eyes and move my fingers.

But if they don't, they have a, they have a poor guess.

It's a, it's a probabilistic guess of what they might be touching.

Now imagine they can post their probability

at the top of little piece of spaghetti.

Each one of them says, I think,

and it's not really a probability distribution.

It's more like a set of possibilities in the brain.

It doesn't work as a probability distribution.

It works as more like what we call a union.

You could say, and one column says,

I think it could be a coffee cup, a soda can or a water bottle.

And another column says, I think it could be a coffee cup

or a, you know, telephone or camera or whatever.

Right.

And all these guys are saying what they think it might be.

And there's these long range connections

in certain layers in the cortex.

So there's some layers in some cell types in each column

send the projections across the brain.

And that's the voting occurs.

And so there's a simple associative memory mechanism.

We've described this in a recent paper and we've modeled this

that says they can all quickly settle on the only

or the one best answer for all of them.

If there is a single best answer, they all vote and say,

yep, it's got to be the coffee cup.

And at that point, they all know it's a coffee cup.

And at that point, everyone acts as if it's a coffee cup.

Yeah, we know it's a coffee.

Even though I've only seen one little piece of this world,

I know it's a coffee cup I'm touching or I'm seeing or whatever.

And so you can think of all these columns are looking

at different parts and different places,

different sensory input, different locations.

They're all different.

But this layer that's doing the voting, it solidifies.

It crystallizes and says, oh, we all know what we're doing.

And so you don't bring these models together in one model,

you just vote and there's a crystallization of the vote.

Great.

That's at least a compelling way to think about the way you

form a model of the world.

Now, you talk about a coffee cup.

Do you see this as far as I understand that you were proposing

this as well, that this extends to much more than coffee cups?

Yeah, it does.

Or at least the physical world.

It expands to the world of concepts.

Yeah, it does.

And well, the first, the primary phase of evidence for that

is that the regions of the neocortex that are associated

with language or high level thought or mathematics or things

like that, they look like the regions of the neocortex

that process vision and hearing and touch.

They don't look any different or they look only marginally

different.

And so one would say, well, if Vernon Mountcastle,

who proposed that all the parts of the neocortex

are the same thing, if he's right, then the parts

that are doing language or mathematics or physics

are working on the same principle.

They must be working on the principle of reference frames.

So that's a little odd thought.

But of course, we had no prior idea how these things happen.

So let's go with that.

And in our recent paper, we talked a little bit about that.

I've been working on it more since.

I have better ideas about it now.

I'm sitting here very confident that that's what's happening.

And I can give you some examples to help you think about that.

It's not that we understand it completely,

but I understand it better than I've described it in any paper

so far.

But we did put that idea out there.

It's a good place to start.

And the evidence would suggest it's how it's happening.

And then we can start tackling that problem one piece at a time.

What does it mean to do high level thought?

What does it mean to do language?

How would that fit into a reference framework?

I don't know if you could tell me if there's a connection,

but there's an app called Anki that helps you remember different concepts.

And they talk about like a memory palace that helps you remember

completely random concepts by trying to put them in a physical space

in your mind and putting them next to each other.

It's called the method of loci.

For some reason, that seems to work really well.

Now that's a very narrow kind of application of just remembering some facts.

But that's a very, very telling one.

Yes, exactly.

So this seems like you're describing a mechanism why this seems to work.

So basically the way what we think is going on is all things you know,

all concepts, all ideas, words, everything, you know,

are stored in reference frames.

And so if you want to remember something,

you have to basically navigate through a reference frame the same way

a rat navigates to a man.

Even the same way my finger rat navigates to this coffee cup.

You are moving through some space.

And so if you have a random list of things you would ask to remember

by assigning them to a reference frame,

you've already know very well to see your house, right?

And the idea of the method of loci is you can say,

okay, in my lobby, I'm going to put this thing.

And then the bedroom, I put this one.

I go down the hall, I put this thing.

And then you want to recall those facts.

So recall those things.

You just walk mentally.

You walk through your house.

You're mentally moving through a reference frame that you already had.

And that tells you there's two things that are really important about that.

It tells us the brain prefers to store things in reference frames.

And the method of recalling things or thinking, if you will,

is to move mentally through those reference frames.

You could move physically through some reference frames,

like I could physically move through the reference frame of this coffee cup.

I can also mentally move through the reference frame of the coffee cup,

imagining me touching it.

But I can also mentally move my house.

And so now we can ask ourselves, are all concepts stored this way?

There was some recent research using human subjects in fMRI.

And I'm going to apologize for not knowing the name of the scientists who did this.

But what they did is they put humans in this fMRI machine,

which was one of these imaging machines.

And they gave the humans tasks to think about birds.

So they had different types of birds, and birds that looked big and small

and long necks and long legs, things like that.

And what they could tell from the fMRI was a very clever experiment.

You get to tell when humans were thinking about the birds,

that the birds, the knowledge of birds was arranged in a reference frame

similar to the ones that are used when you navigate in a room.

These are called grid cells.

And there are grid cell like patterns of activity in the neocortex when they do this.

So that, it's a very clever experiment.

And what it basically says is that even when you're thinking about something abstract

and you're not really thinking about it as a reference frame,

it tells us the brain is actually using a reference frame.

And it's using the same neural mechanisms.

These grid cells are the basic same neural mechanisms that we propose

that grid cells, which exist in the old part of the brain, the entomonic cortex,

that that mechanism is now similar mechanism, is used throughout the neocortex.

It's the same nature to preserve this interesting way of creating reference frames.

And so now they have empirical evidence that when you think about concepts like birds

that you're using reference frames that are built on grid cells.

So that's similar to the method of loci.

But in this case, the birds are related so that makes,

they create their own reference frame, which is consistent with bird space.

And when you think about something, you go through that.

You can make the same example.

Let's take a math mathematics.

Let's say you want to prove a conjecture.

Okay.

What is a conjecture?

A conjecture is a statement you believe to be true,

but you haven't proven it.

And so it might be an equation.

I want to show that this is equal to that.

And you have some places you start with.

You say, well, I know this is true and I know this is true.

And I think that maybe to get to the final proof,

I need to go through some intermediate results.

What I believe is happening is literally these equations

or these points are assigned to a reference frame,

a mathematical reference frame.

And when you do mathematical operations,

a simple one might be multiply or divide,

maybe a little plus transform or something else.

That is like a movement in the reference frame of the math.

And so you're literally trying to discover a path

from one location to another location in a space of mathematics.

And if you can get to these intermediate results,

then you know your map is pretty good

and you know you're using the right operations.

Much of what we think about is solving hard problems

is designing the correct reference frame for that problem,

how to organize the information, and what behaviors

I want to use in that space to get me there.

Yeah, so if you dig in on an idea of this reference frame,

whether it's the math, you start a set of axioms

to try to get to proving the conjecture.

Can you try to describe, maybe take a step back,

how you think of the reference frame in that context?

Is it the reference frame that the axioms are happy in?

Is it the reference frame that might contain everything?

Is it a changing thing as you...

You have many, many reference frames.

In fact, the way the thousand brain theories of intelligence

says that every single thing in the world has its own reference frame.

So every word has its own reference frames.

And we can talk about this.

The mathematics work out this is no problem for neurons to do this.

But how many reference frames does the coffee cup have?

Well, let's say you ask how many reference frames

could the column in my finger that's touching the coffee cup have

because there are many, many models of the coffee cup.

So there is no model of the coffee cup.

There are many models of the coffee cup.

And you can say, well, how many different things can my finger learn?

Is this the question you want to ask?

Imagine I say every concept, every idea,

everything you've ever know about that you can say,

I know that thing has a reference frame associated with it.

And what we do when we build composite objects,

we assign reference frames to point another reference frame.

So my coffee cup has multiple components to it.

It's got a limb.

It's got a cylinder.

It's got a handle.

And those things have their own reference frames.

And they're assigned to a master reference frame,

which is called this cup.

And now I have this mental logo on it.

Well, that's something that exists elsewhere in the world.

It's its own thing.

So it has its own reference frame.

So we now have to say, how can I assign the mental logo reference frame

onto the cylinder or onto the coffee cup?

So we talked about this in the paper that came out in December

of this last year.

The idea of how you can assign reference frames to reference frames,

how neurons could do this.

So my question is, even though you mentioned reference frames a lot,

I almost feel it's really useful to dig into how you think

of what a reference frame is.

It was already helpful for me to understand that you think

of reference frames as something there is a lot of.

OK, so let's just say that we're going to have some neurons

in the brain, not many actually, 10,000, 20,000,

are going to create a whole bunch of reference frames.

What does it mean?

What is a reference frame?

First of all, these reference frames are different than the ones

you might be used to.

We know lots of reference frames.

For example, we know the Cartesian coordinates,

XYZ, that's a type of reference frame.

We know longitude and latitude.

That's a different type of reference frame.

If I look at a printed map, it might have columns,

A through M and rows, 1 through 20,

that's a different type of reference frame.

It's kind of a Cartesian reference frame.

The interesting thing about the reference frames in the brain,

and we know this because these have been established

through neuroscience studying the entorhinal cortex.

So I'm not speculating here.

This is known neuroscience in an old part of the brain.

The way these cells create reference frames,

they have no origin.

So what it's more like, you have a point,

a point in some space,

and you, given a particular movement,

you can then tell what the next point should be.

And you can then tell what the next point would be.

And so on.

You can use this to calculate how to get from one point to another.

So how do I get from my house to my home,

or how do I get my finger from the side of my cup

to the top of the cup?

How do I get from the axioms to the conjecture?

So it's a different type of reference frame.

And I can, if you want, I can describe in more detail.

I can paint a picture how you might want to think about that.

It's really helpful to think.

It's something you can move through.

Yeah.

But is it helpful to think of it as spatial in some sense,

or is there something?

No, it's definitely spatial.

It's spatial in a mathematical sense.

How many dimensions?

Can it be a crazy number of dimensions?

Well, that's an interesting question.

In the old part of the brain, the entorhinal cortex,

they studied rats.

And initially, it looks like, oh, this is just two dimensional.

It's like the rat is in some box in a maze or whatever,

and they know whether the rat is using these two dimensional

reference frames and know where it is in the maze.

We say, OK, well, what about bats?

That's a mammal, and they fly in three dimensional space.

How do they do that?

They seem to know where they are, right?

So this is a current area of active research,

and it seems like somehow the neurons in the entorhinal cortex

can learn three dimensional space.

We just, two members of our team, along with Ilefet from MIT,

just released a paper this literally last week,

it's on bioarchive, where they show that you can,

the way these things work, and unless you want to,

I won't get into the detail, but grid cells

can represent any n dimensional space.

It's not inherently limited.

You can think of it this way, if you had two dimensional,

the way it works is you had a bunch of two dimensional slices.

That's the way these things work.

There's a whole bunch of two dimensional models,

and you can slice up any n dimensional space

with two dimensional projections.

And you could have one dimensional models.

So there's nothing inherent about the mathematics

about the way the neurons do this,

which constrained the dimensionality of the space,

which I think was important.

So obviously, I have a three dimensional map of this cup.

Maybe it's even more than that, I don't know.

But it's a clearly three dimensional map of the cup.

I don't just have a projection of the cup.

But when I think about birds,

or when I think about mathematics,

perhaps it's more than three dimensions.

Who knows?

So in terms of each individual column

building up more and more information over time,

do you think that mechanism is well understood?

In your mind, you've proposed a lot of architectures there.

Is that a key piece, or is it, is the big piece,

the thousand brain theory of intelligence,

the ensemble of it all?

Well, I think they're both big.

I mean, clearly the concept, as a theorist,

the concept is most exciting, right?

A high level concept.

A high level concept.

This is a totally new way of thinking about

how the near characteristics work.

So that is appealing.

It has all these ramifications.

And with that, as a framework for how the brain works,

you can make all kinds of predictions

and solve all kinds of problems.

Now we're trying to work through many of these details right now.

Okay, how do the neurons actually do this?

Well, it turns out, if you think about grid cells

and place cells in the old parts of the brain,

there's a lot that's known about them,

but there's still some mysteries.

There's a lot of debate about exactly the details,

how these work, and what are the signs.

And we have that same level of detail,

that same level of concern.

What we spend here, most of our time doing,

is trying to make a very good list

of the things we don't understand yet.

That's the key part here.

What are the constraints?

It's not like, oh, this seems to work, we're done.

It's like, okay, it kind of works,

but these are other things we know it has to do,

and it's not doing those yet.

I would say we're well on the way here.

We're not done yet.

There's a lot of trickiness to this system,

but the basic principles about how different layers

in the neocortex are doing much of this, we understand.

But there's some fundamental parts

that we don't understand as well.

So what would you say is one of the harder open problems,

or one of the ones that have been bothering you,

keeping you up at night the most?

Well, right now, this is a detailed thing

that wouldn't apply to most people, okay?

Sure.

But you want me to answer that question?

Yeah, please.

We've talked about, as if, oh, to predict

what you're going to sense on this coffee cup,

I need to know where my finger's going to be on the coffee cup.

That is true, but it's insufficient.

Think about my finger touching the edge of the coffee cup.

My finger can touch it at different orientations.

I can touch it at my finger around here.

And that doesn't change.

I can make that prediction, and somehow,

so it's not just the location.

There's an orientation component of this as well.

This is known in the old part of the brain, too.

There's things called head direction cells,

which way the rat is facing.

It's the same kind of basic idea.

So if my finger were a rat, you know, in three dimensions,

I have a three dimensional orientation,

and I have a three dimensional location.

If I was a rat, I would have a,

I think it was a two dimensional location,

or one dimensional orientation, like this,

which way is it facing?

So how the two components work together,

how does it, I combine orientation,

the orientation of my sensor,

as well as the location,

is a tricky problem.

And I think I've made progress on it.

So at a bigger version of that,

the perspective is super interesting,

but super specific.

Yeah, I warned you.

No, no, no, it's really good,

but there's a more general version of that.

Do you think context matters?

The fact that we are in a building in North America,

that we, in the day and age where we have mugs,

I mean, there's all this extra information

that you bring to the table about everything else in the room

that's outside of just the coffee cup.

Of course it is.

How does it get connected, do you think?

Yeah, and that is another really interesting question.

I'm going to throw that under the rubric

or the name of attentional problems.

First of all, we have this model.

I have many, many models.

And also the question, does it matter?

Well, it matters for certain things.

Of course it does.

Maybe what we think about as a coffee cup

in another part of the world is viewed as something completely different.

Or maybe our logo, which is very benign in this part of the world,

it means something very different in another part of the world.

So those things do matter.

I think the way to think about it as the following,

one way to think about it,

is we have all these models of the world.

And we model everything.

And as I said earlier, I kind of snuck it in there.

Our models are actually, we build composite structures.

So every object is composed of other objects,

which are composed of other objects,

and they become members of other objects.

So this room is chairs and a table and a room and walls and so on.

Now we can just arrange these things in a certain way.

And you go, oh, that's in the Nementa conference room.

So, and what we do is when we go around the world,

when we experience the world,

by walking to a room, for example,

the first thing I do is like, oh, I'm in this room.

Do I recognize the room?

Then I can say, oh, look, there's a table here.

And by attending to the table,

I'm then assigning this table in a context of the room.

Then I say, oh, on the table, there's a coffee cup.

Oh, and on the table, there's a logo.

And in the logo, there's the word Nementa.

So if you look in the logo, there's the letter E.

And look, it has an unusual surf.

It doesn't actually, but I pretend it does.

So the point is your attention is kind of drilling deep in and out

of these nested structures.

And I can pop back up and I can pop back down.

I can pop back up and I can pop back down.

So when I attend to the coffee cup,

I haven't lost the context of everything else,

but it's sort of, there's this sort of nested structure.

The attention filters the reference frame formation

for that particular period of time.

Yes.

It basically, a moment to moment, you attend the subcomponents

and then you can attend the subcomponents to subcomponents.

You can move up and down.

You can move up and down.

We do that all the time.

You're not even, now that I'm aware of it,

I'm very conscious of it.

But most people don't even think about this.

You know, you just walk in a room and you don't say,

oh, I looked at the chair and I looked at the board

and looked at that word on the board

and I looked over here.

What's going on?

Right.

So what percentage of your day are you deeply aware of this?

In what part can you actually relax and just be Jeff?

Me personally, like my personal day.

Yeah.

Unfortunately, I'm afflicted with too much of the former.

Well, unfortunately or unfortunately.

Yeah.

You don't think it's useful?

Oh, it is useful.

Totally useful.

I think about this stuff almost all the time.

And one of my primary ways of thinking is

when I'm asleep at night,

I always wake up in the middle of the night

and I stay awake for at least an hour with my eyes shut

in sort of a half sleep state thinking about these things.

I come up with answers to problems very often

in that sort of half sleeping state.

I think about on my bike ride, I think about on walks.

I'm just constantly thinking about this.

I have to almost schedule time to not think about this stuff

because it's very, it's mentally taxing.

Are you, when you're thinking about this stuff,

are you thinking introspectively,

like almost taking a step outside of yourself

and trying to figure out what is your mind doing right now?

I do that all the time, but that's not all I do.

I'm constantly observing myself.

So as soon as I started thinking about grid cells,

for example, and getting into that,

I started saying, oh, well, grid cells can have my place of sense

in the world.

That's where you know where you are.

And it's interesting, we always have a sense of where we are

unless we're lost.

And so I started at night when I got up to go to the bathroom,

I would start trying to do it completely with my eyes closed

all the time and I would test my sense of grid cells.

I would walk five feet and say, okay, I think I'm here.

Am I really there?

What's my error?

And then I would calculate my error again and see how the errors

accumulate.

So even something as simple as getting up in the middle of the

night to go to the bathroom, I'm testing these theories out.

It's kind of fun.

I mean, the coffee cup is an example of that too.

So I think I find that these sort of everyday introspections

are actually quite helpful.

It doesn't mean you can ignore the science.

I mean, I spend hours every day reading ridiculously complex

papers.

That's not nearly as much fun,

but you have to sort of build up those constraints and the knowledge

about the field and who's doing what and what exactly they think

is happening here.

And then you can sit back and say, okay, let's try to have pieces

all together.

Let's come up with some, you know, I'm very in this group here

and people, they know they do this.

I do this all the time.

I come in with these introspective ideas and say, well,

there we ever thought about this.

Now watch, well, let's all do this together.

And it's helpful.

It's not, as long as you don't, if all you did was that,

then you're just making up stuff, right?

But if you're constraining it by the reality of the neuroscience,

then it's really helpful.

So let's talk a little bit about deep learning and the successes

in the applied space of neural networks, ideas of training model

on data and these simple computational units,

artificial neurons that with back propagation have statistical

ways of being able to generalize from the training set on to

data that similar to that training set.

So where do you think are the limitations of those approaches?

What do you think are strengths relative to your major efforts

of constructing a theory of human intelligence?

Yeah.

Well, I'm not an expert in this field.

I'm somewhat knowledgeable.

So, but I'm not.

A little bit in just your intuition.

Well, I have a little bit more than intuition,

but I just want to say like, you know, one of the things that you asked me,

do I spend all my time thinking about neuroscience?

I do.

That's to the exclusion of thinking about things like convolutional neural

networks.

But I try to stay current.

So look, I think it's great the progress they've made.

It's fantastic.

And as I mentioned earlier, it's very highly useful for many things.

The models that we have today are actually derived from a lot of

neuroscience principles.

They are distributed processing systems and distributed memory systems,

and that's how the brain works.

And they use things that we might call them neurons,

but they're really not neurons at all.

So we can just, they're not really neurons.

So they're distributed processing systems.

And nature of hierarchy that came also from neuroscience.

And so there's a lot of things, the learning rules, basically,

not backprop, but other, you know, sort of heavy entire learning.

I'll be curious to say they're not neurons at all.

Can you describe in which way?

I mean, some of it is obvious, but I'd be curious if you have specific

ways in which you think are the biggest differences.

Yeah, we had a paper in 2016 called Why Neurons of Thousands of Synapses.

And if you read that paper, you'll know what I'm talking about here.

A real neuron in the brain is a complex thing.

Let's just start with the synapses on it, which is a connection between neurons.

Real neurons can everywhere from five to 30,000 synapses on them.

The ones near the cell body, the ones that are close to the soma, the cell body,

those are like the ones that people model in artificial neurons.

There's a few hundred of those.

Maybe they can affect the cell.

They can make the cell become active.

95% of the synapses can't do that.

They're too far away.

So if you activate one of those synapses, it just doesn't affect the cell body

enough to make any difference.

Any one of them individually.

Any one of them individually, or even if you do a mass of them.

What real neurons do is the following.

If you activate, or you get 10 to 20 of them active at the same time,

meaning they're all receiving an input at the same time,

and those 10 to 20 synapses or 40 synapses are within a very short distance