The following is a conversation with Yosha Bach, his second time in the podcast.

Yosha is one of the most fascinating minds in the world, exploring the nature of intelligence,

cognition, computation, and consciousness.

To support this podcast, please check out our sponsors, Coinbase, Codecademy, Linode,

Netsuite, and ExpressVPN.

Their links are in the description.

This is the Lex Friedman podcast, and here is my conversation with Yosha Bach.

Thank you for once again coming on to this particular Russian program and sticking to

the theme of a Russian program.

Let's start with the darkest of topics.

So this is inspired by one of your tweets.

You wrote that, quote, when life feels unbearable, I remind myself that I'm not a person.

I am a piece of software running on the brain of a random ape for a few decades.

It's not the worst brain to run on.

Have you experienced low points in your life?

Have you experienced depression?

Of course, we all experience low points in our life, and we get appalled by the things,

by the ugliness of stuff around us.

We might get desperate about our lack of self regulation, and sometimes life is hard.

And I suspect you don't get to your life, nobody does, to get to their life without

low points and without moments where they are despairing.

And I thought that let's capture this state and how to deal with that state.

And I found that very often you realize that when you stop taking things personally, when

you realize that this notion of a person is a fiction, similar as it is in Westworld where

the robots realize that their memories and desires are just stuff that keeps them in

the loop, and they don't have to act on those memories and desires, that our memories and

expectations is what make us unhappy.

And the present rarely does.

The day in which we are, for the most part, it's okay, right?

When we are sitting here, right here, right now, we can choose how we feel.

And the thing that affects us is the expectation that something is going to be different from

what we wanted to be or the memory that something was different from what we wanted it to be.

And once we basically zoom out from all this, what's left is not a person, what's left

is this state of being conscious, which is a software state.

And software doesn't have an identity, it's a physical law.

And it's a law that acts in all of us, and it's embedded in a suitable substrate.

And we didn't pick that substrate, right?

We are mostly randomly instantiated on it.

And they're all these individuals, and everybody has to be one of them.

And eventually, you're stuck on one of them and have to deal with that.

So you're like a leaf floating down the river.

You just have to accept that there's a river and you just float wherever there takes you.

You don't have to do this.

The thing is that the illusion that you are an agent is a construct.

And what part of that is actually under your control.

And I think that our consciousness is largely a control model for our own attention.

So we notice where we are looking and we can influence what we are looking, how we are

disambiguating things, how we put things together in our mind.

And the whole system that runs us is this big cybernetic motivational system.

So we're basically like a little monkey sitting on top of an elephant.

And we can prod this elephant here and there to go this way or that way.

And we might have the illusion that we are the elephant or that we are telling it what

to do.

And sometimes we notice that it walks into a completely different direction.

And we didn't set this thing up.

It just is the situation that we find ourselves in.

How much prodding can we actually do of the elephant?

A lot.

But I think that our consciousness cannot create the motive force.

Is the elephant consciousness in this metaphor?

No, the monkey is the consciousness.

The monkey is the attentional system that is observing things.

There's a large perceptual system combined with a motivational system that is actually

providing the interface to everything and our own consciousness, I think, is the tool

that directs the attention of that system, which means it singles out features and performs

conditional operations for which it needs an index memory.

But this index memory is what we perceive as our stream of consciousness.

But the consciousness is not in charge.

That's an illusion.

So everything outside of that consciousness is the elephant.

So it's the physics of the universe, but it's also society that's outside of your...

I would say the elephant is the agent.

So there is an environment through which the agent is stomping, and you are influencing

a little part of that agent.

So is the agent a single human being?

What which object has agency?

That's an interesting question.

I think a way to think about an agent is that it's a controller with a set point generator.

The notion of a controller comes from cybernetics and control theory.

Control system consists out of a system that is regulating some value and the deviation

of that value from a set point.

And it has a sensor that measures the system's deviation from that set point and an effector

that can be parameterized by the controller.

So the controller tells the effector to do a certain thing.

And the goal is to reduce the distance between the set point and the current value of the

system.

And there's an environment which disturbs the regulated system, which brings it away

from that set point.

So the simplest case is the thermostat.

The thermostat is really simple because it doesn't have a model.

The thermostat is only trying to minimize the set point deviation in the next moment.

And if you want to minimize the set point deviation over a longer time span, you need to integrate

it, you need to model what is going to happen.

So for instance, when you think about that your set point is to be comfortable in life,

maybe you need to make yourself uncomfortable first.

So you need to make a model of what's going to happen when.

And this is the task of the controller is to use its sensors to measure the state of

the environment and the system that is being regulated and figure out what to do.

And if the task is complex enough, the set points are complicated enough.

And if the controller has enough capacity and enough sensor feedback, then the task

of the controller is to make a model of the entire universe that it's in, the conditions

under which it exists and of itself.

And this is a very complex agent and we are in that category.

And an agent is not necessarily a thing in the universe.

It's a class of models that we use to interpret aspects of the universe.

And when we notice the environment around us, a lot of things only make sense at the level

that should we're entangled with them, we interpret them as control systems that make

models of the world and try to minimize their own set points.

So the models are the agents?

The agent is a class of model.

And we notice that we are an agent ourselves.

We are the agent that is using our own control model to perform actions.

We notice we would use a change in the model and things in the world change.

And this is how we discover the idea that we have a body, that we are a situated environment

and that we do have a first person perspective.

Still don't understand what's the best way to think of which object has agency with respect

to human beings.

Is it the body?

Is it the brain?

Is it the contents of the brain that has agency?

Like what's the actuators that you're referring to?

What is the controller and where does it reside?

Or is it these impossible things?

Because I keep trying to ground it to space time, the three dimension of space and the

one dimension of time.

What's the agent in that for humans?

There is not just one.

It depends on the way in which you're looking at the thing in which you're framing it.

Imagine that you are, say, Angela Merkel and you are acting on behalf of Germany.

And you could say that Germany is the agent.

And in the mind of Angela Merkel, she is Germany to some extent because in the way in which

she acts, the destiny of Germany changes.

There are things that she can change that basically affect the behavior of that nation

state.

Okay.

So it's hierarchies to go to another one of your tweets with, I think you're playfully

mocking Jeff Hawkins with saying his brain's all the way down.

So it's like, it's agents all the way down.

It's agents made up of agents, made up of agents.

Like if Angela Merkel is Germany and Germany is made up a bunch of people and the people

are themselves agents in some kind of context.

And then people are made up of cells, each individual.

So is it agents all the way down?

I suspect that has to be like this in a world where things are self organizing.

Most of the complexity that we are looking at, everything in life is about self organization.

So I think up from the level of life, you have agents.

And below life, you rarely have agents because sometimes you have control systems that emerge

randomly in nature and try to achieve a set point, but they're not that interesting agents

that make models.

And because to make an interesting model of the world, you typically need a system that

is during complete.

Can I ask you a personal question?

What's the line between life and non life?

It's personal because you're a life form.

So what do you think in this emerging complexity, at which point does the things that are being

living and have agency?

Personally, I think that the simplest answer that is that life is cells.

Because...

Life is what?

Cells.

Cells.

Biological cells.

So it's a particular kind of principle that we have discovered to exist in nature.

It's modular stuff that consists out of basically this DNA tape with a redried head on top

of it that is able to perform arbitrary computations and state transitions within the cell.

And it's combined with a membrane that insulates the cell from its environment.

And there are chemical reactions inside of the cell that are in disequilibrium.

And the cell is running in such a way that this disequilibrium doesn't disappear.

And if the cell goes into an equilibrium state, it dies.

And it requires some things like a neck entropy extractor to maintain this disequilibrium.

So it's able to harvest neck entropy from its environment and keep itself running.

So there's information and there's a wall to maintain this disequilibrium.

But isn't this very earth centric like what you're referring to as...

I'm not making a normative notion.

You could say that there are probably other things in the universe that are cell like

and life like.

And you could also call them life.

But eventually it's just a willingness to find an agreement of how to use the terms.

I like cells because it's completely co extension of this the way that we use the word even

before we knew about cells.

So people were pointing at some stuff and saying, this is somehow animate.

And this is very different from the non animate stuff.

And what's the difference between the living and the dead stuff?

And it's mostly whether the cells are working or not.

And also this boundary of life where we say that, for instance, the virus is basically

an information packet that is subverting the cell and not life by itself.

That makes sense to me.

And it's somewhat arbitrary.

You could of course say that systems that permanently maintain a disequilibrium and

can self replicate are always life.

And maybe that's a useful definition too.

But this is eventually just how you want to use the word.

Is it so useful for conversation?

But is it somehow fundamental to the universe?

Do you think there's a actual line to eventually be drawn between life and non life?

Or is it all kind of continuum?

I don't think it's a continuum, but there's nothing magical that is happening.

Living systems are a certain type of machine.

What about non living systems?

Is it also a machine?

There are non living machines, but the question is at which point is the system able to perform

arbitrary state transitions to make representations?

And living things can do this.

And of course, we can also build non living things that can do this, but we don't know

anything in nature that is not a cell and is not created by Stalala Life that is able

to do that.

Not only do we not know.

I don't think we have the tools to see otherwise.

I always worry that we look at the world too narrowly.

There could be life of a very different kind right under our noses that we're just not

seeing because we're not either limitations of our cognitive capacity or we're just not

open minded enough, either with the tools of science or just the tools of our mind.

Yeah, that's possible.

I find this thought very fascinating and I suspect that many of us ask ourselves in childhood,

what are the things that we are missing?

What kind of systems and interconnections exist that are outside of our gaze?

But we are looking for it and physics doesn't have much room at the moment for opening up

something that would not violate the conservation of information as we know it.

Yeah, but I wonder about time scale and spatial scale, whether we just need to open up our

idea of how life presents itself.

It could be operating in a much slower time scale, a much faster time scale.

It's almost sad to think that there's all this life around us that we're not seeing

because we're just not thinking in terms of the right scale, both time and space.

What is your definition of life?

What do you understand as life?

Entities of sufficiently high complexity, they're full of surprises.

I don't have a free will, so that just came out of my mouth, I'm not sure that even makes

sense.

There are certain characteristics, so complexity seems to be a necessary property of life and

I almost want to say it has ability to do something unexpected.

It seems to me that life is the main source of complexity on earth.

And complexity is basically a bridgehead that order builds into chaos by modeling, by processing

information in such a way that you can perform reactions that would not be possible for dump

systems.

And this means that you can harvest like entropy, that dump systems cannot harvest, and this

is what complexity is mostly about.

In some sense, the purpose of life is to create complexity.

I mean, there seems to be some kind of universal drive towards increasing pockets of complexity.

I don't know what that is.

That seems to be like a fundamental, I don't know if it's a property of the universe or

it's just a consequence of the way the universe works, but there seems to be this small pockets

of emergent complexity that builds on top of each other and starts having like greater

and greater complexity by having like a hierarchy of complexity.

Little organisms building up a little society that then operates almost as an individual

organism itself and all of a sudden you have Germany and Merkel.

That's not obvious to me.

Everything that goes up has to come down at some point, right?

So if you see this big exponential curve somewhere, it's usually the beginning of an S curve or

something eventually reaches saturation and the S curve is the beginning of some kind

of bump that goes down again.

And there is just this thing that when you are in sight of an evolution of life, you

are on top of a puddle of negentropy that is being sucked dry by life.

And during that happening, you see an increase in complexity because life forms are competing

with each other to get more and more and finer and finer corner of that negentropy extraction.

But I feel like that's a gradual, beautiful process like that almost follows a process

akin to evolution.

And the way it comes down is not the same way it came up.

The way it comes down is usually harshly and quickly.

So usually there's some kind of catastrophic event.

The Roman Empire took a long time.

But would that be, would you classify that as a decrease in complexity though?

Yes.

I think that this size of the cities that could be fed has decreased dramatically.

And you could see that the quality of the art decreased and it did so gradually.

And maybe future generations, when they look at the history of the United States in the

21st century, will also talk about the gradual decline, not something that suddenly happens.

Do you have a sense of where we are?

Are we on the exponential rise?

Are we at the peak?

Or are we at the downslope of the United States Empire?

It's very hard to say from a single human perspective, but it seems to me that we are

probably at the peak.

I think that's probably the definition of optimism and cynicism.

So my nature of optimism is, I think we're on the rise, but I think this is just all

the matter of perspective that nobody knows.

But I do think that erring on the side of optimism, like you need a sufficient number,

you need a minimum number of optimists in order to make that up thing actually work.

And so I tend to be on the side of the optimists.

I think that we are basically species of grasshoppers that have turned into locusts.

And when you are in that locust mode, you see an amazing rise of population numbers

and of the complexity of the interactions between the individuals.

But ultimately, the question is, is it sustainable?

See, I think we're a bunch of lions and tigers that have become domesticated cats to use

a different metaphor.

And so I'm not exactly sure we're so destructive or just softer and nicer and lazier.

I think we have monkeys and not the cats.

And if you look at the monkeys, they are very busy.

The ones that have a lot of sex, those monkeys?

Not just the bonobos.

I think that all the monkeys are basically a discontent species that always needs to

meddle.

Well, the gorillas seem to have a little bit more of a structure, but it's a different

part of the tree.

Okay, you mentioned the elephant and the monkey riding the elephant, and consciousness is

the monkey.

And there's some prodding that the monkey gets to do.

And sometimes the elephant listens.

I heard you got into some content.

Maybe you can correct me, but I heard you got into some contentious free will discussions.

Is this with Sam Harris or something like that?

Not that I know of.

People in clubhouse told me you made a bunch of big debate points about free will.

Well, let me just then ask you, where in terms of the monkey and the elephant, do you think

we land in terms of the illusion of free will?

How much control does the monkey have?

We have to think about what the free will is in the first place.

We are not the machine.

We are not the thing that is making the decisions.

We are a model of that decision making process.

And there is a difference between making your own decisions and predicting your own decisions.

And that difference is the first person perspective.

And what basically makes decision making and the conditions of free will distinct from

just automatically doing the best thing is that we often don't know what the best thing

is.

We make decisions under uncertainty.

We make informed bets using a betting algorithm that we don't yet understand because we haven't

reverse engineered our own minds efficiently.

We don't know the expected rewards.

We don't know the mechanism by which we estimate the rewards and so on.

But there is an algorithm.

We only observe ourselves performing where we see that we weight facts and factors and

the future and then some kind of possibility, some motive gets raised to an intention.

And that's informed bet that the system is making.

And that making of the informed bet, the representation of that is what we call free

will.

And it seems to be paradoxical because we think that's the crucial thing is about it

that it's somehow indeterministic.

And yet if it was indeterministic, it would be random.

And it cannot be random because if it was random, if just dice were being thrown in

the universe randomly forces you to do things, it would be meaningless.

So the important part of the decisions is always the deterministic stuff.

But it appears to be indeterministic to you because it's unpredictable.

Because if it was predictable, you wouldn't experience it as a free will decision.

You would experience it as just doing the necessary right thing.

And you see this continuum between the free will and the execution of automatic behavior

when you're observing other people.

So for instance, when you are observing your own children, if you don't understand them,

you will use this agent model where you have an agent with a set point generator.

And the agent is doing the best it can to minimize the difference to the set point.

And it might be confused and sometimes impulsive or whatever, but it's acting on its own free

will.

And when you understand what happens in the mind of the child, you see that it's automatic.

And you can outmodel the child, you can build things around the child that will lead the

child to making exactly the decision that you are predicting.

And under these circumstances, like when you were a stage misvision or somebody who is

dealing with people that you sell a car to, and you completely understand the psychology

and the impulses and the space of thoughts that this individual can have at that moment.

Under these circumstances, it makes no sense to attribute free will.

Because it's no longer decision making under uncertainty.

You are already certain.

For them, there's uncertainty, but you already know what they're doing.

But what about for you?

So is this akin to like systems like cellular automata, where it's deterministic, but when

you squint your eyes a little bit, it starts to look like there's agents making decisions

at the higher sort of when you zoom out and look at the entities that are composed by the

individual cells.

You know, there's underlying simple rules that make the system evolve in deterministic

ways.

It looks like there's organisms making decisions.

Is that where the illusion of free will emerges?

That jump in scale?

It's a particular type of model, but this jump in scale is crucial.

The jump in scale happens whenever you have too many parts to count, and you cannot make

a model at that level, and you try to find some higher level regularity.

And the higher level regularity is a pattern that you project into the world to make sense

of it.

And agency is one of these patterns, right?

You have all these cells that interact with each other, and the cells in our body are

set up in such a way that they benefit if their behavior is coherent, which means that

they act as if they were serving a common goal, and which that means that they will

evolve regulation mechanisms that act as if they were serving a common goal.

And now you can make sense of all these cells by projecting the common goal into them.

Right.

So for you, then, free will is an illusion?

No, it's a model, and it's a construct.

It's basically a model that the system is making of its own behavior, and it's the

best model that it can come up with under the circumstances, and it can get replaced

by a different model, which is automatic behavior, when you fully understand the mechanism under

which you are acting.

Yeah, but another word for model is what, story.

So the story you're telling, I mean, do you actually have control?

Is there such a thing as a you, and is there such a thing as you having control?

So like, are you manifesting your evolution as an entity?

In some sense, the you is the model of the system that is in control.

It's a story that the system tells itself about somebody who is in control.

And the contents of that model are being used to inform the behavior of the system.

So the system is completely mechanical, and the system creates that story like a loom,

and then it uses the contents of that story to inform its actions and writes the results

of that action into the story.

So how's that non illusion?

The story is written then, or a rather, we're not the writers of the story.

Yes, but we always knew that.

No, we don't know that.

When did we know that?

I think that's mostly a confusion about concepts.

The conceptual illusion in our culture comes from the idea that we live in physical reality,

and that we experience physical reality, and that we have ideas about it.

And then you have this dualist interpretation, where you have two substances, res extensor,

the world that you can touch, and that is made of extended things, and res cogitans,

which is the world of ideas, and in fact, both of them are mental representations.

One is the representations of the world as a game engine that your mind generates to

make sense of the perceptual data, and the other one, yes, that's what we perceive as

the physical world.

But we already know that the physical world is nothing like that, right?

But the mechanics is very different from what you and me perceive as the world.

The world that you and me perceive is a game engine, and there are no colors and sounds

in the physical world that only exist in the game engine generated by your brain.

And then you have ideas that cannot be mapped onto extended regions, right?

So the objects that have a spatial extension in the game engine, res extensor, and the

objects that don't have a physical extension in the game engine are ideas.

And they both interact in our mind to produce models of the world.

But when you play video games, I understand that what's actually happening is zeros and

ones inside of a computer, inside of a CPU and a GPU, but you're still seeing the rendering

of that, and you're still making decisions whether to shoot, to turn left or to turn

right if you're playing a shooter, or every time you start thinking about Skyrim and Elder

Scrolls and walking around in a beautiful nature and swinging a sword.

But it feels like you're making decisions inside that video game, so even though you

don't have direct access in terms of perception to the bits, to the zeros and ones, it still

feels like you're making decisions, and your decisions actually feels like they're being

applied all the way down to the zeros and ones.

So it feels like you have control, even though you don't direct access to reality.

So there is basically a special character in the video game that is being created by

the video game engine, and this character is serving the aesthetics of the video game,

and that is you.

Yes, but I feel like I have control inside the video game, like all those 12 year olds

that kick my ass on the internet.

So when you play the video game, it doesn't really matter that there's zeros and ones,

you don't care about the bits of the bus, you don't care about the nature of the CPU

that it runs on.

What you care about are the properties of the game that you're playing, and you hope

that the CPU is good enough.

And a similar thing happens when we interact with physics.

The world that you and me are in is not the physical world.

The world that you and me are in is a dream world.

How close is it to the real world, though?

We know that it's not very close, but we know that the dynamics of the dream world

match the dynamics of the physical world to a certain degree of resolution, but the causal

structure of the dream world is different.

So you see, for instance, waves crashing on your feet, right, but there are no waves

in the ocean.

There's only water molecules that have tangents between the molecules that are the result

of electrons in the molecules interacting with each other.

But aren't they very consistent where you're seeing a very crude approximation?

Aren't our dream world very consistent, like to the point of being mapped directly one

to one to the actual physical world as opposed to us being completely tricked?

This is like where you have like Donald Hock.

It's not a trick.

That's my point.

It's not an illusion.

It's a form of data compression.

Yeah.

It's an attempt to deal with the dynamics of too many parts to count at the level at

which we are entangled with the best model that you can find.

Yeah.

So we can act in that dream world.

And our actions have impact in the real world, in the physical world.

Yes.

To which we don't have access.

Yes.

But it's basically like accepting the fact that the software that we live in, the dream

that we live in is generated by something outside of this world that you and me are in.

So is the software deterministic and do we not have any control?

So free will is having a conscious being.

Free will is the monkey being able to steer the elephant.

No.

It's slightly different.

Basically, in the same way as you are modeling the water molecules in the ocean that engulf

your feet when you are walking on the beach as waves and there are no waves, but only

the atoms on more complicated stuff underneath the atoms and so on.

You know that, right?

You would accept, yes, there is a certain abstraction that happens here.

It's a simplification of what happens.

And a simplification that is designed in such a way that your brain can deal with it temporarily

and spatially in terms of resources and tuned for the predictive value.

So you can predict with some accuracy whether your feet are going to get wet or not.

But it's a really good, it's a really good interface and approximation.

Yes.

It's like E equals m t squared is a good, equations are good approximations for what

they're much better approximation.

So to me waves is a really nice approximation of what's all the complexity that's happening

underneath.

Yes.

Basically, it's a machine learning model that is constantly tuned to minimize surprises.

So it basically tries to predict as well as it can what you're going to perceive next.

Are we talking about which is the machine learning, our perception system or the dream

world?

The dream world is the result of the machine learning process of the perception system.

That's doing the compression.

Yes.

So the model of you as an agent is not a different type of model or it's a different

type but not different as in its model like nature from the model of the ocean.

Some things are oceans, some things are agents.

And one of these agents is using your own control model, the output of your model, the

things that you perceive yourself as doing.

And that is you.

What about the fact that when you're standing with the water on your feet and you're looking

out into the vast open water of the ocean and then there's a beautiful sunset and the

fact that it's beautiful and then maybe you have friends or loved one with you and you

feel love.

What is that?

As the dream world?

What is that?

Yes.

It's all happening inside of the dream.

Okay.

You see, the word dream makes it seem like it's not real.

Of course, it's not real.

The physical universe is real but the physical universe is incomprehensible and it doesn't

have any feeling of realness.

The feeling of realness that you experience gets attached to certain representations where

your brain assesses this is the best model of reality that I have.

So the only thing that's real to you is the thing that's happening at the very base of

reality like the…

For something to be real, it needs to be implemented.

So the model that you have of reality is real in as far as it is a model.

It's an appropriate description of the world to say that there are models that are being

experienced but the world that you experience is not necessarily implemented.

There is a difference between a reality, a simulation and a simulacrum.

The reality that we're talking about is something that fully emerges over a causally

closed lowest layer.

The idea of physicalism is that we are in that layer, that basically our world emerges

over that.

Every alternative to physicalism is a simulation theory which basically says that we are in

some kind of simulation universe and the real world needs to be an apparent universe of

that where the actual causal structure is.

And when you look at the ocean in your own mind, you are looking at a simulation that

explains what you're going to see next.

So we are living in a simulation.

Yes, but a simulation generated by our own brains.

And this simulation is different from the physical reality because the causal structure

that is being produced, what you are seeing is different from the causal structure of

physics.

But consistent.

Hopefully.

If not, then you are going to end up in some kind of institution where people will take

care of you because your behavior will be inconsistent.

Your behavior needs to work in such a way that it's interacting with an accurately

predictive model of reality and if your brain is unable to make your model of reality predictive,

you will need help.

So what do you think about Donald Hoffman's argument that it doesn't have to be consistent

the dream world to what he calls like the interface to the actual physical reality

where there could be, I think he makes an evolutionary argument, which is like it could

be an evolutionary advantage to have the dream world drift away from physical reality.

I think that only works if you have tenure.

As long as you're still interacting with the ground tools, your model needs to be somewhat

predictive.

Well, in some sense, humans have achieved a kind of tenure in the animal kingdom.

Yeah.

At some point, we became too big to fail, so we became postmodernist.

It all makes sense now, the version of reality that we like.

Oh man.

Okay.

Yeah, but basically, you can do magic.

You can change your assessment of reality, but eventually, reality is going to come bite

you in the ass if it's not predictive.

Do you have a sense of what is that base layer of physical reality?

You have these attempts at the theories of everything, the very, very small of like string

theory or what Stephen Wolfram talks about with the hypergrass, these are these tiny,

tiny, tiny, tiny objects, and then there is more like quantum mechanics that's talking

about objects that are much larger, but still very, very, very tiny.

Do you have a sense of where the tiniest thing is that is like at the lowest level, the turtle

at the very bottom?

Do you have a sense?

I don't think that you can talk about where it is because space is emergent over the activity

of these things.

So coordinates only exist in relation to the things, other things.

And so you could, in some sense, abstract it into locations that can hold information

and trajectories that the information can take between the different locations.

And this is how we construct our notion of space.

And physicists usually have a notion of space that is continuous.

And this is a point where I tend to agree with people like Stephen Wolfram, who are very

skeptical of the geometric notions.

I think that geometry is the dynamics of too many parts to count.

And when there are no infinities, if there were two infinities, you would be able to

be running into contradictions, which is, in some sense, what Gödel and Turing discovered

in response to Hilbert's call.

So there are no infinities.

There are no infinities.

There is unboundedness.

But if you have a language that talks about infinity, at some point, the language is going

to contradict itself, which means it's no longer valid.

In order to deal with infinities and mathematics, you have to postulate the existence initially.

You cannot construct the infinities.

And that's an issue, right?

You cannot build up an infinity from zero.

But in practice, you never do this, right?

When you perform calculations, you only look at the dynamics of too many parts to count.

And usually, these numbers are not that large.

They're not Googles or something.

The infinities that we are dealing with in our universe are, mathematically speaking,

relatively small integers.

And still, what we're looking at is dynamics where a trillion things behave similar to

a hundred trillion things or something that is very, very large, because they're converging.

And these converging dynamics, these operators, this is what we deal with when we are doing

the geometry.

Right?

The geometry is stuff where we can pretend that it's continuous, because if we subdivide

the space sufficiently fine grained, these things approach a certain dynamic.

And this approached dynamic, that is what we mean by it.

But I don't think that infinity would work, so to speak, that you would know the last

digit of pi and that you have a physical process that rests on knowing the last digit of pi.

Yeah, that could be just a peculiar quirk of human cognition that we like discrete.

Discrete makes sense to us.

Infinity doesn't.

So in terms of our intuitions.

No, the issue is that everything that we think about needs to be expressed in some

kind of mental language, not necessarily a natural language, but some kind of mathematical

language that your neurons can speak, that refers to something in the world.

And what we have discovered is that we cannot construct a notion of infinity without running

into contradictions, which means that such a language is no longer valid.

And I suspect this is what made Pythagoras so unhappy when somebody came up with the

notion of irrational numbers before it was time, where there's this myth that he had

this person killed when he blabbed out the secret that not everything can be expressed

as a ratio between two numbers, but that there are numbers between the ratios.

The world was not ready for this.

And I think he was right that has confused mathematicians very seriously because these

numbers are not values, they are functions.

And so you can calculate these functions to a certain degree of approximation, but you

cannot pretend that pi has actually a value.

Pi is a function that would approach this value to some degree.

But nothing in the world rests on knowing pi.

How important is this distinction between discrete and continuous for you to get to

the point?

Because there's a, I mean, in discussion of your favorite flavor of the theory of everything,

there's a few on the table.

So there's string theory, there's a particular, there's a loop quantum gravity, which focus

on one particular unification.

There's just a bunch of favorite flavors of different people trying to propose a theory

of everything.

Eric Weinstein and a bunch of people throughout history.

And then of course, Stephen Wolfram, who I think is one of the only people doing a discrete

theory.

No, no.

There's a bunch of physicists who do this right now.

And like Tofoli and Tomasello and digital physics is something that is, I think, growing

in popularity.

But the main reason why this is interesting is because it is important sometimes to settle

disagreements.

I don't think that you need infinities at all and you never needed them.

You can always deal with very large numbers and you can deal with limits, right?

We are fine with doing that.

You don't need any kind of affinity.

You can build your computer algebra systems just as well without believing in infinity

in the first place.

So you're okay with limits?

Yeah.

So basically a limit means that something is behaving pretty much the same if you make

the number larger, right?

Because it's converging to a certain value and at some point the difference becomes

negligible and you can no longer measure it.

And in this sense, you have things that if you have an n gone, which has enough corners,

then it's going to behave like a circle at some point, right?

And it's only going to be in some kind of esoteric thing that cannot exist in a physical

universe that you would be talking about this perfect circle.

And now it turns out that it also wouldn't work in mathematics because you cannot construct

mathematics that has infinite resolution without running into contradictions.

So that is itself not that important because we never did that, right?

It's just a thing that some people thought we could.

And this leads to confusion.

So for instance, Roger Penrose uses this as an argument to say that there are certain

things that mathematicians can do dealing with infinities.

And by extension, our mind can do that computers cannot do.

Yeah.

He talks about that there's the human mind can do certain mathematical things that the

computer has defined by the universal towing machine cannot.

Yes.

What?

So that it has to do with infinity.

Yes.

It's one of the things.

So he is basically pointing at the fact that there are things that are possible in the

mathematical mind and in pure mathematics that are not possible in machines that can

be constructed in the physical universe.

And because he's an honest guy, he thinks this means that present physics cannot explain

operations that happen in our mind.

Do you think he's right?

So let's leave his discussion of consciousness aside for the moment.

Do you think he's right about just what he's basically referring to as intelligence?

So is the human mind fundamentally more capable as a thinking machine than a universal towing

machine?

No.

But so he's suggesting that, right?

So our mind is actually less than a towing machine.

There can be no towing machine because it's defined as having an infinite tape.

And we always only have a finite tape.

But he's saying it's better.

Our minds can only perform finitely many operations.

But he's saying it can do the kind of computation that the towing machine cannot.

And that's because he thinks that our minds can do operations that have infinite resolution

in some sense.

And I don't think that's the case.

Our minds are just able to discover these limit operators over too many parts to count.

What about his idea that consciousness is more than a computation?

So it's more than something that a towing machine can do.

So again, saying that there's something special about our mind that cannot be replicated in

the machine.

The issue is that I don't even know how to construct a language to express this statement

correctly.

Well, the basic statement is there's a human experience that includes intelligence, that

includes self awareness, that includes the hard problem of consciousness.

And the question is, can that be fully simulated in the computer, in the mathematical model

of the computer, as we understand it today?

As your parenthesis, no.

So the universal towing machine cannot simulate the universe.

So the interesting question is, and you have to ask him this, is why not?

What is the specific thing that cannot be modeled?

And when I looked at his writings, and I haven't read all of it, but when I read, for instance,

the section that he writes in the introduction and wrote to infinity, the thing that he specifically

refers to is the way in which human minds deal with infinities.

And that itself can, I think, easily be deconstructed.

A lot of people feel that our experience cannot be explained in a mechanical way.

And therefore, it needs to be different.

And I concur, our experience is not mechanical.

Our experience is simulated.

It exists only in a simulation.

The only simulation can be conscious.

Physical systems cannot be conscious because they're only mechanical.

Cells cannot be conscious.

Neurons cannot be conscious.

Brains cannot be conscious.

People cannot be conscious as far as you, if you understand them as physical systems.

What can be conscious is the story of a system in the world where you write all these things

into the story.

You have experiences for the same reason that a character and novel have experiences, because

it's written into the story.

And now the system is acting on that story.

And it's not a story that is written in a natural language.

It's written in a perceptual language, in this multimedia language of the game engine.

And in there, you write in what kind of experience you have and what this means for the behavior

of the system, for your behavior tendencies, for your focus, for your attention, for your

experience of valence, and so on.

And this is being used to inform the behavior of the system in the next step.

And then the story updates with the reactions of the system and the changes in the world,

and so on.

And you live inside of that model.

You don't live inside of the physical reality.

And I mean, just to linger on it, like, you see, okay, it's in the perceptual language,

the multimodal perceptual language, that's the experience.

That's what consciousness is within that model, within that story.

But do you have agency?

When you play a video game, you can turn left and you can turn right in that story.

So in that dream world, how much control do you have?

Is there such a thing as you in that story?

Like is it right to say the main character, you know, everybody's NPCs, and then there's

the main character, and you're controlling the main character?

Or is that an illusion?

Is there a main character that you're controlling?

I'm getting to the point of, like, the free will point.

Imagine that you are building a robot that plays soccer.

And you've been to MIT computer science, you basically know how to do that.

And so you would say the robot is an agent that solves a control problem, how to get

the ball into the goal.

And it needs to perceive the world and the world is disturbing him in trying to do this.

So he has to control many variables to make that happen and to project itself and the

ball into the future and understand its position on the field relative to the ball and so on

in the position of its limbs or in the space around it and so on.

So it needs to have an adequate model that abstracting reality in a useful way.

And you could say that this robot does have agency over what it's doing in some sense.

And the model is going to be a control model.

And inside of that control model, you can possibly get to a point where this thing is

sufficiently abstract to discover its own agency.

Our current robots don't do that.

They don't have a unified model of the universe.

But there's not a reason why we shouldn't be getting there at some point in the not

too distant future.

And once that happens, you will notice that the robot tells a story about a robot playing

soccer.

So the robot will experience itself playing soccer in a simulation of the world that it

uses to construct a model of the locations of its legs and limbs in space on the field

with relationship to the ball.

And it's not going to be at the level of the molecules.

It will be an abstraction that is exactly at the level that is most suitable for past

planning of the movements of the robot.

It's going to be a high level abstraction, but a very useful one that is as predictive

as we can make it.

And in that side of that story, there is a model of the agency of that system.

So the model can accurately predict that the contents of the model are going to be driving

the behavior of the robot in the immediate future.

But there's the hard problem of consciousness, which I would also, there's a subjective experience

of free will as well, that I'm not sure where the robot gets that, where that little leap

is.

Because for me right now, everything I imagine with that robot as it gets more and more and

more sophisticated, the agency comes from the programmer of the robot still, of what

was programmed in.

You would probably do an end to end learning system.

You maybe need to give it a few priors, so you nudge the architecture in the right direction

that it converges more quickly.

But ultimately, discovering the suitable hyperparameters of the architecture is also only a search

process, right?

And as the search process was evolution that has informed our brain architecture so we

can converge in a single lifetime on useful interaction with the world and the formation

of a circle.

The problem is if we define hyperparameters broadly, so it's not just the parameters that

control this end to end learning system, but the entirety of the design of the robot.

You have to remove the human completely from the picture.

And then in order to build the robot, you have to create an entire universe because you can't

just shortcut evolution.

You have to go from the very beginning.

In order for it to have, because I feel like there's always a human pulling the strings

and that makes it seem like the robot is cheating.

It's getting a shortcut to consciousness.

And you are looking at the current Boston Dynamics robots.

It doesn't look as if there is somebody pulling the strings.

It doesn't look like cheating anymore.

Okay, so let's go there because I got to talk to you about this.

So obviously with the case of Boston Dynamics, as you may or may not know, it's always either

hard coded or remote controlled.

There's no intelligence.

So I don't know how the current generation of Boston Dynamics robots works, but what

I've been told about the previous ones was that it's basically all cybernetic control,

which means you still have feedback mechanisms and so on.

But it's not deep learning for the most part as it's currently done.

For the most part, just identifying a control hierarchy that is congruent to the limbs that

exist and the parameters that need to be optimized for the movement of these limbs.

And then there is a convergence progress.

So it's basically just regression that you would need to control this.

But again, I don't know whether that's true.

That's just what I've been told about how they work.

We have to separate several levels of discussions here.

So the only thing they do is pretty sophisticated control with no machine learning in order

to maintain balance or to write itself.

It's a control problem in terms of using the actuators to when it's pushed or when it steps

on a thing that's uneven, how to always maintain balance.

And there's a tricky heuristics around that, but that's the only goal.

Everything you see Boston Dynamics doing in terms of that to us humans is compelling,

which is any kind of higher order movement like turning, wiggling its butt, jumping back

on its two feet, dancing, dancing is even worse because dancing is hard coded in.

It's it's choreographed by humans.

There's choreography software, like there is no of all that high level movement.

There's no anything that you can call certainly can't call AI, but there's no even like basic

heuristics.

It's all hard coded in.

And yet we humans immediately project agency onto them, which is this is fast.

So the gap here is it doesn't necessarily have agency.

What it has is cybernetic control.

And the cybernetic control means you have a hierarchy of feedback loops that keep the

behavior in certain boundaries so the robot doesn't fall over and it's able to perform

the movements and the choreography cannot really happen with motion capture because the robot

would fall over because the physics of the robot, the weight distribution and so on is

different from the weight distribution in the human body.

So if you were using the directly motion captured movements of a human body to project

it into this robot, it wouldn't work.

You can do this with a computer animation that will look a little bit off, but who cares?

But if you want to correct for the physics, you need to basically tell the robot where

it should move its limbs.

And then the control algorithm is going to approximate a solution that makes it possible

visit the physics of the robot.

And you have to find the basic solution for making that happen.

And there's probably going to be some regression necessary to get the control architecture to

make these movements.

But those two layers are separate.

Yes.

The thing, the higher level instruction of how you should move and where you should

move is a high level.

I expect that the control level of these robots at some level is dumb.

Because it's just the physical control movement, the motor architecture.

But it's a relatively smart motor architecture.

It's just that there is no high level deliberation about what decisions to make necessarily.

But it doesn't feel like free will or consciousness.

That was not where I was trying to get to.

I think that in our own body, we have that too.

So we have a certain thing that is basically just a cybernetic control architecture that

is moving our limbs.

And deep learning can help in discovering such an architecture if you don't have it

in the first place.

If you already know your hardware, you can maybe handcraft it.

But if you don't know your hardware, you can search for such an architecture.

And this work already existed in the 80s and 90s.

People were starting to search for control architectures by motor babbling and so on

and just use reinforcement learning architectures to discover such a thing.

And now imagine that you have the cybernetic control architecture already inside of you.

And you extend this a little bit.

So you are seeking output, for instance, or rest or and so on.

And you get to have a baby at some point.

And now you add more and more control layers to this.

And the system is reverse engineering its own control architecture and builds a high

level model to synchronize the pursuit of very different conflicting goals.

And this is how I think you get to purposes.

Purposes are models of your goals.

The goals might be intrinsic as the result of the different set point violations that

you have, hunger and thirst for very different things and rest and pain avoidance and so

on.

And you put all these things together and eventually you need to come up with a strategy

to synchronize them all.

And you don't need just to do this alone by yourself because you are state building organisms.

We cannot function as an isolation the way that homo sapiens is set up.

So our own behavior only makes sense when you zoom out very far into a society or even

into ecosystemic intelligence on the planet and our place in it.

So the individual behavior only makes sense in these larger contexts.

And we have a number of priors built into us.

So we are behaving as if we were acting on these high level goals pretty much right from

the start.

And eventually in the course of our life, we can reverse engineer the goals that we

are acting on what actually are our higher level purposes.

And the more we understand that, the more our behavior makes sense.

But this is all at this point, complex stories within stories that are driving our behavior.

Yeah, I just don't know how big of a leap it is to start creating a system that's able

to tell stories within stories, like how big of a leap that is from where currently Boston

Dynamics is or any robot that's operating in the physical space.

And that leap might be big if it requires to solve the hard problem of consciousness,

which is telling a hell of a good story.

I suspect that consciousness itself is relatively simple.

What's hard is perception and the interface between perception and reasoning.

There's, for instance, the idea of the consciousness prior that would be built into such a system

by Yoshua Bengio, and what he describes, and I think that's accurate, is that our own model

of the world can be described through something like an energy function.

The energy function is modeling the contradictions that exist within the model at any given point.

And you try to minimize these contradictions, the tangents in the model.

And to do this, you need to sometimes test things.

You need to conditionally disambiguate figure and ground.

You need to distinguish whether this is true or that is true.

And so on.

Eventually, you get to an interpretation, but you will need to manually depress a few

points in your model to let it snap into a state that makes sense.

And this function that tries to get the biggest dip in the energy function in your model according

to Yoshua Bengio is related to consciousness.

It's a low dimensional discrete function that tries to maximize this dip in the energy

function.

Yeah, I think I would need to dig into details because I think the way he uses the word consciousness

is more akin to self awareness, like modeling yourself within the world as opposed to the

subjective experience, the hard problem.

No, it's not even the self within the world.

The self is the agent and you don't need to be aware of yourself in order to be conscious.

The self is just a particular content that you can have, but you don't have to have.

But you can be conscious in, for instance, a dream at night or during a meditation state

where you don't have a self, where you're just aware of the fact that you are aware.

And what we mean by consciousness and colloquial sense is largely this reflexive self awareness

that we become aware of the fact that you're paying attention, that we are the thing that

pays attention.

We are the thing that pays attention.

Right.

I don't see the awareness that we're aware.

The hard problem doesn't feel like it's solved.

It's called a hard problem for a reason because it seems like there needs to be a major leap.

Yeah, I think the major leap is to understand how it is possible that a machine can dream,

that a physical system is able to create a representation that the physical system is

acting on and that is spun force and so on.

But once you accept the fact that you are not in physics, but that you exist inside

of the story, I think the mystery disappears.

Everything is possible in a story.

Exist inside the story.

Okay.

Your consciousness is being written into the story.

The fact that you experience things is written to the story.

You ask yourself, is this real what I'm seeing?

And your brain writes into the story, yes, it's real.

So what about the perception of consciousness?

So to me, you look conscious.

So the illusion of consciousness, the demonstration of consciousness, I ask for the legged robot,

how do we make this legged robot conscious?

So there's two things that maybe you can tell me if they're neighboring ideas.

One is actually making conscious and the other is make it appear conscious to others.

Are those related?

Let's ask it from the other direction.

What would it take to make you not conscious?

So when you are thinking about how you perceive the world, can you decide to switch from looking

at qualia to looking at representational states?

And it turns out you can.

Yeah.

There is a particular way in which you can look at the world and recognize its machine

nature, including your own.

And in that state, you don't have that conscious experience in this way anymore.

It becomes apparent as a representation.

Everything becomes opaque.

And I think this thing that you recognize everything as a representation, this is typically

what we mean with enlightenment states.

And it can happen on the motivational level, but you can also do this on the experiential

level and the perceptual level.

See, but then I can come back to a conscious state.

Okay, I particularly, I'm referring to the social aspect that the demonstration of consciousness

is a really nice thing at a party when you're trying to meet a new person.

It's a nice thing to know that they're conscious and they can, I don't know how fundamental

consciousness is in human interaction, but it seems like to be at least an important

part.

And I asked that in the same kind of way for robots, you know, in order to create a rich,

compelling human robot interaction, it feels like there needs to be elements of consciousness

within that interaction.

My cat is obviously conscious.

And so my cat can do this party trick.

She also knows that I am conscious.

We're able to have feedback about the fact that we are both acting on models of our own

awareness.

And the question is, how hard is it for the robot, artificially created robot to achieve

cat level and party tricks?

Yes.

So the issue for me is currently not so much on how to build a system that creates a story

about a robot that lives in the world, but to make an adequate representation of the

world.

And the model that you and me have is a unified one.

It's one where you basically make sense of everything that you can perceive every feature

in the world that enters your perception can be relationally mapped to a unified model

of everything.

And we don't have an AI that is able to construct such a unified model yet.

So you need that unified model to do the party trick?

Yes.

I think that it doesn't make sense if this thing is conscious, but not in the same universe

as you, because you could not relate to each other.

So what's the process, would you say, of engineering consciousness in the machine?

What are the ideas here?

So you probably want to have some kind of perceptual system.

This perceptual system is a processing agent that is able to track sensory data and predict

the next frame in the sensory data from the previous frames of the sensory data and the

current state of the system.

So the current state of the system is in perception instrumental to predicting what happens next.

And this means you build lots and lots of functions that take all the blips that you

feel on your skin and that you see on your retina or that you hear and puts them into

a set of relationships that allows you to predict what kind of sensory data, what kind

of sensor of blips, your vector of blips you're going to perceive in the next frame.

This is tuned and it's constantly tuned until it gets as accurate as it can.

You build a very accurate prediction mechanism that is step one of the perception.

So first you predict, then you perceive and see the error in your prediction.

And you have to do two things to make that happen.

One is you have to build a network of relationships that are constraints, that take all the variants

in the world to put each of the variances into a variable that is connected with relationships

to other variables.

And these relationships are computable functions that constrain each other.

So when you see a nose that points a certain direction in space, you have a constraint

that says there should be a phase nearby that has the same direction.

And if that is not the case, you have some kind of contradiction that you need to resolve

because it's probably not a nose, what you're looking at, it just looks like one.

So you have to reinterpret the data until you get to a point where your model converges.

And this process of making the sensory data fit into your model structure is what PRG

calls the assimilation.

And accommodation is the change of the models, where you change your model such a way that

you can assimilate everything.

So you're talking about building a hell of an awesome perception system that's able

to do prediction and perception and correct and improving.

Wait, there's more.

Yes, there's more.

So the first thing that we wanted to do is minimize the contradictions in the model.

And of course, it's very easy to make a model in which you minimize the contradictions just

by allowing that it can be in many, many possible states.

So if you increase degrees of freedom, you will have fewer contradictions.

But you also want to reduce the degrees of freedom because degrees of freedom mean uncertainty.

You want your model to reduce uncertainty as much as possible.

But reducing uncertainty is expensive.

So you have to have a tradeoff between minimizing contradictions and reducing uncertainty.

And you have only a finite amount of compute and experimental time and effort available

to reduce uncertainty in the world.

So you need to assign value to what you observe.

So you need some kind of motivational system that is estimating what you should be looking

at and what you should be thinking about it, how you should be applying your resources

to model what that is.

So you need to have something like convergence links that tell you how to get from the present

state of the model to the next one.

You need to have these compatibility links that tell you which constraints exist and

which constraint violations exist.

And you need to have some kind of motivational system that tells you what to pay attention

to.

So now we have a second agent next to the perceptual agent.

We have a motivational agent.

This is a cybernetic system that is modeling what the system needs, what's important for

the system, and that interacts with the perceptual system to maximize the expected reward.

And you're saying a motivational system is some kind of, like, what is it, a high level

narrative over some lower level?

No, it's just your brainstem stuff, the limbic system stuff that tells you, OK, now you

should get something to eat because I've just measured your geo blood sugar.

So you mean the motivational system, like the lower level stuff, like hungry?

Yes.

So there's basically physiological needs and some cognitive needs and some social needs

and they all interact.

And they all implemented different parts in your nervous system as the motivational system.

But they're basically cybernetic feedback loops.

It's not that complicated, it's just a lot of code.

And so you now have a motivational agent that makes your robot go for the ball or that makes

your worm go to eat food and so on.

And you have the perceptual system that lets it predicts the environment so it's able

to solve that control problem to some degree.

And now what we learned is that it's very hard to build a machine learning system that

looks at all the data simultaneously to see what kind of relationships could exist between

them.

So you need to selectively model the world.

You need to figure out, where can I make the biggest difference if I would put the following

things together?

Sometimes you find a gradient for that, right?

When you have a gradient, you don't need to remember where you came from.

You just follow the gradient until it doesn't get any better.

But if you have a world where the problems are discontinuous and research spaces are

discontinuous, you need to retain memory of what you explored.

You need to construct a plan of what to explore next.

And this thing that means that you have next to this perceptual construction system and

the motivational cybernetics, an agent that is paying attention to what it should select

at any given moment to maximize reward.

And this scanning system, this attention agent is required for consciousness.

And consciousness is its control model.

So it's the indexed memories that this thing retains when it manipulates the perceptual

representations to maximize the value and minimize the conflicts and to increase coherence.

So the purpose of consciousness is to create coherence in your perceptual representations,

remove conflicts, predict the future, construct counterfactual representations so you can coordinate

your actions and so on.

And in order to do this, it needs to form memories.

These memories are partial binding states of the working memory contents that are being

revisited later on to backtrack, to undo certain states, to look for alternatives.

And these indexed memories that you can recall, that is what you perceive as your stream of

consciousness.

And being able to recall these memories, this is what makes you conscious.

If you could not remember what you paid attention to, you wouldn't be conscious.

So consciousness is the index in the memory database, okay.

But let me sneak up to the questions of consciousness a little further.

So we usually relate suffering to consciousness.

So the capacity to suffer, I think to me that's a really strong sign of consciousness, is

a thing that can suffer.

How is that useful?

And, like, in your model, what you just described, which is indexing of memories, and what is

the coherence with the perception, with this predictive thing that's going on in the perception,

how does suffering relate to any of that, you know, the higher level of suffering that

humans do?

Basically, pain is a reinforcement signal.

And pain is a signal that one part of your brain sends to another part of your brain,

or an abstract sense, part of your mind sends to another part of the mind to regulate its

behavior, to tell it the behavior that you're currently exhibiting should be improved.

And this is a signal that I tell you to move away from what you're currently doing and

push into a different direction.

So pain gives you a part of you an impulse to do something differently.

But sometimes this doesn't work, because the training part of your brain is talking to

the wrong region, or because it has the wrong model of the relationships in the world.

Maybe you're mismodeling yourself, or you're mismodeling the relationship of yourself to

the world, or you're mismodeling the dynamics of the world.

So you're trying to improve something that cannot be improved by generating more pain.

But the system doesn't have any alternative.

So it doesn't get better.

What do you do if something doesn't get better, and you want it to get better?

You increase the strength of the signal, and when the signal becomes chronic, when it becomes

permanent, without a change inside, this is what we call suffering.

And the purpose of consciousness is to deal with contradictions, with things that cannot

be resolved.

The purpose of consciousness, I think, is similar to a conductor in an orchestra.

When everything works well, the orchestra doesn't need much of a conductor, as long

as it's coherent.

But when there is a lack of coherence, or something is consistently producing disharmony

and mismatches, then the conductor becomes alert and interacts with it.

So suffering attracts the activity of our consciousness.

And the purpose of that is ideally that we bring new layers online, new layers of modeling

that are able to create a model of the dysregulation so we can deal with it.

And this means that we typically get higher level consciousness, so to speak.

We get some consciousness above our pay grade, maybe, if we have some suffering early in

our life.

There are really interesting people at trauma early on in their childhood.

And trauma means that you are suffering an injury for which the system is not prepared,

which it cannot deal with, which it cannot insulate itself from, so something breaks.

And this means that the behavior of the system is permanently disturbed in a way that some

mismatch exists now in a regulation, that just by following your impulses, by following

the pain in the direction which it hurts, the situation doesn't improve, but get worse.

And so what needs to happen is that you grow up, and that part that is grown up is able

to deal with the part that is stuck in this earlier phase.

Yeah, so it leads to growth, adding extra layers to your cognition.

Let me ask you then, because I got a stick on suffering, the ethics of the whole thing.

So not our consciousness, but the consciousness of others, you've tweeted, one of my biggest

fears is that insects could be conscious.

The amount of suffering on earth would be unthinkable.

So when we think of other conscious beings, is suffering a property of consciousness that

we're most concerned about?

So I'm still thinking about robots, how to make sense of other nonhuman things that appear

to have the depth of experience that humans have.

And to me, that means consciousness and the darkest side of that, which is suffering,

the capacity to suffer.

So I start thinking, how much responsibility do we have for those other conscious beings?

That's where the definition of consciousness becomes most urgent.

Like having to come up with the definition of consciousness becomes most urgent, is who

should we, and should we not be torturing?

There's no general answer to this.

Was jingles kind of doing anything wrong?

It depends on how you look at it.

Well, he drew a line somewhere where this is us and that's them.

It's the circle of empathy.

It's like, you don't have to use the word consciousness, but these are the things that

matter to me if they suffer or not, and these are the things that don't matter to me.

But when one of his commanders failed him, he broke his spine and let him die in a horrible

way.

And so in some sense, I think he was indifferent to suffering, or he was not indifferent in

the sense that he didn't see it as useful if he inflicted suffering.

But he did not see it as something that had to be avoided.

That was not the goal.

The question was, how can I use suffering and the infliction of suffering to reach my goals

from his perspective?

I see so like different societies throughout history put different value on the different

individuals, different psychies, but also even the objective of avoiding suffering.

Some societies probably, I mean, this is where religious belief really helps, that afterlife,

that doesn't matter that you suffer or die, what matters is you suffer honorably so that

you enter the afterlife.

It seems to be superstitious to me, basically beliefs that assert things for which no evidence

exist are incompatible with sound epistemology.

And I don't think that religion has to be superstitious, otherwise it should be condemned

in all cases.

Okay, actually, you're somebody who's saying, we live in a dream world, we have zero evidence

for anything.

It's not the case.

They are limits to what languages can be constructed, mathematics breaks solid evidence

for its own structure.

And once we have some idea of what languages exist and how a system can learn and what

learning itself is in the first place, and so on, we can begin to realize that our intuitions

that we are able to learn about the regularities of the world and minimize surprise and understand

the nature of our own agency to some degree of abstraction.

That's not an illusion.

So just because we live in a dream world doesn't mean mathematics can't give us

a consistent glimpse of physical, of objective reality.

We can basically distinguish useful encodings from useless encodings.

And when we apply our truth seeking to the world, we know we usually cannot find out

whether a certain thing is true.

What we typically do is we take the state vector of the universe separated into separate

objects that interact with each other through interfaces.

And this distinction that we are making is not completely arbitrary.

It's done to optimize the compression that we can apply to our models of the universe.

So we can predict what's happening with our limited resources.

In this sense, it's not arbitrary.

But the separation of the world into objects that are somehow discrete and interacting

with each other is not the true reality.

The boundaries between the objects are projected into the world, not arbitrarily projected,

but still it's only an approximation of what's actually the case.

And we sometimes notice that we run into contradictions when we try to understand high level things

like economic aspects of the world and so on or political aspects or psychological aspects

where we make simplifications and the objects that we are using to separate the world are

just one of many possible projections of what's going on.

And so it's not in this postmodernist sense completely arbitrary and you're free to pick

what you want or dismiss what you don't like because it's all stories.

No, that's not true.

You have to show for every model of how well it predicts the world.

So the confidence that you should have in the entities of your models should correspond

to the evidence that you have.

Can I ask you in a small tangent to talk about your favorite set of ideas and people, which

is postmodernism?

What?

What is postmodernism?

How would you define it?

And why, to you, is it not a useful framework of thought?

Postmodernism is something that I'm really not an expert on.

And postmodernism is a set of philosophical ideas that is difficult to lump together,

that is characterized by some useful thinkers, some of them poststructuralists and so on.

And I'm mostly not interested in it because I think that it's not leading me anywhere

that I find particularly useful.

It's mostly, I think, born out of the insight that the ontologies that we impose on the

world are not literally true and that we can often get to a different interpretation by

the world by using a different ontology that is different separation of the world into

interacting objects.

But the idea that this makes the world a set of stories that are arbitrary, I think, is

wrong, and the people that are engaging in this type of philosophy are working in an

area that I largely don't find productive.

There is nothing useful coming out of this.

So this idea that truth is relative is not something that has, in some sense, informed

physics or theory of relativity, and there is no feedback between those.

There is no meaningful influence of this type of philosophy on the sciences or on engineering

or on politics.

But there is a very strong information of this on ideology because it basically has become

an ideology that is justifying itself by the notion that truth is a relative concept.

And it's not being used in such a way that the philosophers or sociologists that take

up these ideas say, oh, I should doubt my own ideas because maybe my separation of the

world into objects is not completely valid and I should maybe use the different one

and be open to pluralism of ideas.

But it mostly exists to dismiss the ideas of other people.

It becomes a political weapon of sorts to achieve power.

Basically there's nothing wrong, I think, with developing a philosophy around this.

But to develop norms around the idea that truth is something that is completely negotiable

is incompatible with the scientific project.

And I think if the academia has no defense against the ideological parts of the postmodernist

movement, it's doomed.

Right.

You have to acknowledge the ideological part of any movement, actually, including postmodernism.

Well, the question is what an ideology is.

And to me, an ideology is basically a viral memeplex that is changing your mind in such

a way that reality gets warped.

It gets warped in such a way that you're being cut off from the rest of human thought space.

And you cannot consider things outside of the range of ideas of your own ideology as

possibly true.

Right.

So I mean, there's certain properties to an ideology that make it harmful.

One of them is that dogmatism of just certainty, dogged certainty in that you're right.

You have the truth and nobody else does.

But what is creating this certainty?

It's very interesting to look at the type of model that is being produced.

Is it basically just a strong prior?

And you tell people, oh, this idea that you consider to be very true, the evidence for

this is actually just much weaker than you thought.

And look here at some studies, no, this is not how it works.

It's usually normative, which means some thoughts are unthinkable because they would

change your identity into something that is no longer acceptable.

And this cuts you off from considering an alternative.

And many de facto religions use this trick to lock people into a certain mode of thought.

And this removes agency over your own thoughts.

And it's very ugly to me.

It's basically not just a process of domestication, but it's actually an intellectual castration

that happens.

It's an inability to think creatively and to bring forth new thoughts.

And ask you about substances, chemical substances that affect the video game, the dream world.

So psychedelics, that increasing have been getting a lot of research done on them.

So in general, psychedelics, psilocybin, MDMA, but also a really interesting one, the big

one, which is DMT.

What and where are the places that these substances take the mind that is operating in the dream

world?

Do you have an interesting sense how this throws a wrinkle into the prediction model?

Is it just some weird little quirk?

Or is there some fundamental expansion of the mind going on?

I suspect that a way to look at psychedelics is that they induce particular types of lucid

dreaming states.

So it's a state in which certain connections are being severed in your mind and no longer

active.

Your mind basically gets free to move in a certain direction because some inhibition,

some particular inhibition doesn't work anymore.

And as a result, you might stop having yourself or you might stop perceiving the world as three

dimensional.

And you can explore that state.

And I suppose that for every state that can be induced with psychedelics, there are people

that are naturally in that state.

So sometimes psychedelics that shift you through a range of possible mental states.

And they can also shift you out of the range of permissible mental states that is where

you can make predictive models of reality.

And what I observe in people that use psychedelics a lot is that they tend to be overfitting.

Fitting means that you are using more bits for modeling the dynamics of a function than

you should.

And so you can fit your curve to extremely detailed things in the past, but this model

is no longer predictive for the future.

What is it about psychedelics that forces that?

I thought it would be the opposite.

I thought it's a good mechanism for generalization, for regularization.

So it feels like psychedelics expansion of the mind, like forcing your model to be nonpredictive

is a good thing.

Meaning like it's almost like, okay, what I would say is psychedelics are akin to traveling

to a totally different environment, like going, if you've never been to like India or something

like that from the United States, very different set of people, different culture, different

food, different roads and values and all those kinds of things.

So psychedelics can, for instance, teleport people into a universe that is hyperbolic,

which means that if you imagine a room that you are in, you can turn around 360 degrees

and you didn't go full circle.

You need to go 20 degrees to full circle.

So the things that people learn in that state cannot be easily transferred in this universe

that we are in.

It could be that if they're able to abstract and understand what happened to them, that

they understand that some part of their spatial cognition has been desynchronized and has

found a different synchronization.

And this different synchronization happens to be a hyperbolic one, right?

So you learn something interesting about your brain.

It's difficult to understand what exactly happened, but we get pretty good idea once

we understand how the brain is representing geometry.

Yeah, but it doesn't give you a fresh perspective on the physical reality.

Who's making that sound?

Is it inside my head or is it external?

Well, there is no sound outside of your mind, but it's making sense of phenomenon physics.

Yeah, in the physical reality, there's sound waves traveling through air.

Okay.

So that's our model of what happened.

That's our model of what happened.

Right.

Don't psychedelics give you a fresh perspective on this physical reality, not this physical

reality, but this more...

What do you call the dream world that's mapped directly to...

The purpose of dreaming at night, I think, is data augmentation.

Exactly.

So you basically...

That's very similar to psychedelics.

...change parameters about the things that you have learned.

And for instance, when you are young, you have seen things from certain perspectives,

but not from others.

So your brain is generating new perspectives of objects that you already know, which means

they can learn to recognize them later from different perspectives, and I suspect that's

the reason why many of us remember to have flying dreams as children, because it's just

different perspectives of the world that we already know, and that it starts to generate

these different perspective changes, and then it fluidly turns this into a flying dream

to make sense of what's happening.

So you fill in the gaps, and suddenly you see yourself flying.

And similar things can happen with semantic relationships.

So it's not just spatial relationships, but it can also be the relationships between

ideas that are being changed.

And it seems that the mechanisms that make that happen during dreaming are interacting

with these same receptors that are being simulated by psychedelics.

So I suspect that there is a thing that I haven't read really about, the way in which

dreams are induced in the brain is not just that the activity of the brain gets tuned

down because your eyes are closed and you no longer get enough data from your eyes, but

there is a particular type of neurotransmitter that is saturating your brain during these

phases, during the RM phases, and you produce controlled hallucinations.

And psychedelics are linking into these mechanisms, I suspect.

So isn't that another trickier form of data augmentation?

Yes.

But it's also data augmentation that can happen outside of the specification that your brain

is tuned to.

So basically people are overclocking their brains, and that produces states that are

subjectively extremely interesting.

But from the outside, very suspicious.

I think I'm overapplying the metaphor of a neural network in my own mind, which I just

think that doesn't lead to overfitting.

But you were just sort of anecdotally saying my experiences with people that have no psychedelics

are that kind of quality.

I think it typically happens.

So if you look at people like Timothy Leary, and he has written beautiful manifestos about

the effect of LSD on people, he genuinely believed, he writes in these manifestos that

in the future, science and art will only be done on psychedelics because it's so much

more efficient and so much better.

And he gave LSD to children in this community of a few thousand people that he had near San

Francisco.

And basically he was losing touch with reality.

He did not understand the effects that the things that he was doing would have on the

reception of psychedelics by society, because he was unable to think critically about what

happened.

What happened was that he got in a euphoric state, that euphoric state happened because

he was overfitting.

He was taking this sense of euphoria and translating it into a model of actual success in the

world.

He was feeling better, limitations had disappeared, that he experienced to be existing, but he

didn't get superpowers.

I understand what you mean by overfitting now.

There's a lot of interpretation to the term overfitting in this case.

But I got you.

So he was getting positive rewards from a lot of actions that he shouldn't have put

in.

Yeah, but not just this.

So if you take, for instance, John Lilly, who was studying dolphin languages and aliens

and so on, a lot of people that used psychedelics became very loopy.

And the typical thing that you notice when people are on psychedelics is that they are

in a state where they feel that everything can be explained now.

Everything is clear.

Everything is obvious.

And sometimes they have indeed discovered a useful connection, but not always.

Very often these connections are over interpretations.

I wonder, there's a question of correlation versus causation.

And also I wonder if it's the psychedelics or if it's more the social, like being the

outsider and having a strong community of outsiders and having a leadership position

in an outsider cult like community, that could have a much stronger effect of overfitting

than do psychedelics themselves, the actual substances.

Because it's a counterculture thing.

So it could be that as opposed to the actual substance.

If you're a boring person who wears a suit and tie and works at a bank and takes psychedelics,

that could be a very different effect of psychedelics on your mind.

I'm just sort of raising the point that the people you referenced are already weirdos.

I'm not sure exactly.

No, not necessarily.

A lot of the people that tell me that they used psychedelics in a useful way started

out as squares and were liberating themselves because they were stuck.

They were basically stuck in local optimum of their own self model, of their relationship

to the world, and suddenly they had data augmentation.

They basically saw an experience, a space of possibilities.

They experienced what it would be like to be another person.

And they took important lessons from that experience back home.

Yeah.

I mean, I love the metaphor of data augmentation because that's been the primary driver of

self supervised learning in the computer vision domain is data augmentation.

So it's funny to think of like chemically induced data augmentation in the human mind.

There's also a very interesting effect that I noticed.

I know several people who are speared to me that LSD has cured their migraines.

So severe cluster headaches or migraines that didn't respond to standard medication that

disappeared after a single dose.

And I don't recommend anybody doing this, especially not in the US where it's illegal.

And there are no studies on this for that reason.

But it seems that anecdotally that it basically can reset the serotonergic system.

So it's basically pushing them outside of their normal boundaries.

And as a result, it needs to find a new equilibrium and some people that equilibrium is better.

But it also follows that in other people, it might be worse.

So if you have a brain that is already teetering on the boundary to psychosis, it can be permanently

pushed over that boundary.

Well, that's why you have to do good science, which they're starting to do on all these different

substances of how well it actually works for the different conditions like MDMA seems

to help with PTSD.

Same with psilocybin that, you know, you need to do good science, meaning large studies

of large N.

Yeah.

Based on the existing studies with MDMA, it seems that if you look at Rick Doblin's

work and what he has published about this and talks about MDMA seems to be a psychologically

relatively safe drug, but it's physiologically not very safe.

That is, there is neurotoxicity if you would use a too large dose.

And if you combine this with alcohol, which a lot of kids do in party settings during

raves and so on, it's very hippotoxic, so basically you can kill your liver.

And this means that it's probably something that is best and most productively used in

clinical setting by people who really know what they're doing.

And I suspect that's also true for the other psychedelics that is, while the other psychedelics

are probably not as toxic as say alcohol, the effects on the psyche can be much more

profound and lasting.

Yeah.

Well, as far as I know, psilocybin, so mushrooms, magic mushrooms, as far as I know in terms

of the studies they're running, I think have no, like they're allowed to do what they're

calling heroic doses.

So that one does not have a toxicity, so they could do like huge doses in a clinical setting

when they're doing study on psilocybin, which is kind of fun.

Yeah.

It seems that most of the psychedelics work in extremely small doses, which means that

the effect on the rest of the body is relatively low, and MDMA is probably the exception.

Maybe ketamine can be dangerous in larger doses because it can depress breathing and

so on.

But the LSD and psilocybin work in very, very small doses, at least the active part of them

of psilocybin LSD is only the active part.

But the effect that it can have on your mental wiring can be very dangerous, I think.

Let's talk about AI a little bit.

What are your thoughts about GPT3 and language models trained with cell supervised learning?

It came out quite a bit ago, but I wanted to get your thoughts on it.

In the 90s, I was in New Zealand and I had an amazing professor, Ian Witten, who realized

I was bored in class and put me in his lab, and he gave me the task to discover grammatical

structure in an unknown language.

The unknown language that I picked was English because it was the easiest one to find a corpus

four construct one.

He gave me the largest computer at the whole university.

It had two gigabytes of RAM, which was amazing.

I wrote everything in C with some in memory compression to do statistics over the language.

Next would create a dictionary of all the words, which basically tokenizes everything

and compresses things so they don't need to store the whole word, but just a code for

every word.

Then I was taking this all apart in sentences and I was trying to find all the relationships

between all the words and the sentences and do statistics over them.

That proved to be impossible because the complexity is just too large.

If you want to discover the relationship between an article and a noun, and there are three

adjectives in between, you cannot do ngram statistics and look at all the possibilities

that can exist, at least not with the resources that we had back then.

I realized I need to make some statistics over what I need to make statistics over.

I wrote something that was pretty much a hack that did this for at least first order relationships.

I came up with some kind of mutual information graph that was indeed discovering something

that looks exactly like the grammatical structure of the sentence.

Just by trying to encode the sentence in such a way that the words would be written in the

optimal order inside of the model.

What I also found is that if we would be able to increase the resolution of that and not

just use this model to reproduce grammatically correct sentences, we would also be able to

correct stylistically correct sentences by just having more bits in these relationships.

If we wanted to have meaning, we would have to go much higher order.

I didn't know how to make higher order models back then without spending way more years

in research on how to make the statistics over what we need to make statistics over.

This thing that we cannot look at the relationships between all the bits in your input is being

solved in different domains in different ways.

In computer graphics, the computer vision standard method for many years now is convolutional

neural networks.

convolutional neural networks are hierarchies of filters that exploit the fact that neighboring

pixels in images are usually semantically related and distance pixels in images are usually

not semantically related.

You can just by grouping the pixels that are next to each other hierarchically together

reconstruct the shape of objects.

This is an important prior that we build into these models so they can converge quickly.

This doesn't work in language for the reason that adjacent words are often but not always

related and distant words are sometimes related while the words in between are not.

How can you learn the topology of language?

I think for this reason that this difficulty existed, the transformer was invented in natural

language processing not in vision.

What the transformer is doing, it's a hierarchy of layers where every layer learns what to

pay attention to in the given context in the previous layer.

What to make the statistics over.

And the context is significantly larger than the adjacent word.

Yes.

The context that GP3 has been using, the transformer itself is from 2017 and it wasn't using that

large of a context.

OpenAI has basically scaled up this idea as far as they could at the time.

The context is about 2048 symbols, tokens in the language.

These symbols are not characters, but they take the words and project them into a vector

space where words that are statistically cooccurring a lot are neighbors already.

So it's already a simplification of the problem a little bit.

Every word is basically a set of coordinates in a high dimensional space.

And then they use some kind of trick to also encode the order of the words in a sentence

or in the not just sentence, but 2048 tokens is about couple pages of text or two and a

half pages of text.

And so they managed to do pretty exhaustive statistics over the potential relationships

between two pages of text, which is tremendous, right?

I was just using a single sentence back then and I was only looking for first order relationships

and they were really looking for much, much higher level relationships.

And what they discover after they've fed this enormous amount of trim and they're pretty

much the written internet or the subset of it that had some quality, but substantial portion

of the common flaw that they're not only able to reproduce style, but they're also able

to reproduce some pretty detailed semantics like being able to add three digit numbers

and multiply two digit numbers or to translate between pro and languages and things like

that.

So the results that GBT3 got, I think were amazing.

By the way, I actually didn't check carefully.

It's funny you just mentioned how you coupled semantics to the multiplication.

Is it able to do some basic math on two digit numbers?

Yes.

Okay, interesting.

I thought, I thought there's a lot of failure cases.

Yeah, it basically fails if you take larger digit numbers.

So four digit numbers and so on makes carrying mistakes and so on.

And if you take larger numbers, you don't get useful results at all.

And this could be an issue of the training set where there are not many examples of successful

long form addition and standard human written text.

Humans aren't very good at doing three digit numbers either.

Yeah, they're not, you're not writing a lot about it.

And the other thing is that the loss function that is being used is only minimizing surprises.

So it's predicting what comes next in a typical text.

It's not trying to go for causal closure first as we do.

Yeah.

But the fact that that kind of prediction works to generate text that's semantically

rich and consistent is interesting.

So yeah, so it's amazing that it's able to generate semantically consistent text.

It's not consistent.

So the problem is that it loses coherence at some point.

But it's also, I think, not correct to say that GP2C is unable to deal with semantics

at all because you ask it to perform certain transformations in text and it performs these

transformation in text.

And the kind of additions that is able to perform are transformations in text, right?

And there are proper semantics involved.

You can also do more.

There was a paper that was generating lots and lots of mathematically correct text and

was feeding this into a transformer.

And as a result, it was able to learn how to do differentiation, integration in race

that according to the authors, Mathematica could not, to which some of the people in

Mathematica responded that they were not using Mathematica in the right way and so on.

I have not really followed the resolution of this conflict.

This part, as a small tangent, I really don't like in machine learning papers, which they

often do anecdotal evidence.

They'll find one example in some kind of specific use of Mathematica and demonstrate, look,

they'll show successes and failures, but they won't have a very clear representation of

how many cases this actually represents.

Yes, but I think as a first paper, this is a pretty good start.

And so the take home message, I think, is that the authors could get better results

from this and their experiments than they could get from the vein, which they were using

computer algebra systems, which means that was not nothing.

And it's able to perform substantially better than GPTSV can, based on a much larger amount

of training data, using the same underlying algorithm.

Let me ask, again, so I'm using your tweets as if this is like Play Doh, right?