The following is a conversation with Wojciech Zaremba,

cofounder of OpenAI,

which is one of the top organizations in the world

doing artificial intelligence research and development.

Wojciech is the head of language and cogeneration teams,

building and doing research on GitHub copilot,

OpenAI codecs, and GPT3, and who knows,

four, five, six, n, and n plus one.

And he also previously led OpenAI's robotic efforts.

These are incredibly exciting projects to me

that deeply challenge and expand our understanding

of the structure and nature of intelligence.

The 21st century, I think,

may very well be remembered

for a handful of revolutionary AI systems

and their implementations.

GPT codecs and applications of language models

and transformers in general

to the language and visual domains

may very well be at the core of these AI systems.

To support this podcast, please check out our sponsors.

They're listed in the description.

This is the Lex Friedman podcast,

and here is my conversation with Wojciech Zaremba.

You mentioned that Sam Altman asked

about the Fermi Paradox,

and other people at OpenAI

had really sophisticated interesting answers.

So that's when you knew

this is the right team to be working with.

So let me ask you about the Fermi Paradox, about aliens.

Why have we not found overwhelming evidence

for aliens visiting Earth?

I don't have a conviction in the answer,

but rather kind of probabilistic perspective

on what might be, let's say, possible answers.

It's also interesting that the question itself,

even can't touch on your typical question

of what's the meaning of life,

because if you assume that we don't see aliens

because they destroy themselves,

that kind of upgrades the focus

on making sure that we won't destroy ourselves.

But at the moment, the place where I am actually

with my belief, and these things also change over the time,

is I think that we might be alone in the universe,

which actually makes life more, or let's say consciousness,

life more kind of valuable.

And that means that we should more appreciate it.

Have we always been alone?

So what's your intuition about our galaxy, our universe?

Is it just sprinkled with graveyards

of intelligent civilizations,

or is life, intelligent life truly unique?

At the moment, my belief that it is unique.

But I would say I could also,

there was like some footage released with UFO objects,

which makes me actually doubt my own belief.

Yes.

Yeah, I can tell you one crazy answer that I have heard.

Yes.

So apparently when you look actually

at the limits of computation,

you can compute more if the temperature

of the universe would drop down.

So one of the things that aliens might want to do

if they are truly optimizing to maximize amount of compute,

which maybe can lead to more, let's say simulations or so,

it's instead of wasting current entropy of the universe,

because we by living,

we are actually somewhat wasting entropy,

then you can wait for the universe to cool down,

such that you have more computation.

So that's kind of a funny answer.

I'm not sure if I believe in it,

but that would be one of the reasons

why you don't see aliens.

It's also possible, see some people say that,

maybe there is not that much point

in actually going to other galaxies,

if you can go inwards.

So there is no limits of what could be an experience

if we could connect machines to our brains,

while there are still some limits

if we want to explore the universe.

Yeah, there could be a lot of ways to go inwards too.

Once you figure out some aspect of physics

we haven't figured out yet,

maybe you can travel to different dimensions.

I mean, travel in three dimensional space

may not be the most fun kind of travel.

There may be like just a huge amount

of different ways to travel.

It doesn't require a spaceship going slowly

in 3D space to space time.

It also fills in one of the problems

is that speed of light is low and the universe is vast.

And it seems that actually most likely

if we want to travel very far,

then we would instead of actually sending spaceships

with humans that wait a lot,

we would send something similar

to what Uri Miller is working on.

These are like a huge sail which is at first powered.

There is a shot of laser from an air

and it can propel it to quarter of speed of light.

And the sail itself contains a few grams of equipment.

And that might be the way to actually transport matter

through the universe.

But then when you think what would it mean for humans,

it means that we would need to actually

put their 3D printer and 3D print a human on another planet.

I don't know, play them YouTube or let's say,

or like a 3D print like huge human right away

or maybe a womb or so, yeah.

With our current techniques of archeology,

if a civilization was born and died long enough to go

on Earth, we wouldn't be able to tell.

And so that makes me really sad.

And so I think about Earth in that same way.

How can we leave some remnants if we do destroy ourselves?

How can we leave remnants for aliens

in the future to discover?

Like here's some nice stuff we've done.

Like Wikipedia and YouTube.

Do we have it like in a satellite orbiting Earth

with a hard drive?

Like how do we say,

how do we back up human civilization for the good parts

or all of it is good parts

so that it can be preserved longer than our bodies can?

That's kind of a, it's a difficult question.

It also requires the difficult acceptance

of the fact that we may die.

And if we die, we may die suddenly as a civilization.

So let's see, I think it kind of depends on the cataclysm.

We have observed in other parts of the universe

that birds of gamma rays,

these are high energy rays of light

that actually can apparently kill entire galaxy.

So there might be actually nothing event to,

nothing to protect us from it.

I'm also, and I'm looking actually at the past civilizations.

So it's like Aztecs or so,

they disappear from the surface of the Earth

and one can ask, why is it the case?

And the way I'm thinking about it is,

you know, that definitely they had some problem

that they couldn't solve.

And maybe there was a flood

and all of a sudden they couldn't drink,

there was no potable water and they all died.

And I think that so far,

the best solution to such a problems is, I guess, technology.

So I mean, if they would know that you can just boil water

and then drink it after,

then that would save their civilization.

And even now when we look actually at the current pandemic,

it seems that once again, actually science comes to rescue

and somehow science increases size of the action space.

And I think that's a good thing.

Yeah, but nature has a vastly larger action space.

Yeah, but still it might be a good thing for us

to keep on increasing action space.

Well, okay, looking at past civilizations, yes.

But looking at the destruction of human civilization,

perhaps expanding the action space

will add actions that are easily acted upon,

easily executed and as a result destroy us.

So let's see.

I was pondering why actually even

we have negative impact on the globe.

Because if you ask every single individual,

they would like to have clean air.

They would like healthy planet,

but somehow it's not the case that as a collective,

we are not going in this direction.

I think that there exists very powerful system

to describe what we value, that's capitalism.

It assigns actually monetary values to various activities.

At the moment, the problem in the current system

is that there are some things which we value.

There is no cost assigned to it.

So even though we value clean air,

or maybe we also value lack of distraction

on the internet or so,

at the moment these quantities,

companies, corporations can pollute them for free.

So in some sense, I wished,

or like that's I guess purpose of politics

to align the incentive systems.

And we are kind of maybe even moving in this direction.

The first issue is even to be able to measure

the things that we value,

then we can actually assign the monetary value to them.

Yeah, and that's so it's getting the data

and also probably through technology,

enabling people to vote and to move money around

in a way that is aligned with their values.

And that's very much a technology question.

So like having one president and Congress

and voting that happens every four years

or something like that,

that's a very outdated idea.

There could be some technological improvements

to that kind of idea.

So I'm thinking from time to time about these topics,

but it also feels to me that it's a little bit like a,

it's hard for me to actually make correct predictions

what is the appropriate thing to do.

I extremely trust Sam Altman,

our CEO on these topics here.

Okay, I'm more on the side of being,

I guess, naïve hippie that, yeah.

That's your life philosophy.

Well, I think self doubt and I think hippie implies optimism.

Those two things are pretty good way to operate.

I mean, still it is hard for me to actually

understand how the politics works

or exactly how the things would play out.

And Sam is really excellent with it.

What do you think is rarest in the universe?

You said we might be alone.

What's hardest to build is another engineering way

to ask that, life, intelligence or consciousness.

So like you said that we might be alone,

which is the thing that's hardest to get to?

Is it just the origin of life?

Is it the origin of intelligence?

Is it the origin of consciousness?

So let me at first explain in my kind of mental model

what I think is needed for life to appear.

So I imagine that at some point there was this primordial

soup of amino acids and maybe some proteins in the ocean.

And some proteins were turning into some other proteins

through reaction.

And you can almost think about this cycle

of what turns into what,

as there is a graph essentially describing

which substance turns into some other substance.

And essentially life means that all the sudden

in the graph has been created that cycle,

such that the same thing keeps on happening

over and over again.

That's what is needed for life to happen.

And in some sense you can think almost

that you have this gigantic graph

and it needs like a sufficient number of edges

for the cycle to appear.

Then from perspective of intelligence and consciousness,

my current intuition is that they might be quite intertwined.

First of all, it might not be that it's like a binary thing

that you have intelligence or consciousness.

It seems to be a more continuous component.

Let's see, if we look for instance

on the event networks recognizing images,

people are able to show that the activations

of these networks correlate very strongly

with activations in visual cortex of some monkeys.

The same seems to be true about language models.

Also, if you for instance look,

if you train agent in 3D world,

at first it barely recognizes what is going on.

Over the time it recognizes foreground from background.

Over the time it knows where there is a foot

and it just follows it.

Over the time it actually starts having a 3D perception.

So it is possible for instance to look

inside of the head of an agent

and ask what would it see if it looks to the right.

And the crazy thing is initially

when the agents are barely trained

that these predictions are pretty bad,

over the time they become better and better,

you can still see that if you ask what happens

when the head is turned by 360 degrees,

for some time they think that the different thing appears

and then at some stage they understand actually

that the same thing supposed to appear.

So they get like an understanding of 3D structure.

It's also very likely that they have inside some level

of like a symbolic reasoning,

like they're particularly symbols for other agents.

So when you look at DOTA agents,

they collaborate together

and they have some anticipation of

if they would win battle,

they have some expectations with respect to other agents.

I might be too much anthropomorphizing

how the things look for me.

But then the fact that they have a symbol for other agents

makes me believe that at some stage,

as they are optimizing for skills,

they would have also symbol to describe themselves.

This is like a very useful symbol to have.

And this particularity I would call it

like a self consciousness or self awareness.

And still it might be different from the consciousness.

So I guess the way how I'm understanding

the word consciousness,

I'd say the experience of drinking a coffee

or let's say experience of being a bat.

That's the meaning of the word consciousness.

It doesn't mean to be awake.

Yeah, it feels,

it might be also somewhat related to memory

and recurrent connections.

So it's kind of like,

if you look at anesthetic drugs,

they might be like essentially,

they disturb brain waves

such that maybe memory is not formed.

And so there's a lessening of consciousness

when you do that.

Correct.

And so that's the one way to intuit what is consciousness.

There's also kind of another element here.

It could be that it's this kind of self awareness module

that you described.

Plus the actual subjective experience

is a storytelling module

that tells us a story about what we're experiencing.

The crazy thing.

So let's say, I mean, in meditation,

they teach people not to speak story inside of the head.

And there is also some fraction of population

who doesn't have actually narrator.

I know people who don't have a narrator

and they have to use external people

in order to kind of solve tasks

that require internal narrator.

So it seems that it's possible to have the experience

without the talk.

What are we talking about

when we talk about the internal narrator?

So is that the voice when you like read the book?

Yeah, I thought that that's what you are referring to.

Well, I was referring more on it,

like not an actual voice.

I meant like there's some kind of like subjective experience

feels like it's fundamentally about storytelling

to ourselves.

It feels like the feeling is a story

that is much simpler abstraction

than the raw sensory information.

So it feels like it's a very high level abstraction

that is useful for me to feel like entity in this world.

Most useful aspect of it is that because I'm conscious,

I think there's an intricate connection

to me not wanting to die.

So like it's a useful hack to really prioritize not dying.

Like those seem to be somehow connected.

So I'm telling the story of like it's richly feels

like something to be me and the fact that me exists

in this world, I want to preserve me.

And so that makes it a useful agent hack.

So I will just refer maybe to the first part,

as you said about that kind of story of describing who you are.

I was thinking about it even.

So, you know, obviously I like thinking about it

thinking about consciousness.

I like thinking about the AI as well.

And I'm trying to see analogies of these things in AI,

what would it correspond to?

So, you know, OpenAI trained a model called GPT

which can generate pretty amusing text on arbitrary topic.

And one way to control GPT

is by putting into prefix at the beginning of the text

some information what would be the story about.

You can have even chat with GPT by saying

that the chat is with Lex or Elon Musk or so.

And GPT would just pretend to be you or Elon Musk or so.

And it almost feels that this story

that we give ourselves to describe our life,

it's almost like things that you put into context of GPT.

Yeah, and it generates the,

but the context we provide to GPT is multimodal.

It's more, so GPT itself is multimodal.

GPT itself hasn't learned actually

from experience of single human,

but from the experience of humanity.

It's a chameleon.

You can turn it into anything.

And in some sense, by providing context,

it behaves as the thing that you wanted it to be.

It's interesting that people have a stories of who they are.

And as I said, these stories,

they help them to operate in the world.

But it's also interesting,

I guess various people find it out through meditation or so

that there might be some patterns

that you have learned when you were a kid

that actually are not serving you anymore.

And you also might be thinking that that's who you are

and that's actually just a story.

Yeah, so it's a useful hack,

but sometimes it gets us into trouble.

It's a local optima.

It's a local optima.

You wrote that Stephen Hawking, he tweeted,

Stephen Hawking asked what breathes fire into equations,

which meant what makes given mathematical equations

realize the physics of a universe.

Similarly, I wonder what breathes fire into computation.

What makes given computation conscious?

Okay, so how do we engineer consciousness?

How do you breathe fire and magic into the machine?

So it seems clear to me

that not every computation is conscious.

I mean, you can, let's say,

just keep on multiplying one matrix over and over again.

And it might be gigantic matrix.

You can put a lot of computation.

I don't think it would be conscious.

So in some sense, the question is,

what are the computations which could be conscious?

I mean, so one assumption is

that it has to do purely with computation

that you can abstract away matter.

Other possibilities, it's very important

was the realization of computation

that it has to do with some force fields or so

and they bring consciousness.

At the moment, my intuition is that it can be

fully abstracted away.

So in case of computation, you can ask yourself,

what are the mathematical objects

or so that could bring such a properties?

So for instance, if we think about the models,

AI models, what they truly try to do,

or like a models like GPT is,

they try to predict the next word or so.

And this turns out to be equivalent to compressing text.

And because in some sense, compression means

that you learn the model of reality

and you have just to remember what are your mistakes.

The better you are in predicting the,

and in some sense, when we look at our experience,

also when you look for instance at the car driving,

you know in which direction it will go.

You are good like in prediction.

And it might be the case that the consciousness

is intertwined with compression.

It might be also the case that self consciousness

has to do with compressor trying to compress itself.

So I was just wondering what are the objects

in mathematics or computer science,

which are mysterious that could have to do

with consciousness.

And then I thought, you know, you see in mathematics

there is something called Gadot theorem,

which means again, if you have sufficiently

complicated mathematical system,

it is possible to point the mathematical system

back on itself.

In computer science, there is something called

helping problem, it's somewhat similar construction.

So I thought that, you know, if we believe

that under assumption that consciousness

has to do with compression,

then you could imagine that as you are

keeping on compressing things,

then at some point it actually makes sense

for the compressor to compress itself.

Metacompression, consciousness is metacompression.

That's an idea, and in some sense, you know, the crazy.

Thank you.

So, but do you think if we think of a touring machine,

a universal touring machine,

can that achieve consciousness?

So is there something beyond our traditional definition

of computation that's required?

So it's a specific computation.

And I said this computation has to do with compression.

And the compression itself, maybe other way of putting it

is like you are internally creating the model of reality.

In order, it's like you try insight to simplify reality

in order to predict what's gonna happen.

And that also feels somewhat similar to how I think

actually about my own conscious experience.

So clearly I don't have access to reality.

The only access to reality is through, you know,

cable going to my brain.

And my brain is creating a simulation of reality.

And I have access to the simulation of reality.

Are you by any chance aware of the Hutter Prize,

Marcus Hutter?

He made this prize for compression of Wikipedia pages.

And there's a few qualities to it.

One, I think has to be perfect compression,

which makes, I think that little quirk

makes it much less applicable to the general task

of intelligence, because it feels like intelligence

is always going to be messy.

Like perfect compression feels like it's not the right goal,

but it's nevertheless a very interesting goal.

So for him, intelligence equals compression.

And so the smaller you make the file,

given a large Wikipedia page,

the more intelligent the system has to be.

Yeah, that makes sense.

So you can make perfect compression if you store errors.

And I think that actually what he meant

is you have algorithm plus errors.

By the way, Hutter is a,

he was a PhD advisor of Shanleck,

who is a deep mind cofounder.

Yeah, yeah, so there's an interesting,

and now he's a deep mind.

There's an interesting network of people.

And he's one of the people that I think

seriously took on the task of

what would an AGI system look like?

I think for the longest time,

the question of AGI was not taken seriously,

or rather rigorously.

And he did just that.

Like mathematically speaking,

what would the model look like?

If you remove the constraints of it having to be,

having to have a reasonable amount of memory,

reasonable amount of running time complexity,

computation time, what would it look like?

And essentially it's a half math,

half philosophical discussion of how would it,

like a reinforcement learning type of framework

look like for an AGI?

Yeah, so he developed the framework

even to describe what's optimal

with respect to reinforcement learning.

Like there is a theoretical framework,

which is as you said, under assumption,

there is infinite amount of memory and compute.

There was actually one person

before his name is Solomonov,

who extended Solomonov work to reinforcement learning,

but there exists a theoretical algorithm,

which is optimal algorithm to build intelligence.

And I can actually explain you the algorithm.

Yes, let's go, let's go, let's go.

So the task itself, you can...

Can I just pause how absurd it is

for a brain in a skull

trying to explain the algorithm for intelligence?

Just go ahead.

It is pretty crazy.

It is pretty crazy that the brain itself

is actually so small and it can ponder.

How to design algorithms

that optimally solve the problem of intelligence?

Okay, all right, so what's the algorithm?

So let's see.

So first of all, the task itself is describe us,

you have infinite sequence of zeros and ones, okay?

You read n bits and you are about to predict

n plus one bit.

So that's the task.

And you could imagine that every task

could be casted as such a task.

So if, for instance, you have images and labels,

you can just turn every image

into a sequence of zeros and ones,

then label, you concatenate labels,

and that's actually...

You could start by having training data first,

and then afterwards you have test data.

So theoretically, any problem could be casted

as a problem of predicting zeros and ones

on this infinite tape.

So let's say you read already n bits

and you want to predict n plus one bit.

And I will ask you to write every possible program

that generates these n bits, okay?

So, and you can have, you choose programming language.

It can be Python or C++.

And the difference between programming languages might be,

there is a difference by constant, asymptotically,

your predictions will be equivalent.

So you read n bits, you enumerate all the programs

that produce these n bits in their output.

And then in order to predict n plus one bit,

you actually weight the programs according to their length.

And there is some specific formula how you weight them.

And then the n plus one bit prediction

is the prediction from each of these program

according to their weight.

Like statistically, you pick.

Statistically, yeah.

So the smaller the program,

the more likely you are to pick its output.

So that algorithm is grounded in the hope

or the intuition that the simple answer is the right one.

It's a formalization of it.

Yeah.

It also means like if you would ask the question

after how many years would, you know, stand explode,

you can say, hmm, it's more likely the answer

is two to some power because they're shorter program.

Yeah.

And then other.

Well, I don't have a good intuition

about how different the space of short programs

are from the space of large programs.

Like what is the universe for short programs?

Like run things.

So, as I said, the things have to agree with n bits.

So even if you have, you need to start,

okay, if you have very short program

and they're like a still some has,

if it's not perfect with prediction of n bits,

you have to start errors.

What are the errors?

And that gives you the full program

that agrees on n bits.

Oh, so you don't agree perfectly with the n bits

and you store errors.

That's like a longer, a longer programs,

slightly longer program

because it can't take these extra bits of errors.

That's fascinating.

What's your intuition about the programs

that are able to do cool stuff

like intelligence and consciousness?

Are they perfectly, like is it,

is there if then statements in them?

So like, is there a lot of exceptions that they're storing?

So you could imagine if there would be tremendous

amount of each statements,

then they wouldn't be that short.

In case of neural networks,

you could imagine that what happens is

when you start with an uninitialized neural network,

it stores internally many possibilities

how the problem can be solved.

And HDD is kind of magnifying some paths

which are slightly similar to the correct answer.

So it's kind of magnifying correct programs.

And in some sense, HDD is a search algorithm

in the program space.

And the program space is represented

by kind of the wiring inside of the neural network.

And there's like an insane number of ways

how the features can be computed.

Let me ask you the high level basic question.

That's not so basic.

What is deep learning?

Is there a way you'd like to think of it

that is different than like a generic textbook definition?

The thing that I hinted just a second ago

is maybe the closest to how I'm thinking these days

about deep learning.

So now the statement is

neural networks can represent some programs.

Seems that various modules

that we are actually adding up to,

like we want networks to be deep

because we want multiple steps of the computation.

And deep learning provides the way

to represent space of programs which is searchable.

And it's searchable with stochastic gradient descent.

So we have an algorithm to search over

humongous number of programs.

And gradient descent kind of bubbles up

the things that tend to give correct answers.

So a neural network with fixed weights,

that's optimized.

Do you think of that as a single program?

So there is a work by Christopher Oleg

where he, so he works on interpretability

of neural networks.

And he was able to identify inside of the neural network

for instance, a detector of a wheel for a car

or the detector of a mask for a car.

And then he was able to separate them out

and assemble them together

using a simple program for the detector,

for a car detector.

That's like, if you think of traditionally defined programs,

that's like a function within a program

that this particular neural network was able to find.

And you can tear that out,

just like you can copy and paste from Stack Overflow.

So any program is a composition of smaller programs.

Yeah, I mean, the nice thing about the neural networks

is that it allows the things to be more fuzzy

than in case of programs.

In case of programs, you have this like a branch

in this way or that way.

And the neural networks, they have an easier way

to be somewhere in between or to share things.

What to use the most beautiful, surprising idea

in deep learning and the utilization

of these neural networks,

which by the way, for people who are not familiar,

neural networks is a bunch of, what would you say?

It's inspired by the human brain.

There's neurons, there's connection between those neurons.

There's inputs and there's outputs

and there's millions or billions of those neurons.

And the learning happens by adjusting the weights

on the edges that connect these neurons.

Thank you for giving the definition.

I supposed to do it, but I guess you have enough empathy

to listen there to actually know that that might be useful.

No, that's like, so I'm asking Plato

of like, what is the meaning of life?

He's not gonna answer.

You're being philosophical and deep and quite profound

talking about the space of programs,

which is very interesting,

but also for people who are just not familiar

with the hell we're talking about

when we talk about deep learning.

Anyway, sorry, what is the most beautiful

or surprising idea to you in all the time

you've worked at deep learning?

And you worked on a lot of fascinating projects,

the applications of neural networks.

It doesn't have to be big and profound.

It can be a cool trick.

Yeah, I mean, I'm thinking about the trick,

but like it's still amusing to me that it works at all.

That let's say that the extremely simple

algorithm stochastic gradient descent,

which is something that I would be able to derive

on the piece of paper to high school student

when put at the scale of thousands of machines

actually can create the behaviors,

which we call kind of human like behaviors.

So in general, any applications stochastic gradient descent

to neural networks is amazing to you.

So, or is there a particular application

in natural language, reinforcement learning?

And also, would you attribute that success to,

is it just scale?

What profound insight can we take from the fact

that the thing works for gigantic sets of variables?

I mean, the interesting thing is these algorithms,

they were invented decades ago

and people actually gave up on the idea.

And back then they thought that we need

profoundly different algorithms

and they spent a lot of cycles

on very different algorithms.

And I believe that we have seen that various innovations

that say like transformer or dropout

or so they can vastly help.

But it's also remarkable to me

that this algorithm from sixties or so,

or I mean, you can even say that the gradient descent

was invented by Leipniz in I guess 18th century or so.

That actually is the core of learning.

In the past people are, it's almost like a,

out of the maybe an ego,

people are saying that it cannot be the case

that such a simple algorithm is the,

could solve complicated problems.

So they were in search for the other algorithms.

And as I'm saying, like I believe that actually

we are in the game where there is,

there are actually frankly three levers.

There is compute, there are algorithms and there is data.

And if we want to build intelligent systems,

we have to pull all three levers

and they are actually multiplicative.

It's also interesting.

So you ask, is it only compute?

People internally, they did the studies

to determine how much gains they were coming

from different levers.

And so far we have seen that more gains came

from compute than algorithms.

But also we are in the world that in case of compute,

there is a kind of exponential increase in funding.

And at some point it's impossible to invest more.

It's impossible to invest 10 trillion dollars.

We are speaking about the, let's say all taxes in US.

But you're talking about money,

there could be innovation in the compute.

That's true as well.

So I mean, there are like a few pieces.

So one piece is human brain is an incredible supercomputer.

And they're like a,

it has a hundred trillion parameters.

Or like if you try to count the various quantities

in the brain, they're like a neuron, synapses.

They're small number of neurons.

There is a lot of synapses.

It's unclear even how to map synapses

to parameters of neural networks.

But it's clear that there are many more.

So it might be the case that our networks

are still somewhat small.

It also might be the case that they are more efficient

in brain or less efficient by some huge factor.

I also believe that there will be,

like at the moment we are at the stage

that these neural networks,

they require 1000X or like a huge factor

of more data than humans do.

And it will be a matter of,

there will be algorithms that vastly

decrease sample complexity, I believe so.

But the place where we are heading today is

there are domains which contains million X more data.

And even though computers might be 1000 times slower

than humans in learning, that's not the problem.

Like for instance, I believe that it should be possible

to create super human therapists by a,

and they're like even simple steps of doing it.

And the core reason is there is just,

machine will be able to read way more transcripts

of therapists and then it should be able to speak

simultaneously with many more people.

And it should be possible to optimize it all in parallel.

And then.

Well, there's now you're touching on something

I deeply care about and think is way harder than we imagine.

What's the goal of a therapist?

What's the goal of therapist?

So, okay, so one goal, now this is terrifying to me,

but there's a lot of people that contemplate suicide,

suffer from depression,

and they could significantly be helped with therapy.

And the idea that an AI algorithm might be in charge of that,

it's like a life and death task.

It's the stakes are high.

So one goal for a therapist, whether human or AI,

is to prevent suicide ideation, to prevent suicide.

How do you achieve that?

So, let's see.

So, to be clear, I don't think that the current models

are good enough for such a task

because it requires insane amount of understanding empathy

and the models are far from this place, but it's.

But do you think that understanding empathy,

that signal is in the data?

I think there is some signal in the data, yes.

I mean, there are plenty of transcripts of conversations

and it is possible from it to understand

personalities, it is possible from it to understand

if conversation is friendly, amicable, antagonistic.

It is, I believe that given the fact that the models

that we train now, they can have,

they are chameleons that they can have any personality.

They might turn out to be better in understanding

personality of other people than anyone else.

And they should. Be empathetic.

To be empathetic.

Yeah, interesting.

But I wonder if there's some level

of multiple modalities required

to be able to be empathetic of the human experience,

whether language is not enough, to understand death,

to understand fear, to understand childhood trauma,

to understand wit and humor required

when you're dancing with a person

who might be depressed or suffering,

both humor and hope and love and all those kinds of things.

So there's another underlying question

which is self supervised versus supervised.

So can you get that from the data

by just reading a huge number of transcripts?

I actually, so I think that reading

a huge number of transcripts is a step one.

It's like the same way as you cannot learn to dance

if just from YouTube.

By watching it, you have to actually try it out yourself.

So I think that here that's a similar situation.

I also wouldn't deploy the system

in the high stakes situations right away,

but kind of see gradually where it goes.

And obviously initially it would have to go hand in hand

with humans.

But at the moment we are in the situation

that actually there is many more people

who actually would like to have a therapy

or speak with someone than there are therapies out there.

Like I was so fundamentally I was thinking

what are the things that can vastly

increase people well being?

Therapy is one of them.

I think meditation is other one.

I guess maybe human connection is a third one.

And I guess pharmacologically it's also possible.

Maybe direct brain stimulation or something like that.

But these are pretty much options out there.

Then let's say the way I'm thinking about the AGI endeavor

is by default that's an endeavor to increase amount of wealth.

And I believe that we can vastly increase

amount of wealth for everyone.

And simultaneously, so I mean,

these are like two endeavors that make sense to me.

One is like essentially increase amount of wealth.

And second one is increase overall human well being.

And those are coupled together.

And they can, I would say these are different topics.

One can help another.

And you know, therapist is a funny word

because I see friendship and love as therapy.

I mean, so therapist broadly defined

as just friendship as a friend.

So like therapist has a very kind of clinical sense to it.

But what is human connection?

You're like, not to get all Camus and Dostoevsky on you,

but you know, life is suffering and we draw,

we seek connection with other humans

as we desperately try to make sense of this world

in a deep overwhelming loneliness that we feel inside.

So I think connection has to do with understanding.

And I think that almost like a lack of understanding

causes suffering if you speak with someone

and do you feel ignored that actually causes pain

if you are feeling deeply understood that actually

they might not even tell you what to do in life,

but like a pure understanding.

Or just being heard.

Understanding is a kind of, that's a lot, you know,

just being heard, feel like you're being heard.

Like somehow that's a alleviation temporarily

of the loneliness that if somebody knows you're here

with their body language, with the way they are,

with the way they look at you, with the way they talk,

you feel less alone for a brief moment.

Yeah, very much agree.

So I thought in the past about somewhat similar question

to yours, which is what is love?

Rather what is connection?

Yes.

And obviously I think about these things

from AI perspective, what would it mean?

So I said that the, you know,

intelligence has to do with some compression,

which is more or less like you can say

almost understanding of what is going around.

It seems to me that other aspect

is there seem to be reward functions.

And you can have, you know, reward for food,

for maybe human connection, for let's say warmth, sex,

and so on.

And it turns out that the various people

might be optimizing slightly different reward functions.

They essentially might care about different things.

And in case of love, at least the love between two people,

you can say that the, you know,

boundary between people dissolves to such extent

that they end up optimizing each other reward functions.

Yeah.

Oh, that's interesting.

Celebrate the success of each other.

Yeah.

Or in some sense, I would say love means helping others

to optimize their reward functions,

not your reward functions,

not the things that you think are important,

but the things that the person cares about,

you try to help them to optimize it.

So love is, if you think of two reward functions,

you just, it's a condition.

Yeah.

You combine them together.

Yeah, pretty much.

Maybe like with a weight,

and it depends like the dynamic of the relationship.

Yeah, I mean, you could imagine that if you are fully

optimizing someone's reward function without yours,

then maybe you are creating codependency

or something like that.

Yeah.

I'm not sure what's the appropriate weight.

But the interesting thing is,

I even think that the individual person,

we ourselves, we are actually less of a unified insight.

So for instance, if you look at the donut,

on the one level, you might think,

oh, this is like a look space that I would like to eat it.

On another level, you might tell yourself,

I shouldn't be doing it because I want to gain muscles.

So, and you know, you might do it regardless,

kind of against yourself.

So it seems that even within ourselves,

they're almost like a kind of intertwined personas.

And I believe that the self love

means that the love between all these personas,

which also means being able to love yourself

when we are angry or stressed or so.

Combining all those reward functions

of the different selves you have.

Yeah, and accepting that they are there.

Like, you know, often people,

they have a negative self talk,

or they say, I don't like when I'm angry.

And like I try to imagine,

if there would be like a small baby Lex,

like a five years old, angry,

and then you're like, you shouldn't be angry.

Like, stop being angry.

But like, instead, actually you want Lex to come over,

give him a hug, and this, like I say, it's fine.

Okay, you can be angry as long as you want.

And then he would stop, or maybe not.

Or maybe not, but you cannot expect it even.

Yeah, but still, that doesn't explain it.

Still, that doesn't explain the why of love.

Like, why is love part of the human condition?

Why is it useful to combine the reward functions?

It seems like that doesn't, I mean,

I don't think reinforcement learning frameworks

can give us answers to why.

Even the Hutter framework has an objective function

that's static.

So we came to existence as a consequence

of evolutionary process.

And in some sense, the purpose of evolution is survival.

And then this complicated optimization objective

baked into us, let's say compression,

which might help us operate in the reward,

and it baked into us various reward functions.

Yeah.

Then to be clear, at the moment,

we are operating in the regime,

which is somewhat out of distribution

where the even evolution optimized us.

It's almost like love is a consequence

of cooperation that we've discovered is useful.

Correct. In some way, it's even the case if you...

I just love the idea that love is out of distribution.

Or it's not out of distribution.

It's like, as you said, it evolved for cooperation.

Yes.

And I believe that in some sense,

cooperation ends up helping each of us individually.

So it makes sense evolutionary.

And there is, in some sense, and love means

there is this dissolution of boundaries

that you have shared the reward function.

And we evolved to actually identify ourselves

with larger groups.

So we can identify ourselves with a family.

We can identify ourselves with a country

to such extent that people are willing

to give away their life for country.

So we are wired, actually, even for love.

And at the moment, I guess, maybe it

would be somewhat more beneficial if we would identify

ourselves with all the humanity as a whole.

So you can clearly see, when people travel around the world,

when they run into a person from the same country,

they say, oh, which city you are.

And all of a sudden, they find all these similarities.

They befriend those folks earlier than others.

So there is some sense of the belonging.

And I would say, I think it would be overall good thing

to the world for people to move towards,

I think it's even called open individualism,

move toward the mindset of a larger and larger groups.

So the challenge there, that's a beautiful vision,

and I share it, to expand that circle of empathy,

that circle of love towards the entirety of humanity.

But then you start to ask, well, where do you draw the line?

Because why not expand it to other conscious beings?

And then finally, for our discussion,

something I think about is why not expand it to AI systems?

Like, we start respecting each other

when the entity on the other side has the capacity to suffer,

because then we develop a capacity to empathize.

And so I could see AI systems that

are interacting with humans more and more

having conscious like displays.

So they display consciousness through language

and through other means.

And so then the question is, well, is that consciousness?

Because they're acting conscious.

And so the reason we don't like torturing animals

is because they look like they're suffering

when they're tortured.

And if AI looks like it's suffering when it's tortured,

how is that not requiring of the same kind of empathy from us

and respect and rights that animals do and other humans do?

I think it requires empathy as well.

I mean, I would like, I guess, us or humanity

or so make a progressing understanding what consciousness

is, because I don't want just to be speaking

about the philosophy, but rather actually make a scientific.

There was a time that people thought

that there is a force of life and the things that

have this force, they are alive.

And I think that there is actually

a path to understand exactly what consciousness is.

And in some sense, it might require essentially putting

probes inside of a human brain.

What Neuralink does.

So the goal there, I mean, there's

several things with consciousness that

make it a real discipline, which is one,

is rigorous measurement of consciousness.

And then the other is the engineering

of consciousness, which may or may not be related.

I mean, you could also run into trouble,

like, for example, in the United States,

the Department of DOT, Department of Transportation,

and a lot of different places put a value on human life.

I think DOT's value is $9 million per person.

So in that same way, you can get into trouble

if you put a number on how conscious a being is,

because then you can start making policy.

If a cow is 0.1 or like 10% as conscious as a human,

then you can start making calculations

and it might get you into trouble.

But then again, that might be a very good way to do it.

I would like to move to that place that actually we

have scientific understanding what consciousness is.

And then we'll be able to actually assign value.

And I believe that there is even the path

for the experimentation in it.

So we said that you could put the probes inside of the brain.

There is actually a few other things

that you could do with devices like Neuralink.

So you could imagine that the way even

to measure if AI system is conscious

is by literally just plugging into the brain.

I mean, that assumes that it's kind of easy.

But plugging into the brain and asking

a person if they feel that their consciousness expanded.

This direction, of course, has some issues.

You can say, if someone takes a psychedelic drug,

they might feel that their consciousness expanded,

even though that drug itself is not conscious.

Right.

So you can't fully trust the self report of a person

saying their consciousness is expanded or not.

Let me ask you a little bit about psychedelics.

There have been a lot of excellent research

on different psychedelics, psilocybin, MDMA, even DMT,

drugs in general, marijuana too.

What do you think psychedelics do to the human mind?

It seems they take the human mind to some interesting places.

Is that just a little hack, a visual hack,

or is there some profound expansion of the mind?

So let's see.

I don't believe in magic.

I believe in science, in causality.

Still, let's say.

And then as I said, I think that our subjective experience

of reality is we live in the simulation run by our brain.

And the simulation that our brain runs,

they can be very pleasant or very hellish.

Drugs, they are changing some hyperparameters

of the simulation.

It is possible thanks to change of these hyperparameters

to actually look back on your experience

and even see that the given things that we took for granted,

they are changeable.

So they allow to have an amazing perspective.

There is also, for instance, the fact

that after DMT people can see the full movie inside

of their head, gives me further belief

that the brain can generate the full movie,

that the brain is actually learning

the model of reality to such extent

that it tries to predict what's going to happen next.

Yeah, very high resolution.

So it can replay realities.

Extremely high resolution.

Yeah, and it's also kind of interesting to me

that somehow there seems to be some similarity

between these drugs and meditation itself.

And I actually started even these days

to think about meditation as a psychedelic.

Do you practice meditation?

I practice meditation.

I mean, I went a few times on the retreats

and it feels like after second or third day of meditation,

there is almost like a sense of tripping.

What is a meditation retreat entail?

So you wake up early in the morning

and you meditate for an extended period of time.

Alone?

Just to say, Jake's been trying.

Yeah, so it's optimized, even though there are other people,

it's optimized for isolation.

So you don't speak with anyone.

You don't actually look into other people's eyes.

And you sit on the chair.

So vipassana meditation tells you to focus on the breath.

So you try to put all the attention

into breathing in and breathing out.

And the crazy thing is that as you focus attention like that,

after some time, there starts coming back

like some memories that you completely forgotten.

It almost feels like you have a mailbox

and then you are just archiving email one by one.

And at some point, there is like an amazing feeling

of getting to mailbox zero, zero emails.

And it's very pleasant.

It's kind of, it's crazy to me that once you resolve

these inner stories or inner traumas,

then once there is nothing left,

the default state of human mind is extremely peaceful and happy.

Like some sense, it feels that the,

it feels at least to me in the way how when I was a child

that I can look at any object and it's very beautiful.

I have a lot of curiosity about the simple things.

And that's where the usual in meditation takes me.

Are you, what are you experiencing?

Are you just taking in simple sensory information

and are just enjoying the rawness of that sensory information?

So there's no, there's no memories or all that kind of stuff.

You're just enjoying being.

Yeah, pretty much.

I mean, still there is a, that it's,

it's thoughts are slowing down sometimes they pop up,

but it's also somehow the extended meditation

takes you to the space that they are way more friendly.

You're way more positive.

There is also this, this thing that we've extended.

It almost feels that the,

it almost feels that we are constantly getting a little bit of a reward function

and we are just spreading this reward function on various activities.

But if you stay still for extended period of time,

it kind of accumulates, accumulates, accumulates.

And there is a, there is a sense,

there is a sense that some point it passes some threshold

and it feels as drop is falling into kind of ocean of love and bliss.

And that's like a, this is like a very pleasant.

And that's what I'm saying, like a,

that corresponds to the subjective experience.

Some people, I guess, in spiritual community,

they describe it that that's the reality.

And I would say I believe that they're like all sorts of subjective experience

that one can have.

And I believe that, for instance, meditation might take you to the subjective experiences,

which are very pleasant, collaborative.

And I would like a word to move toward a more collaborative place.

Yeah, I would say that's very pleasant and I enjoy doing stuff like that.

I wonder how that maps to your mathematical model of love

with the reward function combining a bunch of things.

It seems like our life then is we're just,

we have this reward function and we're accumulating a bunch of stuff in it with weights.

It's like a, like multi objective.

And what meditation is, is you just remove them, remove them

until the weight on one or just a few is very high.

And that's where the pleasure comes from.

Yeah, so something similar to how I'm thinking about it.

So I told you that there is this like, there is a story of who you are.

And I think almost about it as a, you know, text prepended to GPT.

Yeah.

And some people refer to it as ego.

Okay, it's like a story who, who, who you are.

Okay.

So ego is the prompt for GPT three or GPT.

Yes, yes.

And that's the description of you.

And then with meditation, you can get to the point that actually you

experience things without the prompt.

And you experience things like as they are, you are not biased

over the description, how they supposed to be.

That's very pleasant.

And then we've respected the reward function.

It's possible to get to the point that there is the solution of self.

And therefore you can say that you're, you're having a, you're,

or like a your brain attempts to simulate the reward function of everyone else or

like everything that's, that there is this like a love which feels like a

oneness with everything.

And that's also, you know, very beautiful, very pleasant.

At some point you might have a lot of altruistic thoughts during that

moment and then the self always comes back.

How would you recommend if somebody is interested in meditation, like a big

thing to take on as a project?

Would you recommend a meditation retreat?

How many days, what kind of thing would you recommend?

I think that actually retreat is the way to go.

It almost feels that, as I said, like a meditation is a psychedelic,

but when you take it in the small dose, you might barely feel it.

Once you get the high dose, actually you're going to feel it.

So even cold turkey, if you haven't really seriously meditated for a long

period of time, just go to retreat.

Yeah, how many days?

How many days? Start weekend one.

Weekend, so like two, three days.

And it's like, it's interesting that first or second day, it's hard and

at some point it becomes easy.

There's a lot of seconds in a day.

How hard is the meditation retreat, just sitting there in a chair?

So the thing is actually, it literally just depends on your, on your own framing.

Like if you are in the mindset that you are waiting for it to be over or

you are waiting for Nirvana to happen, it will be very unpleasant.

And in some sense, even the difficulty, it's not even in the lack

of being able to speak with others.

Like you're sitting there, your legs will hurt from sitting.

In terms of like the practical things, do you experience kind of discomfort,

like physical discomfort of just sitting, like your, your butt being numb,

your legs being sore, all that kind of stuff?

Yes, you experience it and then the, the, they teach you to observe it rather.

And it's like a, the crazy thing is, you at first might have a feeling

toward trying to escape it.

And that becomes very apparent that that's extremely unpleasant.

And then you just, just observe it.

And at some point it, it just becomes, it just is.

It's like a, I remember that we've, Ilya told me some time ago that, you know,

he takes a cold shower and his mindset of taking a cold, cold shower was

to embrace suffering.

Yeah, excellent.

I do the same.

This is your style?

Yeah, it's my style.

I like this.

So my style is actually, I also sometimes take cold showers.

It is purely observing how the water goes through my body, like a purely

being present, not trying to escape from there.

Yeah.

And I would say then it actually becomes pleasant.

It's not like, well, that, that's interesting.

I, I'm also, that means that's, that's the way to deal with anything really

difficult, especially in the physical space is to observe it.

To say it's pleasant.

Hmm.

It's a, I would use a different word.

You're, you're accepting of the full beauty of reality, I would say,

because it's a pleasant, but yeah, in some sense, it is pleasant.

That's the only way to deal with a cold shower is to become an observer

and to find joy in it.

Same with like really difficult physical exercise or like running for a

really long time, endurance events, just anytime you're exhausted, any kind of

pain, I think the only way to survive it is not to resist it.

It's to observe it.

You mentioned Ilya, Ilya Satskever.

It's very, he's our chief scientist, but also he's very close friend of mine.

You co founded Open AI with you.

I've spoken with him a few times.

He's brilliant.

I really enjoy talking to him.

His mind, just like yours, works in fascinating ways.

Both of you are not able to define deep learning simply.

What's it like having him as somebody you have technical discussions with

on in space and machine learning, deep learning AI, but also life?

What's it like when these two agents get into a self play situation in a room?

What's it like collaborating with him?

So I believe that we have extreme respect to each other.

So I love Ilya's insight, both like I guess about consciousness, life, AI.

But in terms of the, it's interesting to me because you're a brilliant thinker in

the space and machine learning, like intuition, like digging deep in what works, what doesn't,

why it works, why it doesn't.

And so is Ilya.

I'm wondering if there's interesting deep discussions you've had with him in the past

or disagreements that were very productive.

So I can say I also understood over the time where are my strengths.

So obviously we have plenty of discussions.

And I myself have plenty of ideas, but I consider Ilya one of the most prolific AI scientists

in the entire world.

And I think that I realize that maybe my super skill is being able to bring people to collaborate

together, that I have some level of empathy that is unique in AI world.

And that might come from either meditation, psychedelics, or let's say I read just hundreds

of books on this topic.

And I also went through a journey of, I develop all sorts of algorithms.

So I think that maybe I can, that's my super human skill.

Ilya is one of the best AI scientists, but then I'm pretty good in assembling teams.

And I'm also not holding to people like I'm growing people.

And then people become managers at OpenAI, that's room any of them like a research managers.

So you find places where you're excellent and he finds like his deep scientific insights

is where he is.

And you find ways you can, the puzzle pieces fit together.

Correct.

Like, you know, ultimately, for instance, let's say Ilya, he doesn't manage people.

That's not what he likes or so.

I like hanging out with people.

By default, I'm an extrovert and I care about people.

Interesting.

Okay.

Okay.

Cool.

So that fits perfectly together.

But I mean, I also just like your intuition about various problems in machine learning.

He's definitely one I really enjoy.

I remember talking to him about something I was struggling with, which is coming up with

a good model for pedestrians for human beings that cross the street in the context of autonomous

vehicles.

And he immediately started to like formulate a framework within which you can evolve a

model for pedestrians, like through self play, all that kind of mechanisms.

The depth of thought on a particular problem, especially problems he doesn't know anything

about is fascinating to watch.

And it makes you realize like, yeah, the limits that the human intellect might be limitless.

Or it's just impressive to see a descendant of ape come up with clever ideas.

Yeah.

I mean, so even in the space of deep learning, when you look at various people, there are

people now who invented some breakthroughs once, but there are very few people who did

it multiple times.

And you can think if someone invented it once, that might be just a sure luck.

And if someone invented it multiple times, you know, if the probability of inventing

it once is one over a million, then probability of inventing it twice or three times would

be one over a million square or to the power of three, which would be just impossible.

So it literally means that it's given that it's not the luck.

And Ilya is one of these few people who have a lot of these inventions in his arsenal.

It also feels that, you know, for instance, if you think about folks like Gauss or Euler,

you know, at first they read a lot of books, and then they did thinking, and then they

figure out math.

And that's how it feels with Ilya.

You know, at first he read stuff, and then he spent his thinking cycles.

And that's a really good way to put it.

When I talk to him, I see thinking.

He's actually thinking.

Like he makes me realize that there's like deep thinking that the human mind can do.

Like most of us are not thinking deeply.

Like you really have to put in a lot of effort to think deeply.

Like I have to really put myself in a place where I think deeply about a problem.

It takes a lot of effort.

It's like an airplane taking off or something.

You have to achieve deep focus.

He's just, his brain is like a vertical takeoff in terms of airplane analogy.

So it's interesting.

But I mean, Cal Newport talks about this, his ideas of deep work.

It's, you know, most of us don't work much at all in terms of like deeply think about

particular problems, whether it's in math, engineering, all that kind of stuff.

You want to go to that place often.

And that's real hard work.

And some of us are better than others at that.

So I think that the big piece has to do with actually even engineering our environment

that such that it's conducive to that.

So see both Ilya and I on the frequent basis, we kind of disconnect ourselves from the world

in order to be able to do extensive amount of thinking.

So Ilya usually, he just leaves iPad at hand.

He loves his iPad.

And for me, I'm even sometimes, you know, just going for a few days to different location

to Airbnb.

So I'm turning off my phone and there is no access to me.

And that's extremely important for me to be able to actually just formulate new thoughts

to do deep work rather than to be reactive.

And the older I am, the more of this like a random tasks are at hand.

Before I go on to that thread, let me return to our friend GPT.

Let me ask you another ridiculously big question.

Can you give an overview of what GPT three is?

Or like you say in your Twitter bio, GPT and plus one.

How it works and why it works.

So GPT three is a humongous neural network.

Let's assume that we know what is neural network by definition.

And it is trained on the entire internet just to predict next word.

So let's say it sees part of the article and the only task that it has at hand, it is to

say what would be the next word?

What would be the next word?

And it becomes really exceptional at the task of figuring out what's the next word.

So you might ask, why would this be an important task?

Why would it be important to predict what's the next word?

And it turns out that a lot of problems can be formulated as a text completion problem.

So GPT is purely learning to complete the text.

As you could imagine, for instance, if you are asking a question, who is president of

United States, then GPT can give you an answer to it.

It turns out that many more things can be formulated this way.

You can format text in the way that you have sent us in English.

You make it even look like some content of a website elsewhere, which would be teaching

people how to translate things between languages.

So it would be EN colon, text in English, FR colon, and then you ask model to continue.

And it turns out that such a model is predicting translation from English to French.

The crazy thing is that this model can be used for way more sophisticated tasks.

So you can format text such that it looks like a conversation between two people.

And that might be a conversation between you and Elon Musk.

And because the model read all the text about Elon Musk, it will be able to predict Elon

Musk words as it would be Elon Musk.

It will speak about colonization of Mars, about sustainable future and so on.

And it's also possible to even give arbitrary personality to the model.

You can say, here is a conversation that we've a friendly AI bot.

And the model will complete the text as a friendly AI bot.

So I mean, how do I express how amazing this is?

So just to clarify a conversation, generating a conversation between me and Elon Musk, it

wouldn't just generate good examples of what Elon would say.

It would get the syntax all correct.

So like interview style, it would say like Elon Cohen and Lex Cohen.

It's not just like inklings of semantic correctness.

It's like the whole thing, grammatical, syntactic, semantic.

It's just really, really impressive generalization.

Yeah, I mean, I also want to provide some caveat so it can generate few paragraphs of

coherent text.

But as you go to longer pieces, it actually goes off the rails.

If you would try to write a book, it won't work out this way.

What way does it go off the rails, by the way?

Is there interesting ways in which it goes off the rails?

What falls apart first?

So the model is trained on all the existing data that is out there, which means that it

is not trained on its own mistakes.

So for instance, if it would make a mistake, then I kept so to give you an example.

So let's say I have a conversation with a model pretending that is Elon Musk, and then

I start putting some, I'm start actually making up things which are not factual.

Sounds like Twitter.

But I got you, sorry.

Yeah.

Okay.

I don't know.

I would say that Elon is my wife, and the model will just keep on carrying it on.

As if it's true.

Yes.

And in some sense, if you would have a normal conversation with Elon, he would be, what

the fuck?

Yeah, there would be some feedback.

So the model is trained on things that humans have written, but through the generation process,

there's no human in the loop feedback.

Correct.

That's fascinating.

Makes sense.

So it's magnified.

It's like the errors get magnified and magnified.

And it's also interesting, I mean, first of all, humans have the same problem.

It's just that we will make fewer errors and magnify the errors slower.

I think that actually what happens with humans is if you have a wrong belief about the world

as a kid, then very quickly you will learn that it's not correct because they are grounded

in reality and they are learning from your new experience.

But do you think the model can correct itself too?

Won't it, through the power of the representation, and so the absence of Elon Musk being your

wife, information on the internet, won't it correct itself?

There won't be examples like that.

So the errors will be subtle at first.

And in some sense, you can also say that the data that is not out there is the data which

would represent how the human learns.

And maybe the model would be trained on such a data, then it would be better off.

How intelligent is GPT3, do you think?

When you think about the nature of intelligence, it seems exceptionally impressive.

But then if you think about the big AGI problem, is this footsteps along the way to AGI?

So it seems that intelligence itself is a multiple axis of it.

And I would expect that the systems that we are building, they may end up being superhuman

on some axis and subhuman on some other axis.

It would be surprising to me on all axis simultaneously, they would become superhuman.

Of course, people ask this question, is GPT a spaceship that would take us to Moon?

Or are we building a ladder to Heaven that we are just building bigger and bigger ladder?

And we don't know in some sense which one of these two works better.

Which one is better?

I like Stairway to Heaven, it's a good song, so I'm not exactly sure which one is better.

But you're saying the spaceship to the Moon is actually effective?

Correct.

And people who criticize GPT, they say, you are just building a ladder and it will never

reach the Moon.

And at the moment, I would say the way I'm thinking is like a scientific question.

And I'm also in heart, I'm a builder creator.

And I'm thinking, let's try out, let's see how far it goes.

And so far, we see constantly that there is a progress.

So do you think GPT 4, GPT 5, GPT N plus 1, there will be a phase shift, like a transition

to a place where we'll be truly surprised.

And again, GPT 3 is already very truly surprising.

The people that criticize GPT 3 as a Stairway, as a ladder to Heaven, I think two people

quickly get accustomed to how impressive it is that the prediction of the next word can

achieve such depth of semantics, accuracy of syntax, grammar, and semantics.

Do you think GPT 4 and 5 and 6 will continue to surprise us?

I mean, definitely, there will be more impressive models.

There is a question, of course, if there will be a phase shift.

And also even the way I'm thinking about these models is that when we build these models,

we see some level of the capabilities, but we don't even fully understand everything

that the model can do.

And actually one of the best things to do is to allow other people to probe the model

to even see what is possible.

Since using GPT as an API and opening it up to the world.

Yeah.

I mean, so when I'm thinking from perspective of obviously various people that have concerns

about AGI, including myself, and then when I'm thinking from perspective what's the strategy

even to deploy these things to the world, then the one strategy that I have seen many

times working is the iterative deployment that you deploy slightly better versions

and you allow other people to criticize you.

So you actually or try it out, you see where are their fundamental issues.

And it's almost you don't want to be in that situation that you are holding into powerful

system and there's like a huge overhang, then you deploy it and it might have a random chaotic

impact on the world.

So you actually want to be in the situation that you are gradually deploying systems.

I asked this question of Ilya, let me ask you this question.

I've been reading a lot about Stalin and power.

If you're in possession of a system that's like AGI, that's exceptionally powerful.

Do you think your character and integrity might become corrupted?

Like famously power corrupts and absolutely power corrupts, absolutely.

So I believe that you want at some point to work toward distributing the power.

I think that you want to be in the situation that actually AGI is not controlled by a small

number of people, but essentially by a larger collective.

So the thing is that requires a George Washington style move in the ascent to power.

There's always a moment when somebody gets a lot of power and they have to have the integrity

and the moral compass to give away that power.

That humans have been good and bad throughout history at this particular step.

And I wonder, I wonder we like blind ourselves in, for example, between nations, a race towards

the AI race between nations, we might blind ourselves and justify to ourselves the development

of AI without distributing the power because we want to defend ourselves against China,

against Russia, that kind of logic.

And I wonder how we design governance mechanisms that prevent us from becoming power hungry

and in the process destroying ourselves.

So let's see.

I have been thinking about this topic quite a bit, but I also want to admit that once

again I actually want to rely way more on some outman, on a hero and excellent block

on how even to distribute wealth.

And he proposed in his blog to tax equity of the companies rather than profit and to

distribute it.

And this is an example of Washington move.

I guess I personally have insane trust in some.

He already spent plenty of money running universal basic income project that gives me, I guess,

maybe some level of trust to him, but I also, I guess, love him as a friend.

Yeah.

I wonder, because we're sort of summoning a new set of technologies, I wonder if we'll

be cognizant, like you're describing the process of open AI, but it could also be at other

places like in the US government, right?

Both China and the US are now full steam ahead on autonomous weapons systems development.

And that's really worrying to me because in the framework of something being a national

security danger or a military danger, you can do a lot of pretty dark things that blind

our moral compass.

And I think AI will be one of those things.

In some sense, the mission and the work you're doing at open AI is like the counterbalance

to that.