back to indexNick Bostrom: Simulation and Superintelligence | Lex Fridman Podcast #83
link |
The following is a conversation with Nick Bostrom, a philosopher at University of Oxford
link |
and the director of the Future of Humanity Institute. He has worked on fascinating and
link |
important ideas in existential risk, simulation hypothesis, human enhancement ethics,
link |
and the risks of superintelligent AI systems, including in his book, Superintelligence.
link |
I can see talking to Nick multiple times in this podcast, many hours each time,
link |
because he has done some incredible work in artificial intelligence, in technology space,
link |
science, and really philosophy in general. But we'll have to start somewhere.
link |
This conversation was recorded before the outbreak of the coronavirus pandemic,
link |
that both Nick and I, I'm sure, will have a lot to say about next time we speak.
link |
And perhaps that is for the best, because the deepest lessons can be learned only in retrospect
link |
when the storm has passed. I do recommend you read many of his papers on the topic of existential
link |
risk, including the technical report titled Global Catastrophic Risks Survey that he coauthored with
link |
Anders Sandberg. For everyone feeling the medical, psychological, and financial burden of this crisis,
link |
I'm sending love your way. Stay strong. We're in this together. We'll beat this thing.
link |
This is the Artificial Intelligence Podcast. If you enjoy it, subscribe on YouTube,
link |
review it with five stars on Apple Podcast, support on Patreon, or simply connect with
link |
me on Twitter at Lex Freedman, spelled F R I D M A N. As usual, I'll do one or two minutes of ads now
link |
and never any ads in the middle that can break the flow of the conversation. I hope that works
link |
for you. It doesn't hurt the listening experience. This show is presented by Cash App, the number
link |
one finance app in the App Store. When you get it, use code LexPodcast. Cash App lets you send money
link |
to friends by Bitcoin and invest in the stock market with as little as $1. Since Cash App does
link |
fractional share trading, let me mention that the order execution algorithm that works behind the
link |
scenes to create the abstraction of fractional orders is an algorithmic marvel. So big props to
link |
the Cash App engineers for solving a hard problem that in the end provides an easy interface that
link |
takes a step up to the next layer of abstraction over the stock market, making trading more accessible
link |
for new investors and diversification much easier. So again, if you get Cash App from the App Store,
link |
Google Play, and use the code LexPodcast, you get $10 and Cash App will also donate $10 to first,
link |
an organization that is helping to advance robotics and STEM education for young people around the
link |
world. And now here's my conversation with Nick Bostrom. At the risk of asking the Beatles to
link |
play yesterday or the Rolling Stones to play satisfaction, let me ask you the basics. What
link |
is the simulation hypothesis? That we are living in a computer simulation. What is a computer
link |
simulation? How are we supposed to even think about that? Well, so the hypothesis is meant to be
link |
understood in a literal sense, not that we can kind of metaphorically view the universe as an
link |
information processing physical system, but that there is some advanced civilization who built a
link |
lot of computers and that what we experience is an effect of what's going on inside one of those
link |
computers so that the world around us, our own brains, everything we see and perceive and think
link |
and feel would exist because this computer is running certain programs. So do you think of this
link |
computer as something similar to the computers of today, these deterministic sort of touring machine
link |
type things? Is that what we're supposed to imagine or we're supposed to think of something more
link |
like a quantum mechanical system, something much bigger, something much more complicated,
link |
something much more mysterious from our current perspective? The ones we have today would do
link |
find them bigger, certainly. You'd need more memory and more processing power. I don't think
link |
anything else would be required. Now, it might well be that they do have additional, maybe they
link |
have quantum computers and other things that would give them even more umph. It seems kind of
link |
plausible, but I don't think it's a necessary assumption in order to get to the conclusion
link |
that a technologically mature civilization would be able to create these kinds of computer
link |
simulations with conscious beings inside them. So do you think the simulation hypothesis is an idea
link |
that's most useful in philosophy, computer science, physics, sort of where do you see it
link |
having valuable kind of starting point in terms of the thought experiment of it?
link |
Is it useful? I guess it's more informative and interesting and maybe important,
link |
but it's not designed to be useful for something else.
link |
Okay, interesting, sure. But is it philosophically interesting or is there some kind of implications
link |
of computer science and physics? I think not so much for computer science or physics per se.
link |
Certainly it would be of interest in philosophy, I think also to say cosmology or physics in as
link |
much as you're interested in the fundamental building blocks of the world and the rules that
link |
govern it. If we are in a simulation, there is then the possibility that say physics at the
link |
level where the computer running the simulation could be different from the physics governing
link |
phenomena in the simulation. So I think might be interesting from point of view of religion or
link |
just for trying to figure out what the heck is going on. So we mentioned the simulation hypothesis
link |
so far. There is also the simulation argument, which I tend to make a distinction. So simulation
link |
hypothesis, we are living in a computer simulation argument, this argument that
link |
tries to show that one of three propositions is true, one of which is the simulation hypothesis,
link |
but there are two alternatives in the original simulation argument, which we can get to.
link |
Yeah, let's go there. By the way, confusing terms because people will, I think, probably naturally
link |
think simulation argument equals simulation hypothesis, just terminology wise. But let's go
link |
there. So simulation hypothesis means that we are living in a simulation. The hypothesis that we're
link |
living in a simulation, simulation argument has these three complete possibilities that cover
link |
all possibilities. So what are they? Yeah, so it's like a disjunction. It says at least one of these
link |
three is true, although it doesn't on its own tell us which one. So the first one is that almost all
link |
civilizations at their current stage of technological development
link |
go extinct before they reach technological maturity. So there is some great filter
link |
that makes it so that basically none of the civilizations throughout, you know,
link |
maybe vast cosmos will ever get to realize the full potential of technological development.
link |
And this could be theoretically speaking, this could be because most civilizations kill themselves
link |
too eagerly or destroy themselves too eagerly, or it might be super difficult to build a simulation.
link |
So the span of time. Theoretically, it could be both. Now, I think it looks like we would
link |
technologically be able to get there in a time span that is short compared to, say, the lifetime of
link |
planets and other sort of astronomical processes. So your intuition is the build simulation is not.
link |
Well, so this is interesting concept of technological maturity. It's kind of an
link |
interesting concept to have other purposes as well. We can see, even based on our current
link |
limited understanding, what some lower bound would be on the capabilities that you could
link |
realize by just developing technologies that we already see are possible. So for example,
link |
one of my research fellows here, Eric Drexler back in the 80s, studied molecular manufacturing.
link |
That is, you could analyze using theoretical tools and computer modeling the performance
link |
of various molecularly precise structures that we didn't then and still don't today have the
link |
ability to actually fabricate. But you could say that, well, if we could put these atoms together
link |
in this way, then the system would be stable and it would rotate at this speed and have these
link |
computational characteristics. And he also outlined some pathways that would enable us to get to
link |
this kind of molecularly manufacturing in the fullness of time. And you could do other studies
link |
we've done. You could look at the speed at which say it would be possible to colonize the galaxy
link |
if you had mature technology. We have an upper limit, which is the speed of light. We have
link |
sort of a lower current limit, which is how fast current rockets go. We know we can go faster than
link |
that by just making them bigger and have more fuel and stuff. And you can then start to describe
link |
the technological affordances that would exist once a civilization has had enough time to develop.
link |
Even at least those technologies we already know are possible. Then maybe they would discover other
link |
new physical phenomena as well that we haven't realized that would enable them to do even more.
link |
But at least there is this kind of basic set of capabilities.
link |
Can you just link on that? How do we jump from molecular manufacturing to deep space exploration
link |
to mature technology? What's the connection? These would be two examples of technological
link |
capability sets that we can have a high degree of confidence or physically possible in our
link |
universe under that civilization that was allowed to continue to develop its science and technology
link |
would eventually attain. We can kind of see the set of breakthroughs that are likely to
link |
happen. So you can see, what did you call it, the technological set?
link |
With computers, maybe it's easier stuff. The one is we could just imagine bigger computers
link |
using exactly the same parts that we have. So you can kind of scale things that way, right?
link |
But you could also make processors a bit faster if you had this molecular nanotechnology that
link |
Director Exeter described. He characterized a kind of crude computer built with these parts
link |
that would perform at a million times the human brain while being significantly smaller, the
link |
size of a sugar cube. And he may not claim that that's the optimum computing structure.
link |
You could build a faster computer that would be more efficient, but at least you could do that
link |
if you had the ability to do things that were atomically precise. So you can then combine these
link |
two. You could have this kind of nanomolecular ability to build things atom by atom and then
link |
say at this as a spatial scale that would be attainable through space colonizing technology.
link |
You could then start, for example, to characterize a lower bound on the amount of computing power
link |
that a technologically mature civilization would have if it could grab resources, you know,
link |
planets and so forth and then use this molecular nanotechnology to optimize them for computing.
link |
You'd get a very, very high lower bound on the amount of compute.
link |
So, sorry, I just need to define some terms. So technologically mature civilization is one
link |
that took that piece of technology to its lower bound. What is it technologically mature civilization?
link |
Well, okay. So that means it's a stronger concept than we really need for the simulation hypothesis.
link |
I just think it's interesting in its own right. So it would be the idea that there is
link |
some stage of technological development where you've basically maxed out that you developed all
link |
those general purpose widely useful technologies that could be developed or at least kind of come
link |
very close to the, you know, 99.9% there or something. So that's an independent question.
link |
You can think either that there is such a ceiling or you might think it just goes,
link |
the technology tree just goes on forever. Where does your sense fall?
link |
I would guess that there is a maximum that you would start to asymptote towards.
link |
So new things won't keep springing up. New ceilings.
link |
In terms of basic technological capabilities, I think there is like a finite set of those
link |
that can exist in this universe. Moreover, I mean, I wouldn't be that surprised if we actually
link |
reached close to that level fairly shortly after we have, say, machine superintelligence.
link |
So I don't think it would take millions of years for a human originating civilization
link |
to begin to do this. I think it's like more likely to happen on historical timescales.
link |
But that's an independent speculation from the simulation argument.
link |
I mean, for the purpose of the simulation argument, it doesn't really matter whether it
link |
goes indefinitely far up or whether there is a ceiling, as long as we know we can at least
link |
get to a certain level. And it also doesn't matter whether that's going to happen in 100 years
link |
or 5,000 years or 50 million years. Like the timescales really don't make any difference
link |
for the simulation. Can you look on that a little bit? There's a big difference between 100 years
link |
and 10 million years. So it doesn't really not matter. Because you just said,
link |
does it matter if we jump scales to beyond historical scales? So we described that. So
link |
for the simulation argument, doesn't it matter that if it takes 10 million years,
link |
it gives us a lot more opportunity to destroy civilization in the meantime?
link |
Yeah. Well, so it would shift around the probabilities between these three alternatives.
link |
Right. That is, if we are very, very far away from being able to create these simulations,
link |
if it's like say the billions of years into the future, then it's more likely that we will fail
link |
ever to get there. There's more time for us to kind of, you know, go extinct along the way.
link |
And so it's similarly for other civilizations. So it is important to think about how hard it is
link |
to build a simulation. In terms of figuring out which of the disjuncts. But for the simulation
link |
argument itself, which is agnostic as to which of these three alternatives is true.
link |
You don't have to, like the simulation argument would be true whether or not,
link |
we thought this could be done in 500 years or it would take 500 million years.
link |
No, for sure. The simulation argument stands. I mean, I'm sure there might be some people who
link |
oppose it, but it doesn't matter. I mean, it's very nice those three cases cover it. But the fun part
link |
is at least not saying what the probabilities are, but kind of thinking about kind of
link |
intuitive reasoning about like what's more likely, what are the kind of things that would make
link |
some of the arguments less and more so like, but let's actually, I don't think we went through them.
link |
So number one is we destroy ourselves before we ever create simulation.
link |
Right. So that's kind of sad, but we have to think not just what might destroy us.
link |
I mean, so there could be some whatever disasters or meteorites slamming the earth
link |
a few years from now that could destroy us, right? But you'd have to postulate
link |
in order for this first disjunct to be true that almost all civilizations throughout
link |
the cosmos also failed to reach technological maturity.
link |
And the underlying assumption there is that there is likely a very large number of other
link |
intelligent civilizations. Well, if there are, yeah, then they would virtually all have to
link |
succumb in the same way. I mean, then that leads off another. I guess there are a lot of little
link |
digressions that they're interesting. Definitely, let's go there. Let's go there. I'll keep dragging
link |
us back. Well, there are these, there is a set of basic questions that always come up
link |
in conversations with interesting people. Yeah. Like the Fermi Paradox. Like there's like,
link |
you could almost define whether a person is interesting, whether at some point the question
link |
of the Fermi Paradox comes up. Well, so forward is where it looks to me that the universe is very
link |
big. I mean, in fact, according to the most popular current cosmological theory is infinitely big.
link |
And so then it would follow pretty trivially that it would contain a lot of other civilizations.
link |
In fact, infinite the many. If you have some local stochasticity and infinite the many,
link |
it's like, you know, infinite the many lumps of matter, one next to another, there's kind
link |
of random stuff in each one, then you're going to get all possible outcomes with probability
link |
one, infinitely repeated. So then certainly there would be a lot of random stuff in each one,
link |
so then certainly there would be a lot of extraterrestrials out there.
link |
Even short of that, if the universe is very big, there might be a finite but large number.
link |
If we were literally the only one, yeah, then of course,
link |
if we went extinct, then all of civilizations at our current stage would have gone extinct before
link |
becoming technological material. So then it kind of becomes trivially true that a very high fraction
link |
of those went extinct. But if we think there are many, I mean, it's interesting because there are
link |
certain things that possibly could kill us, like if you look at existential risks,
link |
and it might be a different, like that the best answer to what would be most likely to kill us
link |
might be a different answer than the best answer to the question. If there is something that kills
link |
almost everyone, what would that be? Because that would have to be some risk factor that was kind of
link |
uniform overall possible civilization. Yeah. So in this, for the sake of this argument,
link |
you have to think about not just us, but like every civilization dies out before they create
link |
the simulation or something very close to everybody. Okay. So what's number two in the
link |
Well, so number two is the convergence hypothesis that is that maybe like a lot of some of these
link |
civilizations do make it through to technological maturity. But out of those who do get there,
link |
they all lose interest in creating these simulations. So they just they have the capability
link |
of doing it, but they choose not to. Yeah, not just a few of them decide not to, but
link |
you know, out of a million, maybe not even a single one of them would do it.
link |
And I think when you say lose interest, that sounds like unlikely because it's like they
link |
get bored or whatever, but it could be so many possibilities within that. I mean, losing interest
link |
could be it could be anything from it being exceptionally difficult to do to fundamentally
link |
changing the sort of the fabric of reality. If you do it is ethical concerns, all those kinds
link |
of things could be exceptionally strong pressures. Well, certainly, I mean, yeah, ethical concerns.
link |
I mean, not really too difficult to do. I mean, in a sense, that's the first assumption that you
link |
get to technological maturity where you would have the ability using only a tiny fraction of your
link |
resources to create many, many simulations. So it wouldn't be the case that they would need to
link |
spend half of their GDP forever in order to create one simulation. And they had this like
link |
difficult debate about whether they should, you know, invest half of their GDP for this.
link |
It would more be like, well, if any little fraction of the civilization feels like doing this
link |
at any point during maybe their millions of years of existence, then there would be millions
link |
of simulations. But certainly, there could be many conceivable reasons for why there
link |
would be this convert, many possible reasons for not running ancestor simulations or other
link |
computer simulations, even if you could do so cheaply. By the way, what's an ancestor simulation?
link |
Well, that would be type of computer simulation that would contain people like those we think
link |
have lived on our planet in the past and like ourselves in terms of the types of experiences
link |
they have. And where those simulated people are conscious. So like, not just simulated in the
link |
same sense that a non player character would be simulated in the current computer game where
link |
it kind of has like an avatar body and then a very simple mechanism that moves it forward or
link |
backwards or but something where the simulated being has a brain, let's say that simulated
link |
at the sufficient level of granularity that it would have the same subjective experiences as we
link |
have. So where does consciousness fit into this? Do you think simulation, like is there
link |
different ways to think about how this can be simulated? Just like you're talking about now.
link |
Do we have to simulate each brain within the larger simulation? Is it enough to simulate
link |
just the brain, just the minds and not the simulation, not the big universe itself? Like,
link |
is there different ways to think about this? Yeah, I guess there is a kind of premise
link |
in the simulation argument, rolled in from philosophy of mind, that is that it would be
link |
possible to create a conscious mind in a computer. And that what determines whether
link |
some system is conscious or not is not like whether it's built from organic biological
link |
neurons, but maybe something like what the structure of the computation is that it implements.
link |
Right. So we can discuss that if we want, but I think it would be more forward as far as my view
link |
that it would be sufficient, say, if you had a computation that was identical to the computation
link |
in the human brain down to the level of neurons. So if you had a simulation with 100 billion neurons
link |
connected in the same Western human brain, and you then roll that forward with the same kind of
link |
synaptic weights and so forth, so you actually had the same behavior coming out of this
link |
as a human with that brain, then I think that would be conscious. Now, it's possible you could
link |
also generate consciousness without having that detailed assimilation. There, I'm getting more
link |
uncertain exactly how much you could simplify or abstract away.
link |
Can you look on that? What do you mean? So I missed where you're placing consciousness in
link |
the second. Well, so if you are a computationalist, do you think that what creates consciousness is
link |
the implementation of a computation? So some property, emergent property of the computation
link |
itself is the idea. Yeah, you could say that. But then the question is, what's the class of
link |
computations such that when they are run, consciousness emerges? So if you just have
link |
like something that adds one plus one plus one plus one, like a simple computation, you think maybe
link |
that's not going to have any consciousness. If on the other hand, the computation is one, like our
link |
human brains are performing, where as part of the computation, there is like, you know, a global
link |
workspace, a sophisticated attention mechanism, there is like self representations of other
link |
cognitive processes, and a whole lot of other things that possibly would be conscious. And in
link |
fact, if it's exactly like ours, I think definitely it would. But exactly how much less than the full
link |
computation that the human brain is performing would be required is a little bit, I think,
link |
of an open question. And you asked another interesting question as well, which is, would it be
link |
sufficient to just have, say, the brain or would you need the environment in order to generate
link |
the same kind of experiences that we have? And there is a bunch of stuff we don't know. I mean,
link |
if you look at, say, current virtual reality environments, one thing that's clear is that
link |
we don't have to simulate all details of them all the time in order for, say, the human player to
link |
have the perception that there is a full reality in there. You can have, say, procedurally generated
link |
virtual reality might only render a scene when it's actually within the view of the player character.
link |
And so similarly, if this environment that we perceive is simulated, it might be that
link |
all of the parts that come into our view are rendered at any given time. And a lot of aspects
link |
that never come into view, say, the details of this microphone I'm talking into exactly what
link |
each atom is doing at any given point in time might not be part of the simulation, only a more
link |
coarse grained representation. So that to me is actually from an engineering perspective, why the
link |
simulation hypothesis is really interesting to think about, is how much, how difficult is it to
link |
fake sort of in a virtual reality context? I don't know, fake is the right word, but to construct
link |
a reality that is sufficiently real to us to be immersive in the way that the physical world is,
link |
I think that's actually probably an answerable question of psychology, of computer science,
link |
of how, where's the line where it becomes so immersive that you don't want to leave that
link |
world? Yeah, or that you don't realize while you're in it that it is a virtual world.
link |
Yeah, those are two actually questions. Yours is the more sort of the good question about the realism.
link |
But mine, from my perspective, what's interesting is it doesn't have to be real, but how can we
link |
construct a world that we wouldn't want to leave? Yeah, I mean, I think that might be too low a bar.
link |
I mean, if you think, say, when people first had Pong or something like that, I'm sure there were
link |
people who wanted to keep playing it for a long time because it was fun and they wanted to be in
link |
this little world. I'm not sure we would say it's immersive. I mean, I guess in some sense it is,
link |
but like an absorbing activity doesn't even have to be. But they left that world though.
link |
So I think that bar is deceivingly high. So you can play Pong or Starcraft or would have
link |
more sophisticated games for hours, for months, while the work could be in a big addiction,
link |
but eventually they escaped that. So I mean, when it's absorbing enough that you would spend your
link |
entire, it would choose to spend your entire life in there. And then thereby changing the concept
link |
of what reality is, because your reality becomes the game, not because you're fooled,
link |
but because you've made that choice. Yeah, I mean, people might have different preferences
link |
regarding that. Some might, even if you had any perfect virtual reality,
link |
might still prefer not to spend the rest of their lives there. I mean, in philosophy,
link |
there's this experience machine thought experiment. Have you come across this?
link |
So Robert Nozick had this thought experiment where you imagine some crazy,
link |
super duper neuroscientists of the future have created a machine that could give you any experience
link |
you want if you step in there. And for the rest of your life, you can kind of pre programmed it
link |
in different ways. So your fondest dreams could come true. You could whatever you dream, you want
link |
to be a great artist, a great lover, like have a wonderful life, all of these things. If you step
link |
into the experience machine will be your experiences constantly happy. But would you kind of disconnect
link |
from the rest of reality and you would float there in a tank. And so Nozick thought that most
link |
people would choose not to enter the experience machine. I mean, many might want to go there for
link |
a holiday, but they wouldn't want us to check out of existence permanently. And so he thought that
link |
was an argument against certain views of value, according to what we value is a function of what
link |
we experience. Because in the experience machine, you could have any experience you want. And yet,
link |
many people would think that would not be much value. So therefore, what we value depends on
link |
other things than what we experience. So okay, can you take that argument further? What about
link |
the fact that maybe what we value is the up and down of life. So you could have up and downs in
link |
the experience machine, right? But what can't you have in the experience machine? Well, I mean,
link |
I mean, that then becomes an interesting question to explore. But for example, real connection
link |
with other people, if the experience machine is a solar machine, where it's only you,
link |
like that's something you wouldn't have there, you would have this objective experience that
link |
would be like fake people. But when if you gave somebody flowers, that wouldn't be anybody they
link |
who actually got happy, it would just be a little simulation of somebody smiling. But the
link |
simulation would not be the kind of simulation I'm talking about in the simulation argument where
link |
the simulated creature is conscious, it would just be a kind of smiley face that would look
link |
perfectly real to you. So we're now drawing a distinction between appear to be perfectly real
link |
and actually being real. Yeah. So that could be one thing. I mean, like a big impact on history,
link |
maybe it's also something you won't have if you check into this experience machine. So some people
link |
might actually feel the life I want to have for me is one where I have a big positive impact on
link |
history unfolds. So if you could kind of explore these different possible explanations for why it
link |
is, you wouldn't want to go into the experience machine if that's if that's what you feel.
link |
And one interesting observation regarding this nosic thought experiment and the conclusions
link |
you wanted to draw from it is how much is a kind of status quo effect. So a lot of people might not
link |
want to get this on current reality to plug into this dream machine. But if they instead were told,
link |
well, what you've experienced up to this point was a dream. Now, do you want to disconnect from this
link |
and enter the real world when you have no idea maybe what the real world is? Or maybe you could
link |
say, well, you're actually a farmer in Peru growing peanuts and you could live for the rest of your
link |
life in this. Or would you want to continue your dream life as Alex Friedman going around the
link |
world, making podcasts and doing research? So if the status quo was that they were actually in
link |
the experience machine, I think a lot of people might then prefer to live the life that they are
link |
familiar with rather than sort of bail out into. So essentially the change itself, the leap,
link |
whatever. Yeah, so it might not be so much the reality itself that we're after, but it's more
link |
that we are maybe involved in certain projects and relationships. And we have, you know, a self
link |
identity and these things that's our values are kind of connected with carrying that forward.
link |
And then whether it's inside a tank or outside a tank in Peru or whether inside a computer,
link |
outside a computer, that's kind of less important to what we ultimately care about.
link |
Yeah, so just to linger on it, it is interesting. I find maybe people are different, but I find
link |
myself quite willing to take the leap to the farmer in Peru, especially as the virtual
link |
reality system become more realistic. I find that possibility and I think more people would take
link |
that leap. But so in this thought experiment, just to make sure we are understanding, so in this
link |
case, the farmer in Peru would not be a virtual reality. That would be the real. The real. The
link |
real, your life, like before this whole experience machine started. Well, I kind of assumed from
link |
that description, you're being very specific, but that kind of idea just like,
link |
washes away the concept of what's real. I mean, I'm still a little hesitant about your kind of
link |
distinction between real and illusion. Because when you can have an illusion that feels, I mean,
link |
that looks real, I mean, what, I don't know how you can definitively say something is real or not.
link |
Like what's a good way to prove that something is real in that context?
link |
Well, so I guess in this case, it's more a definition. In one case, you're floating in a
link |
tank with these wires by the super duper neuroscientists, plugging into your head,
link |
giving you like Friedman experiences. In the other, you're actually tilling the soil in
link |
Peru, growing peanuts, and then those peanuts are being eaten by other people all around the
link |
world who buy the exports. And that's two different possible situations in the one and the
link |
same real world that you could choose to occupy.
link |
But just to be clear, when you're in a vat with wires and the neuroscientists,
link |
you can still go farming in Peru, right? No, well, you could, if you wanted to,
link |
you could have the experience of farming in Peru, but that wouldn't actually be any peanuts grown.
link |
Well, but what makes a peanut? So a peanut could be grown and you could feed things with that
link |
peanut. And why can't all of that be done in a simulation?
link |
I hope, first of all, that they actually have peanut farms in Peru. I guess we'll get a lot
link |
of comments out of us from angry. I was with you up until the point when you started talking about
link |
you should know you can't realize you're a lot of these in that climate.
link |
No, I mean, I think, I mean, I, in the simulation, I think there's a sense, the important sense in
link |
which it would all be real. Nevertheless, there is a distinction between inside a simulation and
link |
outside a simulation, or in the case of no six thought experiment, whether you're in the vat
link |
or outside the vat. And some of those differences may or may not be important. I mean, that comes
link |
down to your values and preferences. So if the, if the experience machine only gives you the
link |
experience of growing peanuts, but you're the only one in the experience machines.
link |
No, but there's other, you can, within the experience machine, others can plug in.
link |
Well, there are versions of the experience machine. So in fact, you might want to have
link |
distinguish different thought experiments, different versions of it. So in, like in the
link |
original thought experiment, maybe it's only you, right? Just you. So, and you think, I wouldn't
link |
want to go in there. Well, that tells you something interesting about what you value and what you
link |
care about. Then you could say, well, what if you add the fact that there would be other people in
link |
there and you would interact with them? Well, it starts to make it more attractive, right?
link |
Then you could add in, well, what if you could also have important long term effects on human
link |
history in the world and you could actually do something useful, even though you were in there,
link |
that makes it maybe even more attractive. Like you could actually have a life that had a purpose
link |
and consequences. So as you sort of add more into it, it becomes more similar to the baseline
link |
reality that you were comparing it to. Yeah, but I just think inside the experience machine and
link |
without taking those steps you just mentioned, you still have an impact on long term history
link |
of the creatures that live inside that, of the quote unquote, fake creatures that live inside
link |
that experience machine. And that, like at a certain point, if there's a person waiting for
link |
you inside that experience machine, maybe you're newly found wife and she dies, she has fear,
link |
she has hopes and she exists in that machine when you unplug yourself and plug back in,
link |
she's still there, going on about her life. Well, in that case, yeah, she starts to have
link |
more of an independent existence. Independent existence. But it depends, I think, on how she's
link |
implemented in the experience machine. Take one limit case where all she is is a static picture
link |
on the wall of photograph. So you think, well, I can look at her, but that's it. Then you think,
link |
well, it doesn't really matter much what happens to that. Any more than a normal photograph,
link |
if you tear it up, it means you can't see it anymore, but you haven't harmed the person
link |
whose picture you tore it up. But if she's actually implemented, say, at a neural level
link |
of detail, so that she's a fully realized digital mind with the same behavioral repertoire as you
link |
have, then very possibly she would be a conscious person like you are. And then you would, what you
link |
do in this experience machine would have real consequences for how this other mind felt.
link |
So you have to specify which of these experience machines you're talking about.
link |
I think it's not entirely obvious that it would be possible to have an experience machine that gave
link |
you a normal set of human experiences, which include experiences of interacting with other people,
link |
without that also generating consciousnesses corresponding to those other people. That is,
link |
if you create another entity that you perceive and interact with, that to you looks entirely
link |
realistic. Not just when you say hello, they say hello back, but you have a rich interaction
link |
many days, deep conversations. Like it might be that the only possible way of implementing that
link |
would be one that also as a side effect, instantiated this other person in enough detail
link |
that you would have a second consciousness there. I think that's to some extent an open question.
link |
So you don't think it's possible to fake consciousness and fake intelligence?
link |
Well, it might be. I mean, I think you could certainly fake, if you have a very limited
link |
interaction with somebody, you could certainly fake that. If all you have to go on is somebody
link |
said hello to you, that's not enough for you to tell whether that was a real person there
link |
or a prerecorded message or a very superficial simulation that has no consciousness,
link |
because that's something easy to fake. We could already fake it. Now you can record a voice
link |
recording. But if you have a richer set of interactions where you're allowed to ask open
link |
ended questions and probe from different angles that you couldn't give canned answer to all of
link |
the possible ways that you could probe it, then it starts to become more plausible that the only
link |
way to realize this thing in such a way that you would get the right answer from any which angle
link |
you probe it would be a way of instantiating it where you also instantiated a conscious mind.
link |
Yeah, I'm with you on the intelligence part, but there's something about me that says consciousness
link |
is easier to fake. I've recently gotten my hands on a lot of rubas. Don't ask me why or how.
link |
And I've made them, this is just a nice robotic mobile platform for experiments,
link |
and I made them scream and or moan and pain and so on just to see when they're responding to me.
link |
And it's just a sort of psychological experiment on myself. And I think they appear conscious to me
link |
pretty quickly. Like I, to me, at least my brain can be tricked quite easily. So if I introspect,
link |
it's harder for me to be tricked that something is intelligent. So I just have this feeling that
link |
inside this experience machine, just saying that you're conscious and having certain qualities
link |
of the interaction like being able to suffer, like being able to hurt, like being able to wonder
link |
about the essence of your own existence, not actually, I mean, the creating the illusion
link |
that you're wondering about it is enough to create the feeling of consciousness and
link |
the illusion of consciousness. And because of that, create a really immersive experience
link |
to where you feel like that is the real world. So you think there's a big gap between
link |
being appearing conscious and being conscious? Or is it that you think it's very easy to be
link |
conscious? I'm not actually sure what it means to be conscious. All I'm saying is the illusion of
link |
consciousness is enough for this to create a social interaction that's as good as if the
link |
thing was conscious, meaning I'm making it about myself. Right. Yeah. I mean, I guess there are a
link |
few difficulties. One is how good the interaction is, which might, I mean, if you don't really care
link |
about probing hard for whether the thing is conscious, maybe it would be a satisfactory
link |
interaction, whether or not you really thought it was conscious. Now, if you really care about it being
link |
conscious inside this experience machine, how easy would it be to fake it? And you say
link |
it sounds really easy. But then the question is, would that also mean it's very easy to
link |
instantiate consciousness? It's much more widely spread in the world than we have thought. It
link |
doesn't require a big human brain with 100 billion neurons. All you need is some system that exhibits
link |
basic intentionality and can respond and you already have consciousness. In that case, I guess
link |
you still have a close coupling. I guess a data case would be where they can come apart,
link |
where you could create the appearance of there being a conscious mind without actually not being
link |
another conscious mind. I'm somewhat agnostic exactly where these lines go. I think one
link |
observation that makes it plausible that you could have very realistic appearances
link |
relatively simply, which also is relevant for the simulation argument. And in terms of thinking
link |
about how realistic would a virtual reality model have to be in order for the simulated
link |
creature not to notice that anything was awry. Well, just think of our own humble brains during
link |
the wee hours of the night when we are dreaming. Many times, well, dreams are very immersive,
link |
but often you also don't realize that you're in a dream. And that's produced by simple primitive
link |
three pound lumps of neural matter effortlessly. So if a simple brain like this can create the
link |
virtual reality that seems pretty real to us, then how much easier would it be for a super
link |
intelligent civilization with planetary sized computers optimized over the eons to create
link |
a realistic environment for you to interact with? Yeah, by the way, behind that intuition
link |
is that our brain is not that impressive relative to the possibilities of what technology could
link |
bring. It's also possible that the brain is the epitome is the ceiling. Like just the ceiling.
link |
How is that possible? Meaning like this is the smartest possible thing that the universe could
link |
create. So that seems unlikely to me. Yeah, I mean, for some of these reasons we alluded to
link |
earlier in terms of designs we already have for computers that would be faster by many orders
link |
of magnitude than the human brain. Yeah, but it could be that the constraints, the cognitive
link |
constraints in themselves is what enables the intelligence. So the more powerful you make the
link |
computer, the less likely it is to become super intelligent. This is where I say dumb things
link |
to push back. Yeah, I'm not sure. I mean, so there are different dimensions of intelligence.
link |
A simple one is just speed. Like if you can solve the same challenge faster in some sense,
link |
you're smarter. So there I think we have very strong evidence for thinking that you could have
link |
a computer in this universe that would be much faster than the human brain and therefore have
link |
speed superintelligence, like be completely superior, maybe a million times faster.
link |
Then maybe there are other ways in which you could be smarter as well, maybe more qualitative
link |
ways, right? And there the concepts are a little bit less clear cut. So it's harder to make a very
link |
crisp, neat, firmly logical argument for why that could be qualitative superintelligence as
link |
opposed to just things that were faster. Although I still think it's very plausible.
link |
And for various reasons that are less than watertight arguments. But for example, if you look at
link |
animals and even within humans, there seems to be Einstein versus random person. It's not just
link |
that Einstein was a little bit faster. But how long would it take a normal person to invent
link |
general relativity? It's not 20% longer than it took Einstein or something like that. I don't
link |
know whether they would do it at all or it would take millions of years or some totally bizarre.
link |
But your intuition is that the compute size will get you go. Increasing the size of the computer
link |
and the speed of the computer might create some much more powerful levels of intelligence that
link |
would enable some of the things we've been talking about with the simulation, being able to simulate
link |
an ultra realistic environment, ultra realistic reception of reality.
link |
Yeah. I mean, strictly speaking, it would not be necessary to have superintelligence in order to
link |
have, say, the technology to make these simulations, ancestor simulations or other kinds of simulations.
link |
As a matter of fact, I think if we are in a simulation, it would most likely be one built
link |
by a civilization that had superintelligence. It certainly would help. I mean, it could build
link |
more efficient, larger scale structures if you had superintelligence. I also think that if you had
link |
the technology to build these simulations, that's like a very advanced technology. It seems kind
link |
of easier to get the technology to superintelligence. So I'd expect by the time they could make these
link |
fully realistic simulations of human history with human brains in there, before that, they
link |
got to that stage that would have figured out how to create machine superintelligence or maybe
link |
biological enhancements of their own brains if they were biological creatures to start with.
link |
So we talked about the three parts of the simulation argument. One, we destroy ourselves
link |
before we ever create the simulation. Two, we somehow, everybody somehow loses interest in
link |
creating simulation. Three, we're living in a simulation. So you've kind of, I don't know if
link |
your thinking has evolved on this point, but you kind of said that we know so little that these
link |
three cases might as well be equally probable. So probabilistically speaking, where do you stand
link |
on this? Yeah, I mean, I don't think equal necessarily would be the most supported probability
link |
assignment. So how would you, without assigning actual numbers, what's more or less likely in
link |
your view? Well, I mean, I've historically tended to punt on the question of like as between these
link |
three. So maybe you asked me another way is which kind of things would make each of these more or
link |
less likely? What kind of, yeah. I mean, certainly in general terms, if you think anything that say
link |
increases or reduces the probability of one of these would tend to slash probability around
link |
on the other. So if one becomes less probable, like the other would have to, because it's going to
link |
add up to one. Yes. So if we consider the first hypothesis, the first alternative that there's
link |
this filter that makes it so that virtually no civilization reaches technical maturity.
link |
In particular, our own civilization, if that's true, then it's like very unlikely that we would
link |
reach technical maturity, because if almost no civilization at our stage does it, then
link |
it's unlikely that we do it. So I'm sorry, can you longer on that for a second? Well,
link |
so if it's the case that almost all civilizations at our current stage of technical maturity fail,
link |
at our current stage of technical development, fail to reach maturity,
link |
that would give us very strong reason for thinking we will fail to reach technical maturity.
link |
Oh, and also sort of the flip side of that is the fact that we've reached it means that many
link |
other civilizations have reached this point. Yeah. So that means if we get closer and closer to
link |
actually reaching technical maturity, there's less and less distance left where we could
link |
go extinct before we are there. And therefore, the probability that we will reach increases
link |
as we get closer. And that would make it less likely to be true that almost all civilizations
link |
at our current stage failed to get there. Like we would have this, the one case we had started
link |
ourselves would be very close to getting there. That would be strong evidence is not so hard to
link |
get to technical maturity. So to the extent that we feel we are moving nearer to technical
link |
maturity, that would tend to reduce the probability of the first alternative and increase the probability
link |
of the other to it doesn't need to be a monotonic change. Like if every once in a while, some new
link |
threat comes into view, some bad new thing you could do with some novel technology, for example,
link |
you know, that that could change our probabilities in the other direction.
link |
But that that technology again, you have to think about as that technology has to be able to
link |
equally in an even way affect every civilization out there.
link |
Yeah, pretty much. I mean, that's strictly speaking, it's not true. I mean, I could be two
link |
different existential risk and every civilization, you know, one or the other, like, but none of them
link |
kills more than 50%. Like, yeah, but I incidentally, so in some of my the work, I mean, on machine
link |
superintelligence, like some existential risks related to sort of superintellent AI and how we
link |
must make sure, you know, to handle that wisely and carefully. It's not the right kind of existential
link |
catastrophe to make the first alternative true, though, like it might be bad for us.
link |
If the future lost a lot of value as a result of it being shaped by some process that optimized for
link |
some completely non human value. But even if we got killed by machine superintelligence is that
link |
machine superintelligence might still attain technological maturity.
link |
So I see. So you're not very, you're not human exclusive. This could be any intelligent species
link |
that achieves like it's all about the technological maturity. It's not that the humans have to
link |
So like superintelligence because it replaced us. And that's just as well for the simulation
link |
argument. Yeah, I mean, it could interact with the second by alternative. Like if the thing that
link |
replaced us was either more likely or less likely, then we would be to have an interest in creating
link |
ancestor simulations, you know, that that could affect probabilities. But yeah, to a first order,
link |
like if we all just die, then yeah, we won't produce any simulations because we are dead. But if we
link |
all die and get replaced by some other intelligent thing that then gets to technological maturity,
link |
the question remains, of course, if my not that thing, then use some of its resources to do this
link |
stuff. So can you reason about this stuff? This is given how little we know about the universe.
link |
Is it reasonable to reason about these probabilities? So like how little, well,
link |
well, maybe you can disagree. But to me, it's not trivial to figure out how difficult it is to
link |
build a simulation. We kind of talked about it a little bit. We've also don't know, like as we
link |
try to start building it, like start creating virtual worlds and so on, how that changes the
link |
fabric of society. Like there's all these things along the way that can fundamentally change just
link |
so many aspects of our society about our existence that we don't know anything about. Like the kind
link |
of things we might discover when we understand to a greater degree the fundamental, the physics,
link |
like the theory, if we have a breakthrough, have a theory and everything, how that changes,
link |
how that changes deep space exploration and so on. So like, is it still possible to reason about
link |
probabilities given how little we know? Yes, I think though, there will be a large residual of
link |
uncertainty that we'll just have to acknowledge. And I think that's true for most of these big
link |
picture questions that we might wonder about. It's just we are small, short lived, small brained,
link |
cognitively very limited humans with little evidence and it's amazing we can figure out as
link |
much as we can really about the cosmos. But okay, so there's this cognitive trick that seems to happen
link |
when I look at the simulation argument, which for me, it seems like case one and two feel
link |
unlikely. I want to say feel unlikely as opposed to sort of like, it's not like I have too much
link |
scientific evidence to say that either one or two are not true. It just seems unlikely that every
link |
single civilization destroys itself. And it seems like feels unlikely that the civilizations lose
link |
interest. So naturally, without necessarily explicitly doing it, but the simulation argument
link |
basically says it's very likely we're living in a simulation. Like to me, my mind naturally goes
link |
there. I think the mind goes there for a lot of people. Is that the incorrect place for it to go?
link |
Well, not necessarily. I think the second alternative which has to do with the motivations
link |
and interests of technological and mature civilizations. I think there is much we don't
link |
understand about that. Can you talk about that a little bit? What do you think? I mean, this
link |
question that pops up when you build an AGI system or build a general intelligence or
link |
how does that change your motivations? Do you think it will fundamentally transform our motivations?
link |
Well, it doesn't seem that implausible that once you take this leap to technological maturity.
link |
I mean, I think it involves creating machine superintelligence possibly that would be sort of
link |
on the path for basically all civilizations, maybe before they are able to create large
link |
numbers of ancestor simulations. That possibly could be one of these things that quite radically
link |
changes the orientation of what a civilization is, in fact, optimizing for. There are other
link |
things as well. At the moment, we have not perfect control over our own being, our own mental states,
link |
our own experiences are not under our direct control. For example, if you want to experience
link |
a pleasure and happiness, you might have to do a whole host of things in the external world to
link |
try to get into the stage, into the mental state where you experience pleasure. Like when people
link |
get some pleasure from eating great food, well, they can't just turn that on. They have to kind
link |
of actually go to a nice restaurant and then they have to make money. So there's like all this kind
link |
of activity that maybe arises from the fact that we are trying to ultimately produce mental states,
link |
but the only way to do that is by a whole host of complicated activities in the external world.
link |
Now, at some level of technological development, I think we'll become auto potent in the sense of
link |
gaining direct ability to choose our own internal configuration and enough knowledge and insight
link |
to be able to actually do that in a meaningful way. So then it could turn out that there are a lot
link |
of instrumental goals that would drop out of the picture and be replaced by other instrumental
link |
goals because we could now serve some of these final goals in more direct ways. And who knows how
link |
all of that shakes out after civilizations reflect on that and converge and different
link |
attractors and so on and so forth. And that could be new instrumental considerations that come into
link |
view as well that we are just oblivious to that would maybe have a strong shaping effect on actions,
link |
like very strong reasons to do something or not to do something. And we just don't realize
link |
they are there because we are so dumb, bumbling through the universe. But if almost inevitably
link |
on route to attaining the ability to create many answers to simulations, you do have this
link |
cognitive enhancement or advice from superintelligence or you yourself, then maybe there's like this
link |
additional set of considerations coming into view. And yesterday I asked, it's obvious that the thing
link |
that makes sense is to do X. Whereas right now it seems you could X, Y or Z and different people
link |
will do different things and we are kind of random in that sense. Yeah, because at this time,
link |
with our limited technology, the impact of our decisions is minor. I mean, that's starting
link |
to change in some ways. Well, I'm not sure how it follows that the impact of our decisions is minor.
link |
Well, it's starting to change. I mean, I suppose 100 years ago was minor. It's starting to...
link |
Well, it depends on how you view it. So what people did 100 years ago still have effects on
link |
the world today. Oh, as I see, as a civilization in the togetherness. Yeah. So it might be that
link |
the greatest impact of individuals is not at technological maturity or very far down. It might
link |
be earlier on when there are different tracks, civilization could go down. I mean, maybe the
link |
population is smaller, things still haven't settled out. If you count indirect effects, that
link |
those could be bigger than the direct effects that people have later on. So part three of the
link |
argument says that, so that leads us to a place where eventually somebody creates a simulation.
link |
I think you had a conversation with Joe Rogan. I think there's some aspect here where you get
link |
stuck a little bit. How does that lead to we're likely living in a simulation? So this kind of
link |
probability argument, if somebody eventually creates a simulation, why does that mean that
link |
we're now in a simulation? What you get to if you accept alternative three first is there would be
link |
more simulated people with our kinds of experiences than non simulated ones. If you look at the world
link |
as a whole, by the end of time, as it were, you just count it up, that would be more simulated
link |
ones than non simulated ones. Then there is an extra step to get from that. If you assume that,
link |
suppose for the sake of the argument that that's true, how do you get from that to the statement
link |
we are probably in a simulation? So here you're introducing an indexical statement like it's
link |
that this person right now is in a simulation. There are all these other people that are in
link |
simulations and some that are not in the simulation. But what probability should you have that you
link |
yourself is one of the simulated ones in the setup. So I call it the bland principle of
link |
indifference, which is that in cases like this, when you have two sets of observers,
link |
one of which is much larger than the other, and you can't from any internal evidence you have,
link |
tell which set you belong to. You should assign a probability that's proportional to the size
link |
of these sets so that if there are 10 times more simulated people with your kinds of experiences,
link |
you would be 10 times more likely to be one of those. Is that as intuitive as it sounds?
link |
I mean, that seems kind of, if you don't have enough information, you should rationally just
link |
assign the same probability as the size of the set. It seems pretty plausible to me.
link |
Where are the holes in this? Is it at the very beginning, the assumption that everything stretches
link |
sort of you have infinite time essentially? You don't need infinite time.
link |
You just need how long does the time you take? However long it takes, I guess, for a universe
link |
to produce an intelligent civilization that then attains the technology to run some
link |
ancestry simulations. Got you. When the first simulation is created, that stretch of time
link |
just a little longer than they'll all start creating simulations, kind of like order of
link |
matters. Well, I mean, it might be different. If you think of there being a lot of different
link |
planets and some subset of them have life and then some subset of those get intelligent life
link |
and some of those maybe eventually start creating simulations, they might get started at quite
link |
different times. Like maybe on some planet, it takes a billion years longer before you get
link |
like monkeys or before you get even bacteria than on another planet. So this might happen
link |
when kind of at different cosmological epochs. Is there a connection here to the doomsday
link |
argument and that sampling there? Yeah, there is a connection in that they both
link |
involve an application of anthropic reasoning that is reasoning about these kind of indexical
link |
propositions. But the assumption you need in the case of the simulation argument
link |
is much weaker than the assumption you need to make the doomsday argument go through.
link |
What is the doomsday argument? And maybe you can speak to the anthropic reasoning in more
link |
general. Yeah, that's a big and interesting topic in its own right, anthropics. But the
link |
doomsday argument is this really first discovered by Brandon Carter, who was a theoretical physicist
link |
and then developed by philosopher John Leslie. I think it might have been discovered initially
link |
in the 70s or 80s. And Leslie wrote this book, I think in 96. And there are some other versions
link |
as well by Richard Gott, he's a physicist, but let's focus on the Carter Leslie version where
link |
it's an argument that we have systematically underestimated the probability that
link |
he might not be able to go extinct soon. Now, I should say most people probably
link |
think at the end of the day, there is something wrong with this doomsday argument that it doesn't
link |
really hold. It's like there's something wrong with it. But it's proved hard to say exactly what
link |
is wrong with it. And different people have different accounts. My own view is it seems
link |
inconclusive. But I can say what the argument is. Yeah, that would be good. Yeah, so maybe it's
link |
easy to explain via an analogy to sampling from urns. So imagine you have a big, imagine you
link |
have two urns in front of you, and they have balls in them that have numbers. The two urns look
link |
the same, but inside one, there are 10 balls, ball number one, two, three up to ball number 10.
link |
And then in the other urn, you have a million balls numbered one to a million. And now somebody
link |
puts one of these urns in front of you and asks you to guess what's the chance it's the 10 ball
link |
urn. And you say, well, 50, 50, I can't tell which urn it is. But then you're allowed to
link |
reach in and pick a ball at random from the urn. And that's suppose you find that it's ball number
link |
seven. So that's strong evidence for the 10 ball hypothesis. Like, it's a lot more likely that
link |
you would get such a low numbered ball, if there are only 10 balls in the urn, like it's in fact
link |
10% done, right? Then if there are a million balls, it would be very unlikely you would get number
link |
seven. So you perform a Bayesian update. And if your prior was 50, 50, that it was the 10 ball
link |
urn, you become virtually certain after finding the random sample was seven that it only has 10
link |
balls in it. So in the case of the urns, this is uncontroversial, just elementary probability
link |
theory. The Doomsday argument says that you should reason in a similar way with respect to
link |
different hypotheses about how many balls there will be in the urn of humanity, I said,
link |
for how many humans there will ever be by the time we go extinct. So to simplify, let's suppose we
link |
only consider two hypotheses, either maybe 200 billion humans in total, or 200 trillion humans
link |
in total. You could fill in more hypotheses, but it doesn't change the principle here. So it's
link |
easiest to see if we just consider these two. So you start with some prior based on ordinary,
link |
empirical ideas about threats to civilization and so forth. And maybe you say it's a 5% chance that
link |
we will go extinct. By the time there will have been 200 billion only, you're kind of optimistic,
link |
let's say, I think probably we'll make it through colonized universe. But then, according to this
link |
Doomsday argument, you should take off your own birth rank as a random sample. So your birth rank
link |
is your sequence in the position of all humans that have ever existed. And it turns out you're
link |
about a human number of 100 billion, you know, give or take. That's like roughly how many people
link |
have been born before you. That's fascinating, because I probably, we each have a number.
link |
We would each have a number in this. I mean, obviously, the exact number would depend on
link |
where you started counting, like which ancestors was human enough to count as human. But those
link |
are not really important. They're relatively few. So yeah, so you're roughly 100 billion. Now,
link |
if they're only going to be 200 billion in total, that's a perfectly unremarkable number. You're
link |
somewhere in the middle, right? Run of the mill human, completely unsurprising. Now, if they're
link |
going to be 200 trillion, you would be remarkably early. Like, what are the chances out of these
link |
200 trillion human that you should be human number 100 billion? That seems it would have
link |
a much lower conditional probability. And so analogously to how in the urn case, you thought
link |
after finding this low number random sample, you updated in favor of the urn having few balls.
link |
Similarly, in this case, you should update in favor of the human species having a lower total
link |
number of members that is doomed soon. You said doomed soon. That's the hypothesis in this case
link |
that it will end 100 billion. I just like that term for the hypothesis. Yeah.
link |
So what it kind of crucially relies on, the doomed argument, is the idea that you should reason
link |
as if you were a random sample from the set of all humans that will ever have existed.
link |
If you have that assumption, then I think the rest kind of follows. The question then is,
link |
why should you make that assumption? In fact, you know you're 100 billion, so where do you get this
link |
prior? And then there is like a literature on that with different ways of supporting that assumption.
link |
There's just one example of anthropocrysine, right? That seems to be kind of convenient when you
link |
think about humanity, when you think about sort of even like existential threats and so on,
link |
as it seems that quite naturally that you should assume that you're just an average case.
link |
Yeah, that you're kind of a typical randomly sample. Now in the case of the doomed argument,
link |
it seems to lead to what intuitively we think is the wrong conclusion. Or at least many people
link |
have this reaction that there's got to be something fishy about this argument. Because from very,
link |
very weak premises, it gets this very striking implication that we have almost no chance of
link |
reaching size 200 trillion humans in the future. And how can we possibly get there just by reflecting
link |
on when we were born? It seems you would need sophisticated arguments about the impossibility
link |
of space colonization, blah, blah. So one might be tempted to reject this key assumption. I call
link |
it the self sampling assumption. The idea that you should reason as if you're a random sample from
link |
all observers or in your some reference class. However, it turns out that in other domains,
link |
it looks like we need something like this self sampling assumption to make sense of
link |
bona fide scientific inferences in contemporary cosmology, for example, we have these multiverse
link |
theories. And according to a lot of those, all possible human observations are made. I mean,
link |
if you have a sufficiently large universe, you will have a lot of people observing all kinds
link |
of different things. So if you have two competing theories, say about the value of some constant,
link |
it could be true, according to both of these theories, that there will be some observers
link |
observing the value that corresponds to the other theory, because there will be some observers that
link |
have hallucinations. So there's a local fluctuation or a statistically anomalous measurement,
link |
these things will happen. And if enough observers make enough different observations,
link |
there will be some that sort of by chance make these different ones. And so what we would want to say
link |
is, well, many more observers, a larger proportion of the observers will observe as it were the true
link |
value. And a few will observe the wrong value. If we think of ourselves as a random sample,
link |
we should expect with our own probability to observe the true value and that will then allow us
link |
to conclude that the evidence we actually have is evidence for the theories we think are supported.
link |
It kind of done is a way of making sense of these inferences that clearly seem correct,
link |
that we can make various observations and infer what the temperature of the cosmic background is
link |
and the fine structure constant and all of this. But it seems that without rolling in some assumption
link |
similar to the self sampling assumption, this inference doesn't go through. And there are
link |
other examples. So there are these scientific contexts where it looks like this kind of
link |
anthropic reasoning is needed and makes perfect sense. And yet, in the case of the Dubster argument,
link |
it has this weird consequence and people might think there's something wrong with it there.
link |
So there's then this project that would consistent try to figure out what are the legitimate ways
link |
of reasoning about these indexical facts when observer selection effects are in play. In other
link |
words, developing a theory of anthropics. And there are different views of looking at that.
link |
And it's a difficult methodological area. But to tie it back to the simulation argument,
link |
the key assumption there, this bland principle of indifference, is much weaker than the self
link |
sampling assumption. So if you think about in the case of the Dubster argument, it says you
link |
should reason as if you are a random sample from all humans that would have lived, even though in
link |
fact you know that you are about number 100 billionth human and you're alive in the year 2020,
link |
whereas in the case of the simulation argument, it says that, well, if you actually have no way
link |
of telling which one you are, then you should assign this kind of uniform probability.
link |
Yeah, your role as the observer in the simulation argument is different, it seems like.
link |
Who's the observer? I keep assigning the individual consciousness.
link |
Well, a lot of observers in the simulation, in the context of the simulation argument,
link |
the relevant observers would be A, the people in original histories and B, the people in simulations.
link |
So this would be the class of observers that we need. I mean, there are also maybe the simulators,
link |
but we can set those aside for this. So the question is, given that class of observers,
link |
a small set of original history observers and the large class of simulated observers,
link |
which one should you think is you? Where are you amongst this set of observers?
link |
I'm maybe having a little bit of trouble wrapping my head around the intricacies of
link |
what it means to be an observer in the different instantiations of the anthropic reasoning cases
link |
that we mentioned. It's like the observer. No, I mean, it may be an easier way of putting it,
link |
it's just like, are you simulated or are you not simulated, given this assumption that these
link |
two groups of people exist? Yeah, in the simulation case, it seems pretty straightforward.
link |
Yeah, so the key point is the methodological assumption you need to make to get the simulation
link |
argument to where it wants to go is much weaker and less problematic than the methodological
link |
assumption you need to make to get the doomsday argument to its conclusion. Maybe the doomsday
link |
argument is sound or unsound, but you need to make a much stronger and more controversial assumption
link |
to make it go through. In the case of the simulation argument, I guess one
link |
maybe way intuition popped to support this bland principle of indifference,
link |
is to consider a sequence of different cases where the fraction of people who are simulated
link |
to non simulated approaches one. In the limiting case where everybody is simulated,
link |
obviously you can deduce with certainty that you are simulated. If everybody
link |
with your experiences is simulated and you know you've got to be one of those,
link |
you don't need a probability at all, you just kind of logically
link |
conclude it, right? So then as we move from a case where say 90% of everybody is simulated,
link |
99.9%, it should seem possible that the probability assigned should sort of approach
link |
one, certainty, as the fraction approaches the case where everybody is in the simulation.
link |
And so you wouldn't expect that to be a discrete. Well, if there's one non simulated person,
link |
then it's 50, 50, but if we move that, then it's 100%. There are other
link |
arguments as well one can use to support this bland principle of indifference, but
link |
that might be nice too. But in general, when you start from time equals zero and go into the future,
link |
the fraction of simulated, if it's possible to create simulated worlds,
link |
the fraction simulated worlds will go to one. Well, I mean, is that an obvious kind of thing?
link |
Well, it won't probably go all the way to one. In reality, there would be some
link |
ratio, although maybe a technological mature civilization could run a lot of
link |
simulations using a small portion of its resources. It probably wouldn't be able to
link |
run infinitely many. I mean, if we take say the observed, the physics in the observed universe,
link |
if we assume that that's also the physics at the level of the simulators, that would be limits
link |
to the amount of information processing that anyone civilization could perform in its future
link |
trajectory. Right. Well, first of all, there's limited amount of matter you can get your hands
link |
off because with the positive cosmological constant, the universe is accelerating,
link |
there's like a finite sphere of stuff, even if you travel with the speed of light that you
link |
could ever reach to have a finite amount of stuff. And then if you think there is like a lower limit
link |
to the amount of loss you get when you perform an erasure of a computation, or if you think,
link |
for example, just matter gradually over cosmological time scales, decay, maybe protons, decay,
link |
other things, you radiate out gravitational waves, like there's all kinds of seemingly
link |
unavoidable losses that occur. So eventually, we'll have something like a heat death of the
link |
universe or a close death or whatever. So it's fine. But of course, we don't know which, if there's
link |
many ancestral simulations, we don't know which level we are. So there could be,
link |
couldn't there be like an arbitrary number of simulations that spawned ours? And those had
link |
more resources in terms of physical universe to work with? Sorry, what do you mean that that could
link |
be? Okay, so if simulations spawn other simulations, it seems like each new spawn has fewer resources
link |
to work with. But we don't know at which step along the way we are at,
link |
any one observer doesn't know whether we're in level 42, or 100, or one, or is that not
link |
matter for the resources? I mean, it's true that there would be uncertainty asked,
link |
you could have stacked simulations. Yes. And that could then be uncertainty as to which level we
link |
are at. As you remarked also, all the computations performed in a simulation within a simulation
link |
also have to be expanded at the level of the simulation. So the computer in basement reality
link |
where all these simulations with the simulations with the simulations are taking place, like that
link |
computer ultimately, it's CPU or whatever it is, like that has to power this whole tower, right? So
link |
if there is a finite compute power in basement reality, that would impose a limit to how tall
link |
this tower can be. And if each level kind of imposes a large extra overhead, you might think
link |
maybe the tower would not be very tall, that most people would be low down in the tower.
link |
I love the term basement reality. Let me ask one of the popularizers, you said there's many
link |
through this, when you look at sort of the last few years of the simulation hypothesis,
link |
just like you said, it comes up every once in a while, some new community discovers it and so on.
link |
But I would say one of the biggest popularizers of this idea is Elon Musk. Do you have any kind
link |
of intuition about what Elon thinks about when he thinks about simulation? Why is this of such
link |
interest? Is it all the things we've talked about, or is there some special kind of intuition about
link |
simulation that he has? I mean, you might have a better, I think, I mean, why it's of interest,
link |
I think it seems pretty obvious why to the extent that one think the argument is credible,
link |
why it would be of interest. If it's correct, tell us something very important about the world,
link |
in one way or the other, whichever of the three alternatives for a simulation that seems arguably
link |
one of the most fundamental discoveries, right? Now, interestingly, in the case of someone like
link |
Elon, so there's like the standard arguments for why you might want to take the simulation hypothesis
link |
seriously, the simulation argument, right? In the case that if you are actually Elon Musk, let
link |
us say, there's a kind of an additional reason in that what are the chances you would be Elon Musk?
link |
Like, it seems like maybe there would be more interest in simulating the lives of very unusual
link |
and remarkable people. So, if you consider not just assimilations where all of human history
link |
or the whole of human civilization are simulated, but also other kinds of simulations which only
link |
include some subset of people. Like, in those simulations that only include a subset,
link |
it might be more likely that they would include subsets of people with unusually interesting
link |
or consequential lives. If you're Elon Musk, you got to wonder, right? It's more likely that
link |
if you are Donald Trump or if you are Bill Gates or you're like some particularly
link |
like distinctive character, you might think that that add, I mean, if you just think of yourself
link |
into the shoes, right? It's got to be like an extra reason to think. That's kind of...
link |
So interesting. So, on a scale of like farmer and Peru to Elon Musk, the more you get towards
link |
the Elon Musk, the higher the probability... You'd imagine there would be some extra boost from that.
link |
There's an extra boost. So, he also asked the question of what he would ask an AGI saying,
link |
the question being what's outside the simulation. Do you think about the answer to this question
link |
if we are living a simulation? What is outside the simulation? So, the programmer of the simulation?
link |
Yeah. I mean, I think it connects to the question of what's inside the simulation in that
link |
if you had views about the creators of the simulation, it might help you
link |
make predictions about what kind of simulation it is, what might happen, what happens after
link |
the simulation if there is some after, but also like the kind of setup. So, these two questions
link |
would be quite closely intertwined. But do you think it would be very surprising to... Is the
link |
stuff inside the simulation, is it possible for it to be fundamentally different than the stuff
link |
outside? Yeah. Another way to put it, can the creatures inside the simulation be smart enough
link |
to even understand or have the cognitive capabilities or any kind of information processing
link |
capabilities enough to understand the mechanism that created them? They might understand some
link |
aspects of it. I mean, it's a level of explanation, like degrees to which you can understand. So,
link |
does your dog understand what it is to be human? Well, it's got some idea, like humans are these
link |
physical objects that move around and do things. And a normal human would have a deeper understanding
link |
of what it is to be human. And maybe some very experienced psychologists or great novelists
link |
might understand a little bit more about what it is to be human. And maybe superintelligence
link |
could see right through your soul. So, similarly, I do think that we are quite limited in our
link |
ability to understand all of the relevant aspects of the larger context that we exist in.
link |
But there might be hope for some. I think we understand some aspects of it. But
link |
how much good is that? If there's one key aspect that changes the significance of all
link |
the other aspects. So, we understand maybe seven out of ten key insights that you need.
link |
But the answer actually varies completely, depending on what number eight, nine, and
link |
ten insight is. It's like whether you want to... Suppose that the big task were to guess whether
link |
a certain number was odd or even, like a ten digit number. And if it's even, the best thing
link |
for you to do in life is to go north. And if it's odd, the best thing for you to go south.
link |
Now, we are in a situation where maybe through our science and philosophy, we figured out what
link |
the first seven digits are. So, we have a lot of information, right? Most of it we figured out.
link |
But we are clueless about what the last three digits are. So, we are still completely clueless
link |
about whether the number is odd or even, and therefore whether we should go north or go south.
link |
I feel that's an analogy, but I feel we're somewhat in that predicament. We know a lot about the
link |
universe. We've come maybe more than half of the way there to kind of fully understanding it,
link |
but the parts we are missing are possibly ones that could completely change the overall
link |
upshot of the thing. And including change our overall view about what the scheme of
link |
priorities should be or which strategic direction would make sense to pursue.
link |
Yeah, I think your analogy of us being the dog, trying to understand human beings is an entertaining
link |
one and probably correct. The closer the understanding tends from the dog's viewpoint to us human
link |
psychologist's viewpoint, the steps along the way there will have completely transformative ideas
link |
of what it means to be human. So, the dog has a very shallow understanding. It's interesting to
link |
think that to analogize that a dog's understanding of a human being is the same as our current
link |
understanding of the fundamental laws of physics in the universe. Oh, man. Okay, we spent an hour
link |
or 40 minutes talking about the simulation. I like it. Let's talk about superintelligence,
link |
at least for a little bit. And let's start at the basics. What to you is intelligence?
link |
Yeah, I tend not to get too stuck with the definitional question. I mean, the common sense
link |
understand, like the ability to solve complex problems, to learn from experience, to plan,
link |
to reason, some combination of things like that. Is consciousness mixed up into that or no? Is
link |
consciousness mixed up into that? Well, I don't think, I think it could be fairly intelligent,
link |
at least without being conscious, probably. So, then what is superintelligence? Yeah,
link |
that would be like something that was much more of that, had much more general cognitive capacity
link |
than we humans have. So, if we talk about general superintelligence, it would be much faster learner
link |
be able to reason much better, make plans that are more effective at achieving its goals,
link |
say in a wide range of complex, challenging environments. In terms of, as we turn our eye
link |
to the idea of existential threats from superintelligence, do you think superintelligence
link |
has to exist in the physical world or can it be digital only? We think of our general intelligence
link |
as us humans, as an intelligence that's associated with the body that's able to interact with the
link |
world, that's able to affect the world directly with physically. I mean, digital only is perfectly
link |
fine, I think. I mean, it's physical in the sense that obviously the computers and the memories are
link |
physical. But it's capable to affect the world, sort of? Could be very strong, even if it has a
link |
limited set of actuators. If it can type text on the screen or something like that, that would be,
link |
I think, ample. So, in terms of the concerns of existential threat of AI, how can an AI system
link |
that's in the digital world have existential risk? What are the attack vectors for a digital system?
link |
Well, I mean, I guess maybe to take one step back. I should emphasize that I also think there's this
link |
huge positive potential from machine intelligence, including superintelligence. I want to stress that
link |
because some of my writing has focused on what can go wrong. When I wrote the book,
link |
Superintelligence, at that point, I felt that there was a kind of neglect of what would happen
link |
if AI succeeds. And in particular, a need to get a more granular understanding of where the pitfalls
link |
are so we can avoid them. I think that since the book came out in 2014, there has been a much
link |
wider recognition of that. And a number of research groups are now actually working on
link |
developing, say, AI alignment techniques and so on and so forth. So, yeah, I think now it's
link |
important to make sure we bring back onto the table the upside as well.
link |
And there's a little bit of a neglect now on the upside, which is, I mean, if you look at,
link |
talking to a friend, if you look at the amount of information that is available,
link |
or people talking, or people being excited about the positive possibilities of general
link |
intelligence, that's not, it's far outnumbered by the negative possibilities in terms of our
link |
public discourse. Possibly, yeah, it's hard to measure. What are, can you look at that for a
link |
little bit? What are some, to you, possible big positive impacts of general intelligence,
link |
superintelligence? Well, I mean, super, because I tend to
link |
also want to distinguish these two different contexts of thinking about AI and AI impacts,
link |
the kind of near term and long term, if you want, both of which I think are legitimate things to
link |
think about. And people should, you know, discuss both of them, but they are different and they
link |
often get mixed up. And then, then I get, you get confusion. I think you get simultaneously,
link |
like maybe an overhyping of the near term and an underhyping of the long term. And so I think as
link |
long as we keep them apart, we can have like two good conversations, but, or we can mix them
link |
together and have one bad conversation. Can you clarify just the two things we're talking about,
link |
the near term and the long term? Yeah, and what are the distinctions? Well, it's a, it's a blur
link |
distinction. But say the things I wrote about in this book, superintelligence, long term,
link |
things people are worrying about today with, I don't know, algorithmic discrimination or even
link |
things self driving cars and drones and stuff, more near term. And then, of course,
link |
you could imagine some medium term where they kind of overlap and one evolves into the other.
link |
But anyway, I think both, yeah, the issues look kind of somewhat different depending on
link |
which of these contexts. So I think it would be nice if we can talk about the long term
link |
and think about a positive impact or a better world because of the existence of the long
link |
term superintelligence. Do you have views of such a world? Yeah, I mean, I guess it's a little
link |
hard to articulate because it seems obvious that the world has a lot of problems as it currently
link |
stands. And it's hard to think of any one of those which it wouldn't be useful to have a
link |
like a friendly aligned superintelligence working on. So from health to the economic system to be
link |
able to sort of improve the investment and trade and foreign policy decisions, all that kind of
link |
stuff. All that kind of stuff and a lot more. I mean, what's the killer app? Well, I don't think
link |
there is one. I think AI, especially artificial general intelligence is really the ultimate
link |
general purpose technology. So it's not that there is this one problem, this one area where it will
link |
have a big impact. But if and when it succeeds, it will really apply across the board in all fields
link |
where human creativity and intelligence and problem solving is useful, which is pretty much
link |
all fields, right? The thing that it would do is give us a lot more control over nature.
link |
It wouldn't automatically solve the problems that arise from conflict between humans,
link |
fundamentally political problems. Some subset of those might go away if you just had more
link |
resources and cooler tech, but some subset would require coordination that is not automatically
link |
achieved just by having more technical capability. But anything that's not of that sort, I think
link |
you just get like an enormous boost with this kind of cognitive technology once it goes all
link |
the way. Now, again, that doesn't mean I'm like thinking, oh, people don't recognize what's possible
link |
with current technology. And like sometimes things get overhyped. But I mean, those are
link |
perfectly consistent views to hold the ultimate potential being enormous. And then it's a very
link |
different question of how far are we from that? Or what can we do with near term technology?
link |
Yeah. So what's your intuition about the idea of intelligence explosion? So there's this,
link |
you know, when you start to think about that leap from the near term to the long term,
link |
the natural inclination, like for me, sort of building machine learning systems today,
link |
it seems like it's a lot of work to get the general intelligence. But there's some intuition
link |
of exponential growth of exponential improvement of intelligence explosion. Can you maybe
link |
try to elucidate, to try to talk about what's your intuition about the possibility of
link |
intelligence explosion, that it won't be this gradual slow process that might be a phase shift?
link |
Yeah, I think it's, we don't know how explosive it will be. I think for what it's worth,
link |
it seems fairly likely to me that at some point there will be some intelligence
link |
explosion, like some period of time where progress in AI becomes extremely rapid.
link |
Roughly in the area where you might say it's kind of humanish equivalent in
link |
core cognitive faculties, that the concept of human equivalent starts to break down when
link |
you look too closely at it. And just how explosive does something have to be for it to
link |
be called an intelligence explosion? Like does it have to be like overnight literally,
link |
or a few years? But overall, I guess if you plotted the opinions of different people in
link |
the world, I guess I would be somewhat more probability towards the intelligence explosion
link |
scenario than probably the average AI researcher, I guess.
link |
So, and then the other part of the intelligence explosion, or just forget explosion, just progress,
link |
is once you achieve that gray area of human level intelligence, is it obvious to you that
link |
we should be able to proceed beyond it to get to super intelligence?
link |
Yeah, that seems, I mean, as much as any of these things can be obvious, given we've never had one,
link |
people have different views, smart people have different views, it's like some
link |
some degree of uncertainty that always remains for any big futuristic philosophical grand question
link |
that just we realize humans are fallible, especially about these things.
link |
But it does seem as far as I'm judging things based on my own impressions that it seems very
link |
unlikely that there would be a ceiling at or near human cognitive capacity.
link |
But that's such a, I don't know, that's such a special moment. It says both terrifying and
link |
exciting to create a system that's beyond our intelligence. So maybe you can step back and
link |
say, like, how does that possibility make you feel that we can create something,
link |
it feels like there's a line beyond which it steps, it'll be able to all smart you.
link |
And therefore, it feels like a step where we lose control.
link |
Well, I don't think the latter follows, that is, you could imagine, and in fact, this is what
link |
a number of people are working towards, making sure that we could ultimately
link |
project higher levels of problem solving ability while still making sure that they are aligned,
link |
like they are in the service of human values. I mean, so losing control, I think,
link |
is not a given that that would happen. I asked how it makes me feel. I mean, to some extent,
link |
I've lived with this for so long, since as long as I can remember being an adult or even a teenager,
link |
it seemed to me obvious that at some point, AI will succeed. And so I actually misspoke,
link |
I didn't mean control. I meant, because the control problem isn't an interesting thing.
link |
And I think the hope is, at least we should be able to maintain control over systems that are
link |
smarter than us. But we do lose our specialness. It sort of will lose our place as the smartest,
link |
coolest thing on earth. And there's an ego involved with that, that humans are very good at
link |
dealing with. I mean, I value my intelligence as a human being. It seems like a big transformative
link |
step to realize there's something out there that's more intelligent. I mean, you don't see that
link |
as such a fundamentally... Well, yeah, I think, yes, a lot. I think it would be small. I mean,
link |
I think there are already a lot of things out there that are... I mean, certainly if you think
link |
the universe is big, there's going to be other civilizations that already have super intelligences
link |
or that just naturally have brains the size of beach balls and are completely leaving us in the
link |
dust. And we haven't come face to face with this. We haven't come face to face. But I mean, that's
link |
an open question. What would happen in a kind of post human world? Like how much day to day would
link |
these super intelligences be involved in the lives of ordinary... I mean, you could imagine
link |
some scenario where it would be more like a background thing that would help protect against
link |
some things. But you wouldn't... They wouldn't be this intrusive kind of making you feel bad by
link |
like making clever jokes on your experience. But there's like all sorts of things that maybe in
link |
the human context would feel awkward about that. You wouldn't want to be the dumbest kid in your
link |
class, everybody picks it. Like a lot of those things maybe you need to abstract away from
link |
if you're thinking about this context where we have infrastructure that is in some sense
link |
beyond any of our old humans. I mean, it's a little bit like, say, the scientific community
link |
as a whole. If you think of that as a mind, it's a little bit of metaphor. But I mean,
link |
obviously it's got to be like way more capacious than any individual. So in some sense, there is this
link |
mind like thing already out there. That's just vastly more intelligent than any individual is.
link |
And we think, okay, that's... You just accept that as a fact. That's the basic fabric of our
link |
existence. There's a super intelligent thing. Yeah, you get used to a lot of... I mean,
link |
there's already Google and Twitter and Facebook, these recommender systems that are the basic
link |
fabric of our... I could see them becoming... I mean, do you think of the collective intelligence
link |
of these systems as already perhaps reaching superintelligence level? Well, I mean, so here
link |
it comes to the concept of intelligence and the scale and what human level means.
link |
The kind of vagueness and the determinacy of those concepts starts to
link |
dominate how you would answer that question. So, say the Google search engine has a very high
link |
capacity of a certain kind, like retrieving... Remembering and retrieving information,
link |
particularly like text or images that are... You have a kind of string, a word string key.
link |
Obviously, superhuman at that. But a vast set of other things it can't even do at all,
link |
not just not do well. So you have these current AI systems that are superhuman in some limited
link |
domain and then radically subhuman in all other domains. Same with a chess...
link |
A simple computer that can multiply really large numbers, right? So it's going to have this one
link |
spike of superintelligence and then a kind of zero level of capability across all other cognitive
link |
fields. Yeah, I don't necessarily think the generalness... I mean, I'm not so attached with it,
link |
but it's a gray area and it's a feeling, but to me, alpha zero is somehow much more intelligent,
link |
much, much more intelligent than D blue. And to say which domain... Well, you could say, well,
link |
these are both just board games. They're both just able to play board games. Who cares if they're
link |
going to do better or not? But there's something about the learning and the self play that makes
link |
it crosses over into that land of intelligence that doesn't necessarily need to be general.
link |
And the same way Google is much closer to D blue currently in terms of its search engine than it
link |
is to sort of the alpha zero. And the moment it becomes... And the moment these recommender systems
link |
really become more like alpha zero, but being able to learn a lot without the constraints of
link |
being heavily constrained by human interaction, that seems like a special moment in time.
link |
I mean, certainly learning ability seems to be an important facet of general intelligence,
link |
that you can take some new domain that you haven't seen before and you weren't specifically
link |
preprogrammed for and then figure out what's going on there and eventually become really good at it.
link |
So that's something alpha zero has much more of than D blue had. And in fact, I mean, systems
link |
like alpha zero can learn not just go, but other, in fact, probably beat D blue in chess and so
link |
forth. So you do see this general and it matches the intuition. We feel it's more intelligent.
link |
And it also has more of this general purpose learning ability. And if we get systems that
link |
have even more general purpose learning ability, it might also trigger an even stronger intuition
link |
that they're actually starting to get smart. So if you were to pick a future, what do you think a
link |
utopia looks like with AGI systems? Is it the neural link brain computer interface world where
link |
we're kind of really closely interlinked with AI systems? Is it possibly where AGI systems replace
link |
us completely while maintaining the values and the consciousness? Is it something like it's a
link |
completely invisible fabric? Like you mentioned a society where just aids and a lot of stuff that
link |
we do like curing diseases and so on. What is utopia if you get to pick? Yeah, I mean, it is a good
link |
question and a deep and difficult one. I'm quite interested in it. I don't have all the answers
link |
yet, but might never have. But I think there are some different observations one can make. One is if
link |
this scenario actually did come to pass, it would open up this vast space of possible modes of
link |
being. On one hand, material and resource constraints would just be expanded dramatically.
link |
So there would be a lot of a big pie, let's say. Also, it would enable us to do things
link |
including to ourselves or like that, it would just open up this much larger design space
link |
and option space than we have ever had access to in human history. So I think two things follow
link |
from that. One is that we probably would need to make a fairly fundamental rethink of what
link |
ultimately we value. Think things through more from first principles. The context would be so
link |
different from the familiar that we could have just take what we've always been doing and then
link |
oh, well, we have this cleaning robot that cleans the dishes in the sink and a few other small
link |
things. I think we would have to go back to first principles. So even from the individual
link |
level, go back to the first principles of what is the meaning of life, what is happiness,
link |
what is fulfillment. And then also connected to this large space of resources is that it
link |
would be possible and I think something we should aim for is to do well by the lights
link |
of more than one value system. That is, we wouldn't have to choose only one value criterion and say
link |
we're going to do something that scores really high on the metric of, say, hedonism. And then it's
link |
like a zero by other criteria, like kind of wireheaded brains in a vat. And it's like a lot
link |
of pleasure. That's good. But then like no beauty, no achievement like that. I think to some
link |
significant, not unlimited sense, but the significant sense, it would be possible to do
link |
very well by many criteria. Like maybe you could get like 98% of the best according to several
link |
criteria at the same time, given this great expansion of the option space. And so have
link |
competing value systems, competing criteria as sort of forever, just like our Democrat versus
link |
Republican, there seems to be this always multiple parties that are useful for our progress in society,
link |
even though it might seem dysfunctional inside the moment, but having the multiple value systems
link |
seems to be beneficial for, I guess, a balance of power. So that's not exactly what I have in mind,
link |
that it's, well, although it may be in an indirect way, it is. But that if you had the chance to do
link |
something that scored well on several different metrics, our first instinct should be to do that
link |
rather than immediately leap to the thing, which ones of these value systems are we going to screw
link |
over? Let's first try to do very well by all of them. Then it might be that you can't get 100%
link |
of all, and you would have to then have the hard conversation about which one will only get 97%.
link |
There you go. There's my cynicism that all of existence is always a tradeoff. But you say,
link |
maybe it's not such a bad tradeoff. Let's first try that. Well, this would be a distinctive context
link |
in which at least some of the constraints would be removed. There's probably still be tradeoffs in
link |
the end. It's just that we should first make sure we at least take advantage of this abundance.
link |
In terms of thinking about this, one should think in this kind of frame of mind of generosity
link |
and inclusiveness to different value systems and see how far one can get there first.
link |
And I think one could do something that would be very good according to many different criteria.
link |
We talked about AGI fundamentally transforming the value system of our existence, the meaning of
link |
life. But today, what do you think is the meaning of life? What are you the silliest or perhaps the
link |
biggest question? What's the meaning of life? What's the meaning of existence? What gives
link |
your life fulfillment, purpose, happiness, meaning? Yeah, I think these are, I guess, a bunch of
link |
different related questions in there that one can ask. Happiness, meaning,
link |
yeah. You could imagine somebody getting a lot of happiness from something that
link |
they didn't think was meaningful. Mindless watching reruns of some television series,
link |
waiting junk food, maybe some people that gives pleasure, but they wouldn't think
link |
it had a lot of meaning. Whereas, conversely, something that might be quite loaded with meaning
link |
might not be very fun always. Some difficult achievement that really helps a lot of people
link |
maybe requires self sacrifice and hard work. And so these things can, I think, come apart,
link |
which is something to bear in mind also when you're thinking about these utopia questions that
link |
you might actually start to do some constructive thinking about that. You might have to isolate
link |
it and distinguish these different kinds of things that might be valuable in different ways.
link |
Make sure you can sort of clearly perceive each one of them, and then you can think about how
link |
you can combine them. And just as you said, hopefully come up with a way to maximize all of
link |
them together. Yeah, or at least get, I mean, maximize or get a very high score on a wide
link |
range of them, even if not literally all. You can always come up with values that are
link |
exactly opposed to one another, right? But I think for many values, they're kind of opposed with,
link |
if you place them within a certain dimensionality of your space, like there are shapes that are kind
link |
of, you can't untangle like in a given dimensionality. But if you start adding dimensions, then it
link |
might in many cases just be that they are easy to pull apart. And you could. So we'll see how
link |
much space there is for that. But I think that there could be a lot in this context of radical
link |
abundance, if ever we get to that. I don't think there's a better way to end it, Nick. You've
link |
influenced a huge number of people to work on what could very well be the most important
link |
problems of our time. So it's a huge honor. Thank you so much for talking to me. Well,
link |
thank you for coming by, Lex. That was fun. Thank you. Thanks for listening to this conversation
link |
with Nick Bostrom. And thank you to our presenting sponsor, Cash App. Please consider supporting
link |
the podcast by downloading Cash App and using code Lex podcast. If you enjoy this podcast,
link |
subscribe on YouTube, review it with five stars on Apple podcast, support it on Patreon, or simply
link |
connect with me on Twitter at Lex Friedman. And now let me leave you with some words from Nick
link |
Bostrom. Our approach to existential risks cannot be one of trial and error. There's no opportunity
link |
to learn from errors. The reactive approach, see what happens, limit damages and learn from experience
link |
is unworkable. Rather, we must take a proactive approach. This requires foresight to anticipate
link |
new types of threats and a willingness to take decisive preventative action and to bear the
link |
costs, moral and economic of such actions. Thank you for listening and hope to see you next time.