The following is a conversation with Max Tagmark, his second time in the podcast.

In fact, the previous conversation was episode number one of this very podcast.

He is a physicist and artificial intelligence researcher at MIT, co founder

of the Future of Life Institute and author of Life 3.0, being human

in the age of artificial intelligence.

He's also the head of a bunch of other huge fascinating projects and has

written a lot of different things that you should definitely check out.

He has been one of the key humans who has been outspoken about long term

existential risks of AI and also its exciting possibilities and solutions

to real world problems, most recently at the intersection of AI and physics.

And also in reengineering the algorithms that divide us by controlling

the information we see and thereby creating bubbles and all other kinds

of complex social phenomena that we see today.

In general, he's one of the most passionate and brilliant people I

have the fortune of knowing.

I hope to talk to him many more times on this podcast in the future.

Quick mention of our sponsors, the Jordan Harbinger Show, Four

Sigmatic Mushroom Coffee, BetterHelp Online Therapy and ExpressVPN.

So the choice is wisdom, caffeine, sanity or privacy.

Choose wisely, my friends, and if you wish, click the sponsor links

below to get a discount at the support this podcast.

As a side note, let me say that much of the researchers in the machine

learning and artificial intelligence communities do not spend much time

thinking deeply about existential risks of AI.

Because our current algorithms are seen as useful but dumb, it's difficult

to imagine how they may become destructive to the fabric of human

civilization in the foreseeable future.

I understand this mindset, but it's very troublesome to me.

This is both a dangerous and uninspiring perspective, reminiscent of

the lobster sitting in a pot of lukewarm water that a minute ago was cold.

I feel a kinship with this lobster.

I believe that already the algorithms that drive our interaction on social

media have an intelligence and power that far outstrip the intelligence

and power of any one human being.

Now really is the time to think about this, to define the trajectory

of the interplay of technology and human beings in our society.

I think that the future of human civilization very well may be at

stake over this very question of the role of artificial intelligence

in our society.

If you enjoy this thing, subscribe on YouTube, review it on Apple

Podcasts, follow on Spotify, support on Patreon, or connect with me on

Twitter, Alex Friedman.

And now, here's my conversation with Max Tagmark.

So people might not know this, but you were actually episode number

one of this podcast just a couple of years ago, and now we're back.

And it so happens that a lot of exciting things happened in both physics

and artificial intelligence, both fields that you're super passionate about.

Can we try to catch up to some of the exciting things happening in artificial

intelligence, especially in the context of the way it's cracking, open the

different problems of the sciences?

Yeah, I'd love to, especially now as we start 2021 here, it's a really fun

time to think about what were the biggest breakthroughs in AI.

Not the ones necessarily the media wrote about, but they really matter.

And what does that mean for our ability to do better science?

What does it mean for our ability to do better science?

To help people around the world?

And what does it mean for new problems that they could cause if we're not

smart enough to avoid them?

So, you know, what do we learn basically from this?

Yes, absolutely.

So one of the amazing things you're part of is the AI Institute for

Artificial Intelligence and Fundamental Interactions.

What's up with this institute?

What are you working on?

What are you thinking about?

Well, the idea is something I'm very on fire with, which is basically AI

meets physics.

And, you know, it's been almost five years now since I shifted my own MIT

research from physics to machine learning.

And in the beginning, I noticed a lot of my colleagues, even though they

were polite about it, well, I kind of, what is Max doing?

What is this weird stuff?

He's lost his mind.

But then, but then gradually, I, together with some colleagues, were

able to persuade more and more of the other professors in our physics

department to get interested in this.

And now we got this amazing NSF center, so 20 million bucks for the next

five years, MIT and a bunch of neighboring universities here also.

And I noticed now those colleagues who were looking at me, funny, have

stopped asking what the point is of this, because it's becoming more clear.

And I really believe that, of course, AI can help physics a lot to do

better physics, but physics can also help AI a lot, both by building better

hardware.

My colleague, Martin Solzacic, for example, is working on an optical chip for

much faster machine learning, where the computation is done, not by moving

electrons around, but by moving photons around, dramatically less energy

use, faster, better.

We can also help AI a lot, I think, by having a different set of tools and a

different, maybe more audacious attitude.

You know, AI has, to a significant extent, been an engineering discipline,

where you're just trying to make things that work.

And being less, more interested in maybe selling them than in figuring out

exactly how they work and proving theorems about that they will always work.

Contrast that with physics, you know, when Elon Musk sends a rocket to the

International Space Station, they didn't just train with machine learning,

oh, let's fire it a little bit left, more to the left, a bit more to the right,

so that also missed, let's try here, no, you know, we figured out Newton's

laws of gravitation and other things and got a really deep fundamental

understanding, and that's what gives us such confidence in rockets.

And my vision is that in the future, all machine learning systems that

actually have impact on people's lives will be understood at a really,

really deep level, right, so we trust them, not because some sales rep told

us to, but because they've earned our trust, and really safety critical

things even prove that they will always do what we expect them to do.

That's very much the physics mindset, so it's interesting if you look at

big breakthroughs that have happened in machine learning this year, you know,

from dancing robots, you know, is pretty fantastic, not just because it's cool,

but if you just think about not that many years ago, this YouTube video at

this DARPA challenge where the MIT robot comes out of the car and face

plants, how far we've come in just a few years.

Similarly, Alpha Fold 2, you know, crushing the protein folding problem,

we can talk more about implications for medical research and stuff,

but hey, you know, that's huge progress.

You can look at the GPT3, they can spout off English texts,

which sometimes really, really blows you away.

You can look at the Google, at DeepMind's Mu Zero,

which doesn't just kick our butt and go and chest and chogi,

but also in all these Atari games, and you don't even have to teach it

the rules now.

You know, what all of those have in common is besides being powerful is

we don't fully understand how they work.

And that's fine if it's just some dancing robots,

and the worst thing that can happen is they face plant, right?

Or if they're playing Go, and the worst thing that can happen is

that they make a bad move and lose the game, right?

It's less fine if that's what's controlling your self driving car

or your nuclear power plant.

And we've seen already that even though Hollywood had all these movies

where they try to make us worry about the wrong things like machines

turning evil, the actual bad things that have happened with automation

have not been machines turning evil.

They've been caused by over trust in things we didn't understand

as well as we thought we did, right?

Even very simple automated systems like what Boeing put into the 737 Max, right?

Yes.

Killed a lot of people.

Was it that that little simple system was evil?

Of course not, but we didn't understand it

as well as we should have, right?

And we trusted without understanding.

Exactly.

And over trust.

We didn't even understand that we didn't understand, right?

The humility is really at the core of being a scientist.

I think step one, if you want to be a scientist,

is don't ever fool yourself into thinking you understand things

when you actually don't, right?

That's probably good advice for humans in general.

I think humility in general can do us good.

In science, it's so spectacular.

Why did we have the wrong theory of gravity ever from Aristotle

onward and close to Galileo's time?

Why would we believe something so dumb

as that if I throw this water bottle,

it's going to go up with constant speed

until it realizes that its natural motion is down.

It changes its mind.

Because people just kind of assumed

Aristotle was right.

He's an authority.

We understand that.

Why did we believe things like that the sun is going around the Earth?

Why did we believe that time flows at the same rate

for everyone until Einstein?

Same exact mistake over and over again.

We just weren't humble enough to acknowledge

that we actually didn't know for sure.

We assumed we knew.

So we didn't discover the truth

because we assumed there was nothing there to be discovered, right?

There was something to be discovered about the 737 Max.

And if you had been a bit more suspicious

and tested it better, we would have found it.

And it's the same thing with most harm

that's been done by automation so far, I would say.

Did you hear of a company called Night Capital?

No.

So good.

That means you didn't invest in them earlier.

They deployed this automated rating system.

Yes.

All nice and shiny.

They didn't understand it as well as they thought.

And it went about losing 10 million bucks per minute

for 44 minutes straight.

No.

Until someone presumably was like, oh, no, shut this up.

You know, was it evil?

No.

It was, again, misplaced trust,

something they didn't fully understand, right?

And there have been so many,

even when people have been killed by robots,

it's just quite rare still.

But in factory accidents,

it's in every single case been not malice,

just that the robot didn't understand that,

hey, a human is different from an auto part or whatever.

So this is where I think there's so much opportunity

for a physics approach,

where you just aim for a higher level of understanding.

And if you look at all these systems that we talked about

from reinforcement learning systems and dancing robots

to all these neural networks that power GPT3

and go playing software stuff,

they're all basically black boxes,

much like not so different from if you teach a human something,

you have no idea how their brain works, right?

Except the human brain at least has been error corrected

during many, many centuries of evolution

in a way that some of these systems have not, right?

And my MIT research is entirely focused on

demystifying this black box.

Intelligible intelligence is my slogan.

That's a good line.

Intelligible intelligence.

Yeah, it's not that we shouldn't settle for something

that seems intelligent,

but it should be intelligible

so that we actually trust it because we understand it, right?

Like, again, Elon trusts his rockets

because he understands Newton's laws

and thrusts and how everything works.

And let me tell you,

can I tell you why I'm optimistic about this?

Yes.

I think we've made a bit of a mistake

where some people still think

that somehow we're never going to understand neural networks.

And we're just going to have to learn to live with this.

It's this very powerful black box.

Basically, for those who haven't spent time

building their own,

it's super simple what happens inside.

You send in a long list of numbers,

and then you do a bunch of operations on them,

multiply by matrices, et cetera, et cetera,

and some other numbers come out.

That's the output of it.

And then there are a bunch of knobs you can tune.

And when you change them,

it affects the computation, the input output relation.

And then you just give the computer some definition of good,

and it keeps optimizing these knobs

until it performs as good as possible.

And often you go, like, wow, that's really good.

This robot can dance.

Or this machine is beating me at chest now.

And in the end, you have something,

which even though you can look inside it,

you have very little idea of how it works.

You can print out tables of all the millions of parameters in there.

Is it crystal clear now how it's working?

And of course not, right?

Many of my colleagues seem willing to settle for that.

And I'm like, no.

That's like the halfway point.

Some have even gone as far as sort of guessing

that the mystery, the inscrutability of this

is where some of the power comes from

and some sort of mysticism.

I think that's total nonsense.

I think the real power of neural networks

comes not from inscrutability,

but from differentiability.

And what I mean by that is simply that

the output changes only smoothly

if you tweak your knobs.

And then you can use all these powerful methods

we have for optimization in science.

We can just tweak them a little bit

and see, did that get better or worse?

That's the fundamental idea of machine learning,

that the machine itself can keep optimizing

until it gets better.

Suppose you wrote this algorithm instead

in Python or some other programming language.

And then what the knobs did was

they just changed random letters in your code.

Now it would just epically fail, right?

You change one thing and instead of saying print,

it says synth, syntax error.

You don't even know, was that for the better

or for the worse, right?

This to me is, this is what I believe

is the fundamental power of neural networks.

Just to clarify, the changing the different letters

in a program would not be a differentiable process.

It would make it an invalid program typically

and then you wouldn't even know

if you changed more letters,

if it would make it work again, right?

So that's the magic of neural networks,

the inscrubility.

The differentiability, that every setting

of the parameters is a program

and you can tell is it better or worse, right?

So you don't like the poetry of the mystery

of neural networks as the source of its power?

I generally like poetry, but...

Not in this case.

It's so misleading and above all,

it shortchanges us, it makes us underestimate

the good things we can accomplish

because so what we've been doing in my group

is basically step one,

train the mysterious neural network

to do something well.

And then step two, do some additional AI techniques

to see if we can now transform this black box

into something equally intelligent

that you can actually understand.

So for example, I'll give you one example

of this AI Feynman project that we just published, right?

So we took the 100 most famous

or complicated equations

from one of my favorite physics textbooks,

in fact the one that got me into physics

in the first place, the Feynman lectures on physics.

And so you have a formula, you know,

maybe it has what goes into the formula

as six different variables

and then what comes out as one.

So then you can make like a giant Excel spreadsheet

with seven columns.

You put in just random numbers for the six columns

for those six input variables

and then you calculate with the formula

the seventh column, the output.

So maybe it's like the force equals

in the last column some function of the other.

And now the task is, okay,

if I don't tell you what the formula was,

can you figure that out

from looking at my spreadsheet that I gave you?

This problem is called symbolic regression.

If I tell you that the formula

is what we call a linear formula.

So it's just that the output is

some sum of all the things

input at the time, some constants.

That's the famous easy problem we can solve.

We do it all the time in science and engineering.

But the general one,

if it's more complicated functions

with logarithms or cosines or other math,

it's a very, very hard one

and probably impossible to do fast in general

just because the number of formulas

with n symbols just grows exponentially.

Just like the number of passwords you can make

grow dramatically with length.

So we had this idea that

if you first have a neural network

that can actually approximate the formula,

you just train that even if you don't understand how it works,

that can be the first step

towards actually understanding how it works.

So that's what we do first.

And then we study that neural network now

and put in all sorts of other data

that wasn't in the original training data

and use that to discover

simplifying properties of the formula.

And that lets us break it apart

often into many simpler pieces

in a kind of divide and conquer approach.

So we were able to solve all of those 100 formulas,

discover them automatically,

plus a whole bunch of other ones.

But it's actually kind of humbling to say

that this code, which anyone who wants now

is listening to this can type pip install

AI Feynman on the computer and run it.

It can actually do what Johannes Kepler

spent four years doing when he stared at Mars data

until he was like, finally, Eureka, this is an ellipse.

This will do it automatically for you in one hour, right?

Or Max Planck.

He was looking at how much radiation comes out

at different wavelengths from a hot object

and discovered the famous black body formula.

This discovers it automatically.

I'm actually excited about

seeing if we can discover not just old formulas again,

but new formulas that no one has seen before.

And do you like this process of using kind of a neural network

to find some basic insights

and then dissecting the neural network

and gain the final so that that's in that way

you've forcing the explainability issue,

really trying to analyze the neural network

for the things it knows in order to come up

with the final beautiful simple theory

underlying the initial system that you were looking at.

I love that.

And the reason I'm so optimistic that it can be generalized

is because that's exactly what we do as human scientists.

Think of Galileo whom we mentioned, right?

I bet when he was a little kid,

if his dad threw him an apple, he would catch it.

Why?

Because he had a neural network in his brain

that he had trained to predict the parabolic orbit

of apples that are thrown under gravity.

If you throw a tennis ball to a dog,

it also has this same ability of deep learning

to figure out how the ball is going to move and catch it.

But Galileo went one step further when he got older.

He went back and was like, wait a minute.

I can write down a formula for this.

Y equals X squared, a parabola.

And he helped revolutionize physics as we know it, right?

So there was a basic neural network in there from childhood

that captured the experiences of observing

different kinds of trajectories.

And then he was able to go back in with another extra little neural network

and analyze all those experiences and be like, wait a minute.

There's a deeper rule here.

Exactly. He was able to distill out in symbolic form

what that complicated black box neural network was doing.

Not only did he, the formula he got,

ultimately become more accurate.

And similarly, this is how Newton got Newton's laws,

which is why Elon can send rockets to the space station now, right?

So it's not only more accurate, but it's also simpler, much simpler.

And it's so simple that we can actually describe it to our friends

and each other, right?

We've talked about it just in the context of physics now,

but hey, isn't this what we're doing when we're talking to each other also?

We go around with our neural networks just like dogs and cats

and chipmunks and blue jays.

And we experience things in the world.

But then we humans do this additional step on top of that,

where we then distill out certain high level knowledge

that we've extracted from this in a way that can communicate it to each other

in a symbolic form in English in this case, right?

So if we can do it and we believe that we are information processing entities,

then we should be able to make machine learning that does it also.

Well, do you think the entire thing could be learning?

Because this dissection process, like for AI Feynman,

the secondary stage feels like something like reasoning.

And the initial step feels like more like the more basic kind of differentiable learning.

Do you think the whole thing could be differentiable learning?

Do you think the whole thing could be basically neural networks on top of each other?

It's like turtles all the way down.

Could it be neural networks all the way down?

I mean, that's a really interesting question.

We know that in your case, it is neural networks all the way down

because that's all you'll have in your skull as a bunch of neurons doing their thing, right?

But if you ask the question more generally,

what algorithms are being used in your brain,

I think it's super interesting to compare.

I think we've gotten a little bit backwards historically

because we humans first discovered good old fashioned AI,

the logic based AI that we often call GOFI for good old fashioned AI.

And then more recently, we did machine learning

because it required bigger computers, so we had to discover it later.

So we think of machine learning with neural networks as the modern thing

and the logic based AI as the old fashioned thing.

But if you look at evolution on Earth, right,

it's actually been the other way around.

I would say that, for example, an eagle has a better vision system

than I have using, and dogs are just as good at casting tennis balls as I am.

All this stuff which is done by training in neural network

and not interpreting it in words, you know,

is something so many of our animal friends can do, at least as well as us, right?

What is it that we humans can do that the chipmunks and the eagles cannot?

It's more to do with this logic based stuff, right,

where we can extract out information in symbols, in language,

and now even with equations if you're a scientist, right?

So basically what happened was first we built these computers

that could multiply numbers real fast and manipulate symbols

and we felt they were pretty dumb.

And then we made neural networks that can see as well as a cat can

and do a lot of this inscrutable black box neural networks.

What we humans can do also is put the two together in a useful way.

Yes, in our own brain.

Yes, in our own brain.

So if we ever want to get artificial general intelligence

that can do all jobs as well as humans can, right,

then that's what's going to be required to be able to combine the neural networks with symbolic.

Combine the old AI with a new AI in a good way.

We do it in our brains and there seems to be basically two strategies I see in industry now.

One scares the heebie jeebies out of me and the other one I find much more encouraging.

Can we break them apart? Which other two?

The one that scares the heebie jeebies out of me is this attitude

that we're just going to make ever bigger systems that we still don't understand

until they can be as smart as humans.

What could possibly go wrong?

I think it's just such a reckless thing to do and unfortunately,

and if we actually succeed as a species to build artificial general intelligence,

then we still have no clue how it works.

I think at least 50% chance we're going to be extinct before too long.

It's just going to be an utter epic own goal.

Plus that 44 minute losing money problem or like the paperclip problem

where we don't understand how it works and it's just in a matter of seconds runs away

in some kind of direction that's going to be very problematic.

Even long before you have to worry about the machines themselves

somehow deciding to do things and to us that we have to worry about people using machines.

They're short of AI, AGI and power to do bad things.

I mean, just take a moment and if anyone who's not worried particularly about advanced AI,

just take 10 seconds and just think about your least favorite leader on the planet right now.

Don't tell me who it is.

I want to keep this apolitical.

But just see the face in front of you, that person for 10 seconds.

Now imagine that that person has this incredibly powerful AI under their control

and can use it to impose their will on the whole planet.

How does that make you feel?

Yeah.

Can we break that apart just briefly?

For the 50% chance that we'll run into trouble with this approach,

do you see the bigger worry in that leader or humans using the system to do damage

or are you more worried and I think I'm in this camp

more worried about accidental unintentional destruction of everything.

So humans trying to do good and in a way where everyone agrees it's kind of good,

it's just they're trying to do good without understanding.

Because I think every evil leader in history thought they're, to some degree,

thought they're trying to do good.

Oh yeah.

I'm sure Hitler thought he was doing a good job.

I've been reading a lot about Stalin.

He legitimately thought that communism was good for the world

and that he was doing good.

I think Mao Zedong thought what he was doing with a great leap forward was good too.

I'm actually concerned about both of those.

Before, I promised to answer this in detail, but before we do that,

let me finish answering the first question,

because I told you that there were two different routes

we could get to artificial general intelligence and one scares the FPGVs out of me,

which is this one where we build something,

we just say bigger neural networks, ever more hardware,

and it's just trying to get more data and poof, now it's very powerful.

That, I think, is the most unsafe and reckless approach.

The alternative to that is the intelligible intelligence approach instead,

where we say neural networks is just a tool for the first step to get the intuition,

but then we're going to spend also serious resources on other AI techniques

for demystifying this black box and figuring out what it's actually doing

so we can convert it into something that's equally intelligent,

but that we actually understand what it's doing.

Maybe we can even prove theorems about it,

that this car here will never be hacked when it's driving,

because here is a proof.

There is a whole science of this, but it doesn't work for neural networks.

There are big black boxes, but it works well and certain other kinds of codes.

That approach, I think, is much more promising.

That's exactly why I'm working on it, frankly,

not just because I think it's cool for science,

but because I think the more we understand these systems,

the better the chances that we can make them do the things that are good for us

that are actually intended, not unintended.

You think it's possible to prove things about something as complicated as a neural network?

That's the hope?

Well, ideally, there's no reason there has to be a neural network in the end, either.

We discovered that Newton's laws of gravity with neural network in Newton's head,

but that's not the way it's programmed into the navigation system of Elon Musk's rocket anymore.

It's written in C++, or I don't know what language he uses exactly.

And then there are software tools called symbolic verification.

DARPA and the US military has done a lot of really great research on this,

because they really want to understand that when they build weapon systems,

they don't just go fire at random or malfunction, right?

And there's even a whole operating system called Cell 3 that's been developed by a DARPA grant

where you can actually mathematically prove that this thing can never be hacked.

Well, one day, I hope that will be something you can say about the OS that's running on our laptops, too,

as you know, but we're not there.

But I think we should be ambitious, frankly.

And if we can use machine learning to help do the proofs and so on as well, right,

then it's much easier to verify that a proof is correct than to come up with a proof in the first place.

That's really the core idea here.

If someone comes on your podcast and says they proved the Riemann hypothesis

or some sensational new theorem, it's much easier for someone else to take some smart math grad students

and check, oh, there's an error here on equation 5, or this really checks out

than it was to discover the proof.

Yeah, although some of those proofs are pretty complicated, but yes, it's still nevertheless much easier to verify the proof.

I love the optimism.

You know, even with the security of systems, there's a kind of cynicism that pervades people who think about this,

which is like, oh, it's hopeless.

I mean, in the same sense, exactly like you're saying when you own networks,

oh, it's hopeless to understand what's happening.

With security, people are just like, well, there's always going to be attack vectors

and waste to attack the system.

But you're right, we're just very new with these computational systems.

We're even new with these intelligence systems, and it's not out of the realm of possibility.

Just like people that understand the movement of the stars and the planets and so on.

Yeah.

It's entirely possible that within, hopefully soon, but it could be within 100 years,

we start to have an obvious laws of gravity about intelligence.

Yeah.

And God forbid, well, consciousness too, that one.

Agreed.

You know, I think, of course, if you're selling computers that get hacked a lot,

that's in your interest as a company that people think it's impossible to make it safe.

So, you know, but he's going to get the idea of suing you.

But I want to really inject optimism here.

It's absolutely possible to do much better than we're doing now.

And you know, your laptop does so much stuff.

You don't need the music player to be super safe in your future self driving car, right?

If someone hacks it and starts playing music, you don't like the world on end.

But what you can do is you can break out and say the drive computer that controls your safety

must be completely physically decoupled entirely from the entertainment system.

And it must physically be such that it can't take on over the air updates while you're driving.

And it can be, it can have, it's not that, it can have ultimately some operating system on it,

which is symbolically verified and proven that it's always going to do what it's supposed to do, right?

We can basically have, and companies should take that attitude too.

They should look at everything they do and say, what are the few systems in our company

that threaten the whole life of the company if they get hacked, you know,

and have the highest standards for them, and then they can save money

by going for the El Chippo poorly understood stuff for the rest, you know.

This is very feasible, I think.

And coming back to the bigger question about, that you worried about,

that there'll be unintentional failures, I think, there are two quite separate risks here, right?

We talked a lot about one of them, which is that the goals are noble of the human.

The human says, I want this airplane to not crash,

because this is not Muhammad Atta now flying the airplane, right?

And now there's this technical challenge of making sure that the autopilot

is actually going to behave as the pilot wants.

If you set that aside, there's also the separate question.

How do you make sure that the goals of the pilot are actually aligned with the goals of the passenger?

How do you make sure very much more broadly that if we can all agree as a species

that we would like things to kind of go well for humanity as a whole,

that the goals are aligned here, the alignment problem.

And yeah, there's been a lot of progress in the sense that there's suddenly huge amounts of research going on about it.

I'm very grateful to Elon Musk for giving us that money five years ago,

so we could launch the first research program on technical AI safety and alignment.

There's a lot of stuff happening.

I think we need to do more than just make sure little machines do always what their owners do.

That wouldn't have prevented September 11.

Muhammad Atta said, OK, autopilot, please fly into World Trade Center.

And it's like, OK, that even happened.

In a different situation, there was this depressed pilot named Andreas Lubitz,

who told his German wings passenger jet to fly into the Alps.

He just told the computer to change the altitude to 100 meters or something like that.

And you know what the computer said?

OK.

And it had the frigging topographical map of the Alps in there.

It had GPS, everything.

No one had bothered teaching it even the basic kindergarten ethics of, like, no.

We never want airplanes to fly into mountains under any circumstances.

And so we have to think beyond just the technical issues

and think about how do we align, in general, incentives on this planet for the greater good.

So starting with simple stuff like that, every airplane that has a computer in it

should be taught whatever kindergarten ethics it's smart enough to understand.

Like, no, don't fly into fixed objects if the pilot tells you to do so.

And then go on autopilot mode, send an email to the cops

and land at the latest airport, nearest airport.

Any car with a forward facing camera should just be programmed by the manufacturers

so that it will never accelerate into a human ever.

That would avoid things like the niece attack

and many horrible terrorist vehicle attacks where they deliberately did that, right?

There's not some sort of thing, oh, you know, US and China, different views.

No, there was not a single car manufacturer in the world who wanted the cars to do this.

They just hadn't thought to do the alignment.

And if you look at, more broadly, problems that happen on this planet,

the vast majority have to do a poor alignment.

I mean, think about, let's go back really big, because I know you're so good at that.

So long ago in evolution, we had these genes and they wanted to make copies of themselves.

That's really all they cared about.

So some genes said, hey, I'm going to build a brain on this body I'm in

so that I can get better at making copies to myself.

And then they decided for their benefit to get copied more,

to align your brain's incentives with their incentives.

So it didn't want you to starve to death.

So it gave you an incentive to eat.

And it wanted you to make copies of the genes.

So it gave you an incentive to fall in love and do all sorts of naughty things

to make copies of itself, right?

So that was successful value alignment done on the genes.

They created something more intelligent than themselves,

but they made sure to try to align the values.

But then something went a little bit wrong against the idea of what the genes wanted

because a lot of humans discovered, hey, we really like this business about sex

that the genes have made us enjoy, but we don't want to have babies right now.

So we're going to hack the genes and use birth control.

And I really feel like drinking a Coca Cola right now,

but I don't want to get a potbelly, so I'm going to drink Diet Coke.

We have all these things we've figured out because we're smarter than the genes,

how we can actually subvert their intentions.

So it's not surprising that we humans now, when we're in the role of these genes,

creating other nonhuman entities with a lot of power have to face the same exact challenge.

How do we make other powerful entities have incentives that are aligned with ours

so they won't hack them?

Corporations, for example, right?

We humans decided to create corporations because it can benefit us greatly.

Now all of a sudden there's a supermarket. I can go buy food there.

I don't have to hunt. Awesome.

And then to make sure that this corporation would do things that were good for us

and not bad for us, we created institutions to keep them in check.

Like if the local supermarket sells poisonous food,

then the owners of the supermarket have to spend some years reflecting behind bars, right?

So we created incentives to get to align them.

But of course, just like we were able to see through this thing,

well, birth control, if you're a powerful corporation,

you also have an incentive to try to hack the institutions that are supposed to govern you

because you ultimately as a corporation have an incentive to maximize your profit.

Just like you have an incentive to maximize the enjoyment your brain has, not for your genes.

So if they can figure out a way of bribing regulators, then they're going to do that.

In the US, we kind of caught on to that and made laws against corruption and bribery.

Then in the late 1800s, Teddy Roosevelt realized that,

no, we were still being kind of hacked because the Massachusetts railroad companies had like a bigger budget

than the state of Massachusetts and they were doing a lot of very corrupt stuff.

So he did the whole trust busting thing to try to align these other nonhuman entities,

the companies, again, more with the incentives of Americans as a whole.

It's not surprising though that this is a battle you have to keep fighting.

Now we have even larger companies than we ever had before.

And of course, they're going to try to, again, subvert the institutions.

Not because, you know, I think people make a mistake of getting all too black thinking about things in terms of good and evil.

Like arguing about whether corporations are good or evil or whether robots are good or evil.

A robot isn't good or evil. It's tool.

And you can use it for great things like robotic surgery or for bad things.

And a corporation also is a tool, of course.

And if you have good incentives to the corporation, it'll do great things like start a hospital or a grocery store.

If you have really bad incentives, then it's going to start maybe marketing addictive drugs to people and you'll have an opioid epidemic.

It's all about, we should not make a mistake of getting into some sort of fairytale, good, evil thing about corporations or robots.

We should focus on putting the right incentives in place.

My optimistic vision is that if we can do that, then we can really get good things.

We're not doing so great with that right now, either on AI, I think, or on other intelligent, nonhuman entities like big companies.

We just have a new secretary of defense.

There's going to start up now in the Biden administration who was an active member of the board of Raytheon.

I have nothing against Raytheon.

I'm not a pacifist, but there's an obvious conflict of interest if someone is in the job where they decide who they're going to contract with.

I think somehow we have, maybe we need another Teddy Roosevelt to come along again and say, hey, we want what's good for all Americans.

We need to go do some serious realigning again of the incentives that we're giving to these big companies.

Then we're going to be better off.

Naturally, with human beings, just like you beautifully described the history of this whole thing, it all started with the genes and they're probably pretty upset by all the unintended consequences that happened since.

It seems that it kind of works out.

It's in this collective intelligence that emerges at the different levels.

It seems to find, sometimes last minute, a way to realign the values or keep the values aligned.

It finds a way.

Different leaders, different humans pop up all over the place that reset the system.

Do you have an explanation why that is?

Or is that just survivor bias?

Also, is that somehow fundamentally different than with the AI systems where you're no longer dealing with something that was a direct, maybe companies are the same, a direct byproduct of the evolutionary process?

I think there is one thing which has changed.

That's why I'm not all optimistic. That's why I think there's about a 50% chance if we take the dumb route with artificial intelligence that humanity will be extinct in this century.

First, just the big picture.

Companies need to have the right incentives.

Even governments, right?

We used to have governments, usually there were just some king who was the king because his dad was the king.

Then there were some benefits of having this powerful kingdom or empire of any sort because then it could prevent a lot of local squabbles.

So at least everybody in that region would stop warring against each other.

Their incentives of different cities in the kingdom became more aligned.

That was the whole selling point.

Harari.

Harari has a beautiful piece on how empires were collaboration enablers.

Harari has invented money for that reason so we could have better alignment and trade even with people we didn't know.

This sort of stuff has been playing out since time immemorial.

What's changed is that it happens on ever larger scales.

Technology keeps getting better because science gets better.

So now we can communicate over larger distances, transport things faster over larger distances.

So the entities get ever bigger but our planet is not getting bigger anymore.

So in the past, you could have one experiment that just totally screwed up like Easter Island where they actually managed to have such poor alignment that when they went extinct, people there, there was no one else to come back and replace them.

If Elon Musk doesn't get us to Mars and then we go extinct on a global scale, then we're not coming back.

That's the fundamental difference.

And that's a mistake I would rather we don't make for that reason.

In the past, of course, history is full of fiascos, but it was never the whole planet.

And then, okay, now there's this nice uninhabited land here.

Some other people could move in and organize things better.

This is different.

The second thing which is also different is that technology gives us so much more empowerment both to do good things and also to screw up.

In the Stone Age, even if you had someone whose goals were really poorly aligned,

maybe he was really pissed off because his Stone Age girlfriend dumped him and he just wanted to kill as many people as he could.

How many could he really take out with a rock and a stick before he was overpowered?

Right, just handful, right?

Now, with today's technology, if we have an accidental nuclear war between Russia and the US,

which we almost have about a dozen times and then we have a nuclear winter,

it could take out 7 billion people or 6 billion people, we don't know.

So the scale of damage is bigger that we can do.

And if there's obviously no law of physics that says that technology will never get powerful enough that we could wipe out our species entirely,

that would just be fantasy to think that science is somehow doomed not to get more powerful than that, right?

And it's not at all unfeasible in our lifetime that someone could design a designer pandemic which spreads as easily as COVID,

but just basically kills everybody.

We already had smallpox, it killed one third of everybody who got it.

What do you think of the, here's an intuition, maybe it's completely naive and this optimistic intuition I have,

which it seems, and maybe it's a biased experience that I have,

but it seems like the most brilliant people I've met in my life all are really fundamentally good human beings.

And not like naive, good, like they really want to do good for the world in a way that well maybe is aligned to my sense of what good means.

And so I have a sense that the people that will be defining the very cutting edge of technology,

there will be much more of the ones that are doing good versus the ones that are doing evil.

So the race, I'm optimistic on us always like last minute coming up with a solution.

So if there's an engineered pandemic that has the capability to destroy most of the human civilization,

it feels like to me either leading up to that before or as it's going on,

there will be, we're able to rally the collective genius of the human species.

I could tell by your smile that you're at least some percentage doubtful,

but could that be a fundamental law of human nature that evolution only creates,

like karma is beneficial, good is beneficial and therefore will be alright?

I hope you're right.

I would really love it if you're right,

if there's some sort of law of nature that says that we always get lucky in the last second

because of karma, but I prefer not playing it so close and gambling on that.

And I think, in fact, I think it can be dangerous to have too strong faith in that

because it makes us complacent.

Like if someone tells you you never have to worry about your house burning down,

then you're not going to put in a smoke detector because why would you need to, right?

Even if it's sometimes very simple precautions, we don't take them.

If you're like, oh, the government is going to take care of everything for us.

I can always trust my politicians.

We abdicate our own responsibility.

I think it's a healthier attitude to say, yeah, maybe things will work out,

but maybe I'm actually going to have to myself step up and take responsibility.

And the stakes are so huge.

I mean, if we do this right, we can develop all this ever more powerful technology

and cure all diseases and create a future where humanity is healthy and wealthy

or not just the next election cycle, but like billions of years throughout our universe.

That's really worth working hard for and not just, you know, sitting and hoping

for some sort of fairytale karma.

Well, I just mean, so you're absolutely right.

From the perspective of the individual, like for me,

like the primary thing should be to take responsibility

and to build the solutions that your skill set allows to build.

Which is a lot.

I think we underestimate often very much how much good we can do.

If you or anyone listening to this is completely confident that our government

would do a perfect job on handling any future crisis with engineered pandemics

or future AI.

The one or two people out there.

On what actually happened in 2020.

Do you feel that government by and large around the world is handled flawlessly?

That's a really sad and disappointing reality that hopefully is a wake up call for everybody.

For the scientists, for the engineers, for the researchers and AI especially.

It was disappointing to see how inefficient we were at collecting the right amount of data

in a privacy preserving way and spreading that data

and utilizing that data to make decisions, all that kind of stuff.

I think when something bad happens to me, I made myself a promise many years ago

that I would not be a whiner.

So when something bad happens to me, of course it's just a process of disappointment.

But then I try to focus on what did I learn from this

that can make me a better person in the future.

And there's usually something to be learned when I fail.

And I think we should all ask ourselves, what can we learn from the pandemic

about how we can do better in the future?

And you mentioned there's a really good lesson.

We were not as resilient as we thought we were.

And we were not as prepared maybe as we wish we were.

You can even see very stark contrast around the planet.

South Korea, they have over 50 million people.

Do you know how many deaths they have from COVID last time I checked?

It's about 500.

Why is that?

Well, the short answer is that they had prepared.

They were incredibly quick, incredibly quick to get on it

with very rapid testing and contact tracing and so on,

which is why they never had more cases than they could contract trace effectively, right?

They even had to have the kind of big lockdowns we had in the West.

But the deeper answer to it's not just Koreans are just somehow better people.

The reason I think they were better prepared was because they had already had a pretty bad hit

from the SARS pandemic, which never became a pandemic.

Something like 17 years ago, I think.

So it was kind of a fresh memory that we need to be prepared for pandemics.

So they were, right?

So maybe this is a lesson here for all of us to draw from COVID

that rather than just wait for the next pandemic or the next problem

with AI getting out of control or anything else,

maybe we should just actually set aside a tiny fraction of our GDP

to have people very systematically do some horizon scanning

and say, okay, what are the things that could go wrong?

And let's do get out and see which are the more likely ones

and which are the ones that are actually actionable and then be prepared.

So one of the observations as one little ant slash human that I am of disappointment

is the political division over information that has been observed that I observed this year

that it seemed the discussion was less about sort of what happened

and understanding what happened deeply and more about there's different truths out there.

And it's like an argument, my truth is better than your truth.

And it's like red versus blue or different.

It was like this ridiculous discourse that doesn't seem to get at any kind of notion of the truth.

It's not like there's some kind of scientific process.

Even science got politicized in ways that's very heartbreaking to me.

You have an exciting project on the AI front of trying to rethink one of the mentioned corporations.

There's one of the other collective intelligence systems that have emerged

from this is social networks and just the spread of information on the internet,

our ability to share that information.

There's all different kinds of news sources and so on.

And so you said like that's from first principles.

Let's rethink how we think about the news, how we think about information.

Can you talk about this amazing effort that you're undertaking?

Oh, I'd love to.

But this has been my big COVID project has been nights and weekends on ever since the lockdown.

To segue into this, actually, let me come back to what you said earlier,

that you had this hope that in your experience, people who you felt were very talented,

often idealistic and wanted to do good.

Frankly, I feel the same about all people by and large.

There are always exceptions, but I think the vast majority of everybody,

regardless of education and whatnot, really are fundamentally good, right?

So how can it be that people still do so much nasty stuff?

I think it has everything to do with the information that we're given.

If you go into Sweden 500 years ago and you start telling all the farmers that those Danes in Denmark,

they're so terrible people and we have to invade them because they've done all these terrible things

that you can't fact check yourself.

A lot of people in Sweden did that.

And we've seen so much of this today in the world, both geopolitically,

where we are told that China is bad and Russia is bad and Venezuela is bad

and people in those countries are often told that we are bad.

And we also see it at a micro level, where people are told that,

oh, those who voted for the other party are bad people.

It's not just an intellectual disagreement, but they're bad people

and we're getting ever more divided.

And so how do you reconcile this with intrinsic goodness in people?

I think it's pretty obvious that it has again to do with this,

with information that we're fed and given, right?

We evolved to live in small groups where you might know 30 people in total, right?

So you then had a system that was quite good for assessing who you could trust

and who you could not.

And if someone told you that Joe there is a jerk,

but you had interacted with him yourself and seen him in action,

you would quickly realize maybe that that's actually not quite accurate, right?

But now that most people on the planet are people we've never met,

it's very important that we have a way of trusting information we're given.

So, okay, so where does the news project come in?

Well, throughout history, you can go read Machiavelli from the 1400s

and you'll see how already then there were busy manipulating people with propaganda and stuff.

Propaganda is not new at all.

And the incentives to manipulate people is just not new at all.

What is it that's new?

What's new is machine learning meets propaganda.

That's what's new.

That's why this has gotten so much worse.

Some people like to blame certain individuals like in my liberal university bubble,

many people blame Donald Trump and say it was his fault.

I see it differently.

I think Donald Trump just had this extreme skill at playing this game

in the machine learning algorithm age.

A game he couldn't have played 10 years ago.

So what's changed?

What's changed is, well, Facebook and Google and other companies.

I'm not a bad man, I think them.

I have a lot of friends who work for these companies, good people.

They deployed machine learning algorithms just to increase their profit a little bit

to just maximize the time people spent watching ads.

And they had totally underestimated how effective they were going to be.

This was, again, the black box, non intelligible intelligence.

They just noticed, oh, we're getting more ad revenue, great.

It took a long time until even realize why and how damaging this was for society.

Because, of course, what the machine learning figured out was

that the by far most effective way of gluing you to your little rectangle

was to show you things that triggered strong emotions, anger, et cetera, resentment.

And if it was true or not, it didn't really matter.

It was also easier to find stories that weren't true.

If you weren't limited, that's just a limitation to show people.

That's a very limiting fact.

And before long, we got these amazing filter bubbles on a scale we had never seen before.

A couple of this to the fact that also the online news media were so effective

that they killed a lot of print journalism.

There's less than half as many journalists now in America, I believe,

as there was a generation ago.

He just couldn't compete with the online advertising.

So, all of a sudden, most people are not getting even reading newspapers.

They get their news from social media.

And most people only get news in their little bubble.

So, along comes now some people like Donald Trump who figured out

among the first successful politicians to figure out how to really play this new game

and become very, very influential.

But I think Donald Trump took advantage of it.

He didn't create the fundamental conditions were created by machine learning

taking over the news media.

So, this is what motivated my little COVID project here.

I said before, machine learning and tech in general is not evil,

but it's also not good.

It's just a tool that you can use for good things or bad things.

And as it happens, machine learning and news was mainly used by the big players,

big tech, to manipulate people and to watch as many ads as possible,

which had this unintended consequence of really screwing up our democracy

and fragmenting it into filter bubbles.

So, I thought, well, machine learning algorithms are basically free.

They can run on your smartphone for free also if someone gives them away to you, right?

There's no reason why they only have to help the big guy to manipulate the little guy.

They can just as well help the little guy to see through all the manipulation attempts

from the big guy.

So, did this project, you can go to improvethenews.org.

The first thing we've built is this little news aggregator.

Looks a bit like Google News except it has these sliders on it

to help you break out of your filter bubble.

So, if you're reading, you can click click and go to your favorite topic.

And then, if you just slide the left right slider all the way over to the left.

There's two sliders, right?

Yeah.

There's the one, the most obvious one is the one that has left to right labeled on us.

You go to left, you get one set of articles, you go to the right,

you see a very different truth appearing.

Well, that's literally left and right on the political spectrum.

Yeah, so if you're reading about immigration, for example, it's very, very noticeable.

And I think step one, always if you want to not get manipulated,

it's just to be able to recognize the techniques people use.

So, it's very helpful to just see how they spin things on the two sides.

I think many people are under the misconception that the main problem is fake news.

It's not.

I had an amazing team of MIT students where we did an academic project,

used machine learning to detect the main kinds of bias over the summer.

Yes, of course, sometimes there's fake news where someone just claims something that's false, right?

Like, oh, Hillary Clinton just got divorced or something.

Yes.

But what we see much more of is actually just omissions.

If you go to, there's some stories which just won't be mentioned by the left or the right

because it doesn't suit their agenda.

And then they also mentioned other ones very, very, very much.

So, for example, we've had a number of stories about the Trump family's financial dealings.

And then there's been a bunch of stories about the Biden family's, Hunter Biden's financial dealings, right?

Surprise, surprise, they don't get equal coverage on the left and the right.

One side loves to cover the Biden, Hunter Biden's stuff.

And one side loves to cover the Trump, you can never guess which is which, right?

But the great news is if you're a normal American citizen and you dislike corruption in all its forms,

then slide, slide, you can just look at both sides and you'll see all those political corruption stories.

It's really liberating to just take in the both sides, the spin on both sides.

It somehow unlocks your mind to think on your own, to realize that, I don't know,

it's the same thing that was useful in the Soviet Union times for when everybody was much more aware that they're surrounded by propaganda.

That is so interesting what you're saying, actually.

So, Noam Chomsky used to be our MIT colleague once said that propaganda is to democracy.

What violence is to totalitarianism.

And what he means by that is if you have a really totalitarian government, you don't need propaganda.

People will do what you want them to do anyway out of fear, right?

But otherwise, you need propaganda.

So, I would say actually that the propaganda is much higher quality in democracies, much more believable.

And it's really striking when I talk to colleagues, science colleagues like from Russia and China and so on,

I notice they are actually much more aware of the propaganda in their own media

than many of my American colleagues are about the propaganda in Western media.

That's brilliant. That means the propaganda in the Western media is just better.

Yes, that's so brilliant.

Even the propaganda.

But once you realize that, you realize there's also something very optimistic there that you can do about it, right?

Because, first of all, omissions.

As long as there's no outright censorship, you can just look at both sides

and pretty quickly piece together a much more accurate idea of what's actually going on, right?

And develop a natural skepticism too.

Just an analytical scientific mind about what you're taking information from.

And I think, I have to say, sometimes I feel that some of us in the academic bubble are too arrogant about this

and somehow think, oh, it's just people who aren't as educated as us for a fool.

When we are often just as gullible also, we read only our media and don't see through things.

Anyone who looks at both sides like this in comparison will immediately start noticing the shenanigans being pulled at.

And I think what I try to do with this app is that big tech has to some extent tried to blame the individual

for being manipulated much like big tobacco tried to blame the individuals entirely for smoking.

And later on, our government stepped up and said, actually, you can't just blame little kids for starting to smoke.

You have to have more responsible advertising and this and that.

I think it's a bit the same here. It's very convenient for a big tech to blame.

So it's just people who are so dumb and get fooled.

The blame usually comes in saying, oh, it's just human psychology.

People just want to hear what they already believe.

But Professor David Rand at MIT actually partly debunked that with a really nice study showing that people tend to be interested

in hearing things that go against what they believe if it's presented in a respectful way.

Suppose, for example, that you have a company and you're just about to launch this project and you're convinced it's going to work.

And someone says, you know, Lex, I hate to tell you this, but this is going to fail.

And here's why. Would you be like, shut up. I don't want to hear it.

Would you? You would be interested, right?

And also, if you're on an airplane back in the pre COVID times, you know,

and the guy next to you is clearly from the opposite side of the political spectrum,

but is very respectful and polite to you.

Wouldn't you be kind of interested to hear a bit about how he or she thinks about things?

Of course.

But it's not so easy to find out respectful disagreement now,

because like, for example, if you are a Democrat and you're like, oh, I want to see something on the other side.

So you just go bright bar.com.

And then after the first 10 seconds, you feel deeply insulted by something.

It's not going to work.

Or if you take someone who votes Republican and they go to something on the left and they just get very offended very quickly

by them having put a deliberately ugly picture of Donald Trump on the front page or something, it doesn't really work.

So this news aggregator also has a nuanced slider, which you can pull to the right.

And then to make it easier to get exposed to actually more sort of academic style or more respectful portrayals of different views.

And finally, the one kind of bias I think people are mostly aware of is the left right,

because it's so obvious because both left and right are very powerful here, right?

Both of them have well funded TV stations and newspapers and it's kind of hard to miss.

But there's another one, the establishment slider, which is also really fun.

I love to play with it.

And that's more about corruption.

Because if you have a society where almost all the powerful entities want you to believe a certain thing,

that's what you're going to read in both the big mainstream media on the left and on the right, of course.

And powerful companies can push back very hard.

Like tobacco companies push back very hard back in the day when some newspaper started writing articles about tobacco being dangerous.

So it was hard to get a lot of coverage about it initially.

And also if you look geopolitically, right?

Of course, in any country when you read their media, you're mainly going to be reading a lot about articles about how our country is the good guy

and the other countries are the bad guys, right?

So if you want to have a really more nuanced understanding, you know, like the Germans used to be told that the British used to be told that the French were the bad guys

and the French used to be told that the British were the bad guys.

Now they visit each other's countries a lot and have a much more nuanced understanding.

I don't think there's going to be any more wars between France and Germany.

On the geopolitical scale, it's just as much as ever, you know, big Cold War now, US, China, and so on.

And if you want to get a more nuanced understanding of what's happening geopolitically, then it's really fun to look at this establishment slider

because it turns out there are tons of little newspapers, both on the left and on the right, who sometimes challenge establishment

and say, you know, maybe we shouldn't actually invade Iraq right now.

Maybe this weapons and mass destruction thing is BS.

If you look at journalism research afterwards, you can actually see that.

Clearly, both CNN and Fox were very pro.

Let's get rid of Saddam.

There are weapons and mass destruction.

Then there were a lot of smaller newspapers.

They were like, wait a minute, this evidence seems a bit sketchy and maybe we...

But of course, they were so hard to find.

Most people didn't even know they existed, right?

Yet, it would have been better for American national security if those voices had also come up.

I think it harmed America's national security, actually, that we invaded Iraq.

And arguably, there's a lot more interest in that kind of thinking, too, from those small sources.

So, like, when you say big, it's more about kind of the reach of the broadcast.

But it's not big in terms of the interest.

I think there's a lot of interest in that kind of antiestablishment or skepticism towards...

Out of the box thinking, there's a lot of interest in that kind of thing.

Do you see this news project or something like it being basically taken over the world as the main way we consume information?

Like, how do we get there?

So, okay, the idea is brilliant. You're calling it your little project in 2020.

But how does that become the new way we consume information?

I hope, first of all, just to plant a little seed there.

Because normally, the big barrier of doing anything in media is you need a ton of money.

But this costs no money at all.

I've just been paying myself.

You pay a tiny amount of money each month to Amazon to run the thing in their cloud.

There will never be any ads.

The point is not to make any money off of it.

And we just train machine learning algorithms to classify the articles and stuff.

So, it just kind of runs by itself.

So, if it actually gets good enough at some point that it starts catching on, it could scale.

And if other people carbon copy it and make other versions that are better, that's the more the merrier.

I think there's a real opportunity for machine learning to empower the individual against the powerful players.

As I said in the beginning here, it's been mostly the other way around so far,

that the big players have the AI and then they tell people this is the truth, this is how it is.

But it can just as well go the other way around.

When the internet was born, actually, a lot of people had this hope that maybe this will be a great thing for democracy,

make it easier to find out about things.

And maybe machine learning and things like this can actually help again.

And I have to say, I think it's more important than ever now,

because this is very linked also to the whole future of life as we discussed earlier.

We're getting this ever more powerful tack.

Frank, it's pretty clear if you look on the one or two generation, three generation timescale,

that there are only two ways this can end, geopolitically.

Either it ends great for all humanity, or it ends terribly for all of us.

There's really no way in between.

And we're so stuck in, because technology knows no borders.

And you can't have people fighting when the weapons just keep getting ever more powerful indefinitely.

Eventually, the luck runs out.

And right now we have, I love America, but the fact of the matter is,

what's good for America is not opposites in the long term to what's good for other countries.

It would be if this was some sort of zero sum game like it was thousands of years ago,

when the only way one country could get more resources was to take land from other countries,

because that was basically the resource.

Look at the map of Europe, some countries kept getting bigger and smaller, endless wars.

But then, since 1945, there hasn't been any war in Western Europe, and they all got way richer, because of tech.

So the optimistic outcome is that the big winner in this century is going to be America and China,

and Russia, and everybody else, because technology just makes us all healthier and wealthier.

And we just find some way of keeping the peace on this planet.

But I think, unfortunately, there are some pretty powerful forces right now

that are pushing in exactly the opposite direction and trying to demonize other countries,

which just makes it more likely that this ever more powerful tech we're building is going to be in disastrous ways.

Yeah, for aggression versus cooperation, that kind of thing.

Yeah, even look at just military AI now, right?

It was so awesome to see these dancing robots.

I loved it, right? But one of the biggest growth areas in robotics now is, of course, autonomous weapons.

And 2020 was like the best marketing year ever for autonomous weapons,

because in both Libya, Civil War, and in Nagorno Karabakh, they made the decisive difference, right?

And everybody else is like watching this, oh yeah, we want to build autonomous weapons too.

In Libya, you had, on one hand, our ally, the United Arab Emirates,

that were flying their autonomous weapons that they bought from China, bombing Libyans.

And on the other side, you had our other ally, Turkey, flying their drones.

They had no skin in the game, any of these other countries.

And of course, it was the Libyans who really got screwed.

In Nagorno Karabakh, you had actually, again, now Turkey is sending drones built by this company

that was actually founded by a guy who went to MIT AeroAstrode, you know that?

So MIT has a direct responsibility for ultimately this.

And a lot of civilians were killed there.

So because it was militarily so effective, now suddenly there's like a huge push.

Yeah, yeah, let's go build ever more autonomy into these weapons.

And it's going to be great.

And I think actually people who are obsessed about some sort of future terminers,

NATO scenario right now, should start focusing on the fact that we have two

much more urgent threats happening for machine learning.

One of them is the whole destruction of democracy that we've talked about now,

where our flow of information is being manipulated by machine learning.

And the other one is that right now, you know, this is the year when the big arms race

out of control arms race in at least Thomas weapons is going to start or it's going to stop.

So you have a sense that there is, like 2020 was an instrumental catalyst for the race of the autonomous weapons race.

Yeah, because it was the first year when they proved decisive in the battlefield.

And these ones are still not fully autonomous, mostly they're remote controlled, right?

But, you know, we could very quickly make things about, you know, the size and cost of a smartphone,

which you just put in the GPS coordinates or the face of the one you want to kill,

a skin color or whatever and it flies away and does it.

And the real good reason why the US and all the other superpowers should put the kibosh on this

is the same reason we decided to put the kibosh on bio weapons.

So, you know, we gave the future of life award that we can talk more about later.

Matthew Messelsen from Harvard before for convincing Nixon to ban bio weapons.

And I asked him, how did you do it?

And he was like, well, I just said, look, we don't want there to be a $500 weapon of mass destruction

that even all our enemies can afford, even non state actors.

And Nixon was like, good point.

You know, it's in America's interest that the powerful weapons are all really expensive.

So only we can afford them or maybe some more stable adversaries, right?

Nuclear weapons are like that.

But bio weapons were not like that.

That's why we banned them.

And that's why you never hear about them now.

That's why we love biology.

So you have a sense that it's possible for the big powerhouses in terms of the big nations in the world

to agree that autonomous weapons is not a race we want to be on.

That it doesn't end well.

Yeah, because we know it's just going to end in mass proliferation

and every terrorist everywhere is going to have these super cheap weapons that they will use against us.

And our politicians have to constantly worry about being assassinated every time they go outdoors

by some anonymous little mini drone.

We don't want that.

And even if the U.S. and China and everyone else could just agree that you can only build these weapons

if they cost at least 10 million bucks, that would be a huge win for the superpowers.

And frankly for everybody, people often push back and say,

well, it's so hard to prevent cheating.

But hey, you can say the same about bioweapons.

Take any of your RMIT colleagues in biology.

Of course they could build some nasty bioweapon if they really wanted to.

But first of all, they don't want to because they think it's disgusting because of the stigma.

And second, even if there's some sort of nutcase and want to,

it's very likely that some of their grad students or someone would rat them out

because everyone else thinks it's so disgusting.

And in fact, we now know there was even a fair bit of cheating on the bioweapons ban.

But none, no countries used them because it was so stigmatized

that it just wasn't worth revealing that they had cheated.

You talk about drones, but you kind of think that drones is the remote operation.

Which they are mostly still.

But you're not taking the next intellectual step of like, where does this go?

You're kind of saying the problem with drones is that you're removing yourself from direct violence.

Therefore, you're not able to sort of maintain the common humanity

required to make the proper decisions strategically.

But that's the criticism as opposed to like, if this is automated,

and just exactly as you said, if you automate it and there's a race,

then the technology is going to get better and better and better,

which means getting cheaper and cheaper and cheaper.

And unlike perhaps nuclear weapons, which is connected to resources in a way,

like it's hard to get the, it's hard to engineer.

It feels like it's, you know, there's too much overlap between the tech industry

and autonomous weapons to where you could have smartphone type of cheapness.

If you look at drones, you know, it's a, you know, for $1,000,

you have an incredible system that's able to maintain flight autonomously for you

and take pictures and stuff.

You could see that going into the autonomous weapon space that's,

but like, why is that not thought about or discussed enough in the public?

Do you think you see those dancing Boston Dynamics robots and everybody has this kind of,

like as if this is like a far future.

They have this like fear, like, oh, this will be Terminator in like some,

I don't know, unspecified 20, 30, 40 years.

And they don't think about, well, this is like some much less dramatic version of that is actually happening now.

It's not going to have, it's not going to be legged, it's not going to be dancing,

but it's already has the capability to use artificial intelligence to kill humans.

Yeah, the Boston Dynamics leg robots, I think the reason we imagine them holding guns

is just because you've all seen Arnold Schwarzenegger, right?

That's our reference point.

That's pretty useless.

That's not going to be the main military use of them.

They might be useful in law enforcement in the future.

And there's a whole debate about you want robots showing up at your house with guns

telling you who'll be perfectly obedient to whatever dictator controls them.

But let's leave that aside for a moment and look at what's actually relevant now.

There's a spectrum of things you can do with AI in the military.

And again, to put my card on the table, I'm not the pacifist.

I think we should have good defense.

So, for example, a predator drone is basically a fancy little remote controlled airplane.

There's a human piloting it and the decision ultimately about whether to kill somebody with it is made by a human still.

And this is a line I think we should never cross.

There's a current DOD policy.

Again, you have to have a human in the loop.

I think algorithms should never make life or death decisions.

They should be left to humans.

Now, why might we cross that line?

Well, first of all, these are expensive, right?

So, for example, when Azerbaijan had all these drones and Armenia didn't have any,

they started trying to jerry rig little cheap things, fly around.

But then, of course, the Armenians would jam them.

The Azeris would jam them.

And remote controlled things can be jammed.

That makes them inferior.

Also, there's a bit of a time delay between, you know, if we're piloting something far away,

speed of light, and the human has a reaction time as well,

it would be nice to eliminate that jamming possibility in the time delay by having it fully autonomous.

But now you might be crossing that exact line.

You might program it to just, oh, yeah, dear drone, go hover over this country for a while

and whenever you find someone who is a bad guy, you know, kill them.

Now, the machine is making these sort of decisions.

And some people who defend this still say, well, that's morally fine

because we are the good guys and we will tell it the definition of bad guy

that we think is moral.

But now it would be very naive to think that if ISIS buys that same drone

that they're going to use our definition of bad guy.

Maybe for them, bad guy is someone wearing a U.S. Army uniform.

Or maybe there will be some weird ethnic group

who decides that someone of an other ethnic group, they are the bad guys, right?

The thing is, human soldiers, with all of our faults, right,

we still have some basic wiring in us.

Like, no, it's not okay to kill kids and civilians.

And Thomas Reppin has none of that.

It's just going to do whatever is programmed.

It's like the perfect Adolf Eichmann on steroids.

Like, they told him, Adolf Eichmann, you know, you want you to do this and this and this

to make the Holocaust more efficient.

And he was like, yeah, and off he went and did it, right?

Do we really want to make machines that are like that, like completely amoral

and will take the user's definition of who is the bad guy?

And do we then want to make them so cheap that all our adversaries can have them?

Like, what could possibly go wrong?

That's the big argument for why we want to, this year, really put the kibosh on this.

And I think you can tell there's a lot of very active debate even going on

within the U.S. military and undoubtedly in other militaries around the world also

about whether we should have some sort of international agreement

to at least require that these weapons have to be above a certain size and cost,

so that things just don't totally spiral out of control.

And finally, just for your question now, but is it possible to stop it?

Because some people tell me, oh, just give up, you know.

But again, so Matthew Messelsen again from Harvard, right, who, the bio weapons hero,

he had exactly this criticism also with bio weapons.

People were like, how can you check for sure that the Russians aren't cheating?

And he told me this, I think really ingenious insight, he said, you know, Max,

some people think you have to have inspections and things,

and you have to make sure that people, you can catch the cheaters with 100% chance.

You don't need 100%, he said.

1% is usually enough.

Because if it's another big state,

I suppose China and the US have signed a treaty, drawing a certain line and saying,

yeah, these kind of drones are okay, but these fully autonomous ones are not.

Now suppose you are China and you have cheated and secretly developed some clandestine little thing,

or you're thinking about doing it, you know, what's your calculation that you do?

Well, you're like, okay, what's the probability that we're going to get caught?

If the probability is 100%, of course, we're not going to do it.

But if the probability is 5% that we're going to get caught,

then it's going to be like a huge embarrassment for us.

And we still have our nuclear weapons anyway, so it doesn't really make any enormous difference

in terms of deterring the US, you know.

And that feeds the stigma that you kind of establish, like this fabric, this universal stigma over the thing.

Exactly.

It's very reasonable for them to say, well, you know, we probably get away with it,

but if we don't, then the US will know we cheated,

and then they're going to go full tilt with their program and say, look, the Chinese are cheaters,

and now we have all these weapons against us, and that's bad.

So the stigma alone is very, very powerful.

And again, look what happened with bioweapons, right?

It's been 50 years now.

When was the last time you read about a bioterrorism attack?

The only deaths I really know about with bioweapons that have happened,

when we Americans managed to kill some of our own with anthrax,

you know, the idiot who sent them to Tom Daschel and others in letters, right?

And similarly, in Sverlovsk in the Soviet Union,

they had some anthrax in some lab there.

Maybe they were cheating or who knows,

and it leaked out and killed a bunch of Russians.

I'd say that's a pretty good success, right?

50 years, just two own goals by the superpowers, and then nothing.

And that's why whenever I ask anyone what they think about biology,

they think it's great.

They associate it with new cures, new diseases, maybe a good vaccine.

This is how I want to think about AI in the future.

And I want others to think about AI too,

as a source of all these great solutions to our problems,

not as, oh, AI.

Oh, yeah, that's the reason I feel scared going outside these days.

Yeah, it's kind of brilliant that the bio weapons and nuclear weapons,

we've figured out, I mean, of course, there's still a huge source of danger,

but we figured out some way of creating rules and social stigma

over these weapons that then creates a stability to our,

whatever that game theoretic stability there, of course.

Exactly, exactly.

And we don't have that with AI, and you're kind of screaming from the top

of the mountain about this, that we need to find that,

because just like, it's very possible with the future of life,

as you've pointed out, Institute Awards pointed out that with nuclear weapons,

we could have destroyed ourselves quite a few times.

And it's a learning experience that is very costly.

We gave this Future Life Award, we gave it the first time to this guy,

Vasily Arkhipov, he was on, most people haven't even heard of him.

Yeah, can you say who he is?

Vasily Arkhipov, he has, in my opinion,

made the greatest positive contribution to humanity of any human in modern history.

And maybe it sounds like hyperbole here, like I'm just over the top,

but let me tell you the story, and I think maybe you'll agree.

So during the Cuban Missile Crisis, we Americans first didn't know

that the Russians had sent four submarines, but we caught two of them,

and we didn't know that, so we dropped practice depth charges on the one that he was on,

trying to force it to the surface.

But we didn't know that this nuclear submarine actually was a nuclear submarine

with a nuclear torpedo.

We also didn't know that they had an authorization to launch it without clearance from Moscow.

And we also didn't know that they were running out of electricity,

their batteries were almost dead, they were running out of oxygen,

sailors were fainting left and right.

The temperature was about 110, 120 Fahrenheit on board,

it was really hellish conditions, really just a kind of doomsday.

And at that point, these giant explosions start happening

from Americans dropping these.

The captain thought World War III had begun.

They decided that they were going to launch the nuclear torpedo.

And one of them shouted, you know, we're all going to die,

but we're not going to disgrace our navy.

We don't know what would have happened if there had been a giant mushroom cloud all of a sudden

against Americans, but since everybody had their hands on the triggers,

you don't have to be too creative to think that it could have led to an all out nuclear war,

in which case we wouldn't be having this conversation now, right?

What actually took place was they needed three people to approve this.

The captain had said yes, there was the Communist Party political officer,

he also said yes, let's do it.

And the third man was this guy Vasily Arkhipov, who said,

yeah, for some reason he was just more chill than the others

and he was the right man at the right time.

I don't want us as a species rely on the right person being there at the right time.

You know, we tracked down his family living in relative poverty outside Moscow.

When he flew his daughter, he had passed away and flew them to London.

They had never been to the West even.

It was incredibly moving to get to honor them for this.

The next year we gave this future life award to Stanislav Petrov.

Have you heard of him?

Yes.

He was in charge of the Soviet early warning station which was built with Soviet technology

and honestly not that reliable.

It said that there were five US missiles coming in.

Again, if they had launched at that point, we probably wouldn't be having this conversation.

He decided based on just mainly gut instinct to just not escalate this.

I'm very glad he wasn't replaced by an AI that was just automatically falling orders.

Then we gave the third one to Matthew Messelsen.

Last year we gave this award to these guys who actually used technology for good,

not avoiding something bad, but for something good.

The guys who eliminated this disease, which is way worse than COVID,

that had killed half a billion people in its violent century.

Smallpox.

You mentioned it earlier.

COVID on average kills less than 1% of people who get it.

Smallpox, about 30%.

Ultimately, Viktor Zhdanov and Bill Fagy, most of my colleagues have never heard of either of them,

one American, one Russian, they did this amazing effort.

Not only was Zhdanov able to get the US and the Soviet Union to team up against smallpox

during the Cold War,

but Fagy came up with this ingenious strategy for making it actually go all the way

to defeat the disease without funding for vaccinating everyone.

As a result, we went from 15 million deaths the year I was born in smallpox.

So what do we have in COVID now?

A little bit short of 2 million, right?

Yes.

To zero deaths, of course, this year.

And forever, there have been 200 million people,

who would have died since then by smallpox had it not been for this.

So isn't science awesome when you use it for good?

And the reason we want to celebrate these sort of people is to remind them of this.

Science is so awesome when you use it for good.

And those awards actually, the variety there, it's a very interesting picture.

So the first two are looking at, it's kind of exciting to think that these average humans,

in some sense, there are products of billions of other humans that came before them, evolution.

And some little, you said gut, but there's something in there that stopped the annihilation of the human race.

And that's a magical thing, but that's like this deeply human thing.

And then there's the other aspect where it's also very human,

which is to build solution to the existential crises that we're facing,

to build it, to take responsibility, to come up with different technologies and so on.

And both of those are deeply human.

The gut and the mind, whatever that is.

The best is when they work together.

Archipelago, I wish I could have met him, of course, but he had passed away.

He was really a fantastic military officer, combining all the best traits that we in America admire in our military.

Because first of all, he was very loyal, of course.

He never even told anyone about this during his whole life, even though you think he had some bragging rights, right?

But he just was like, this is just business, just doing my job.

It only came out later after his death.

And second, the reason he did the right thing was not because he was some sort of liberal,

not because he was just, oh, you know, peace and love.

It was partly because he had been the captain on another submarine that had a nuclear reactor meltdown.

And it was his heroism that helped contain this.

That's why he died of cancer later also.

But he's seen many of his crew members die.

And I think for him, that gave him this gut feeling that, you know,

if there's a nuclear war between the US and the Soviet Union, the whole world is going to go through

what I saw my dear crew members suffer through.

It wasn't just an abstract thing for him.

I think it was real.

And second, though, not just the gut, the mind, right?

He was, for some reason, very level headed personality and very smart guy,

which is exactly what we want our best fighter pilots to be also.

I never forget Neil Armstrong when he's landing on the moon and almost running out of gas.

And he doesn't even change, let me say 30 seconds.

He doesn't even change the tone of voice, just keeps going.

Archipelago, I think, was just like that.

So when the explosions start going off and his captain is screaming and we should nuke them and all,

he's like, I don't think the Americans are trying to sink us.

I think they're trying to send us a message.

That's pretty badass.

Coolness.

Because he said, if they wanted to sink us, he said, listen, listen,

it's alternating one loud explosion on the left, one on the right, one on the left, one on the right.

He was the only one to notice this pattern.

And he's like, I think this is them trying to send us a signal that they wanted to surface

and they're not going to sink us.

And somehow this is how he then managed it ultimately with his combination of gut

and also just cool analytical thinking, was able to deescalate the whole thing.

And yeah, so this is some of the best in humanity.

I guess coming back to what we talked about earlier is the combination of the neural network,

the instinctive, you know, with I'm tearing up here, getting emotional.

But he is one of my superheroes having both the heart and the mind combined.

And especially in that time, there's something about the, I mean, this is a very,

in America, people are used to this kind of idea of being the individual of like on your own thinking.

I think in the Soviet Union under communism, it's actually much harder to do that.

Oh yeah, he didn't even, he even got, he didn't get any accolades either when he came back for this, right?

They just wanted to hush the whole thing up.

Yeah, there's echoes of that with Chernobyl, there's all kinds of, that's one,

that's a really hopeful thing that amidst big centralized powers,

whether it's companies or states, there's still the power of the individual to think on their own to act.

But I think we need to think of people like this, not as a panacea we can always count on,

but rather as a wake up call, you know.

So because of them, because of Arkhipov, we are alive to learn from this lesson,

to learn from the fact that we shouldn't keep playing Russian roulette

and almost have a nuclear war by mistake now and then,

because relying on luck is not a good long term strategy.

If you keep playing Russian roulette over and over again,

the probability of surviving just drops exponentially with time.

And if you have some probability of having an accidental nuclear war every year,

the probability of not having one also drops exponentially.

I think we can do better than that.

So I think the message is very clear, once in a while shit happens,

and there's a lot of very concrete things we can do to reduce the risk of things like that happening in the first place.

On the AI front, if we just link on that for a second.

So you're friends with, you often talk with Elon Musk throughout history.

You've did a lot of interesting things together.

He has a set of fears about the future of artificial intelligence, AGI.

Do you have a sense, we've already talked about the things we should be worried about with AI.

Do you have a sense of the shape of his fears in particular,

about AI, which subset of what we've talked about, whether it's creating,

it's that direction of creating these giant computational systems that are not explainable,

they're not intelligible intelligence, or is it the...

And then as a branch of that, is it the manipulation by big corporations of that,

or individual evil people to use that for destruction, or the unintentional consequences.

Do you have a sense of where his thinking is on this?

From my many conversations with Elon, I certainly have a model of how he thinks.

It's actually very much like the way I think also, I'll elaborate on it a bit.

I just want to push back on when you said evil people.

I don't think it's a very helpful concept, evil people.

Sometimes people do very, very bad things, but they usually do it because they think it's a good thing,

because somehow other people had told them that that was a good thing,

or given them incorrect information, or whatever.

I believe in the fundamental goodness of humanity that if we educate people well,

and they find out how things really are, people generally want to do good and be good.

There's a sense of value alignment.

It's about information, it's about knowledge, and then once we have that,

we'll likely be able to do good in the way that's aligned with everybody else who thinks it's good.

Yeah, and it's not just the individual people we have to align,

so we don't just want people to be educated to know the way things actually are,

and to treat each other well, but we also need to align other nonhuman entities.

We've talked about corporations, there has to be institutions,

so that what they do is actually good for the country they're in,

and we should make sure that what countries do is actually good for the species as a whole, etc.

Coming back to Elon, my understanding of how Elon sees this is really quite similar to my own,

which is one of the reasons I like him so much, and enjoy talking with him so much.

He's quite different from most people in that he thinks much more than most people about their really big picture,

not just what's going to happen in the next election cycle,

and millennia, millions and billions of years from now.

When you look in this more cosmic perspective, it's so obvious that we're gazing out into this universe

that as far as we can tell is mostly dead, with life being almost imperceptibly tiny perturbation, right?

And he sees this enormous opportunity for our universe to come alive,

first to become an interplanetary species.

Mars is obviously just first stop on this cosmic journey,

and precisely because he thinks more long term, it's much more clear to him than to most people

that what we do with this Russian roulette thing we keep playing with our nukes is a really poor strategy,

a really reckless strategy, and also that we're just building these ever more powerful AI systems

that we don't understand is also a really reckless strategy.

I feel Elon is a humanist in the sense that he wants an awesome future for humanity.

He wants it to be us that control the machines, rather than the machines that control us.

And why shouldn't we insist on that? We're building them after all, right?

Why should we build things that just make us into some little cog in the machinery

that has no further say in the matter, right?

It's not my idea of an inspiring future either.

Yeah, if you think on the cosmic scale in terms of both time and space, so much is put into perspective.

Whenever I have a bad day, that's what I think about. It immediately makes me feel better.

It makes me sad that for us individual humans, at least for now, the ride ends too quickly.

We don't get to experience the cosmic scale.

I mean, I think of our universe sometimes as an organism that has only begun to wake up a tiny bit.

Just like the very first little glimmers of consciousness you have in the morning when you start coming around.

Before the coffee.

Before the coffee. Even before you get out of bed, before you even open your eyes, start to wake up a little bit.

There's something here, you know. That's very much how I think of where we are.

All those galaxies out there, I think they're really beautiful.

But why are they beautiful?

They're beautiful because conscious entities are actually observing them and experiencing them through our telescopes.

I define consciousness as subjective experience.

Whether it be colors or emotions or sounds.

So beauty is an experience.

Meaning is an experience.

Purpose is an experience.

If there was no conscious experience observing these galaxies, they wouldn't be beautiful.

If we do something dumb with advanced AI in the future here and Earth originating, life goes extinct.

And that was it for this.

If there is nothing else with telescopes in our universe, then it's kind of game over for beauty and meaning and purpose in our whole universe.

And I think that would be just such an opportunity lost, frankly.

And I think when Elon points this out, he gets very unfairly maligned in the media for all the dumb media bias reasons we talked about, right?

They want to print precisely the things about Elon out of context that are really clickbaity.

Like he has gotten so much flak for this summoning the demon statement.

I happen to know exactly the context because I was in the front row when he gave that talk. It was at MIT, you'll be pleased to know.

It was the AeroAstro anniversary.

They had Buzz Aldrin there from the moon landing, a whole house, a Kresge auditorium packed with MIT students.

And he had this amazing Q&A.

It might have gone for an hour and they talked about rockets and Mars and everything.

At the very end, this one student who has actually hit my class asked him, what about AI?

Elon makes this one comment and they take this out of context, print it, goes viral.

Was it like with AI where summoning the demons and stuff like that?

And try to cast him as some sort of doom and gloom dude.

You know Elon.

He's not the doom and gloom dude.

He is such a positive visionary.

And the whole reason he warns about this is because he realizes more than most what the opportunity cost is of screwing up.

That there is so much awesomeness in the future that we can and our descendants can enjoy if we don't screw up, right?

I get so pissed off when people try to cast him as some sort of technophobic Luddite.

And at this point, it's kind of ludicrous when I hear people say that people who worry about artificial general intelligence are Luddites.

Because of course, if you look more closely, you have some of the most outspoken people making warnings.

Are people like Professor Stuart Russell from Berkeley who's written the best selling AI textbook, you know.

So claiming that he is a Luddite who doesn't understand AI is the joke is really on the people who said it.

But I think more broadly, this message is really not sunk in at all.

What it is that people worry about.

They think that Elon and Stuart Russell and others are worried about the dancing robots picking up an AR15 and going on a rampage, right?

They think they're worried about robots turning evil.