back to indexRajat Monga: TensorFlow | Lex Fridman Podcast #22
link |
The following is a conversation with Rajat Manga.
link |
He's an engineer and director of Google,
link |
leading the TensorFlow team.
link |
TensorFlow is an open source library
link |
at the center of much of the work going on in the world
link |
in deep learning, both the cutting edge research
link |
and the large scale application of learning based approaches.
link |
But it's quickly becoming much more than a software library.
link |
It's now an ecosystem of tools for the deployment of machine
link |
learning in the cloud, on the phone, in the browser,
link |
on both generic and specialized hardware.
link |
TPU, GPU, and so on.
link |
Plus, there's a big emphasis on growing a passionate community
link |
Rajat, Jeff Dean, and a large team of engineers at Google
link |
Brain are working to define the future of machine
link |
learning with TensorFlow 2.0, which is now in alpha.
link |
I think the decision to open source TensorFlow
link |
is a definitive moment in the tech industry.
link |
It showed that open innovation can be successful
link |
and inspire many companies to open source their code,
link |
to publish, and in general engage
link |
in the open exchange of ideas.
link |
This conversation is part of the Artificial Intelligence
link |
If you enjoy it, subscribe on YouTube, iTunes,
link |
or simply connect with me on Twitter at Lex Friedman,
link |
And now, here's my conversation with Rajat Manga.
link |
You were involved with Google Brain since its start in 2011
link |
It started with this belief, the proprietary machine learning
link |
library, and turned into TensorFlow in 2014,
link |
the open source library.
link |
So what were the early days of Google Brain like?
link |
What were the goals, the missions?
link |
How do you even proceed forward once there's
link |
so much possibilities before you?
link |
It was interesting back then when I started,
link |
or when you were even just talking about it,
link |
the idea of deep learning was interesting and intriguing
link |
It hadn't yet taken off, but it held some promise.
link |
It had shown some very promising and early results.
link |
I think the idea where Andrew and Jeff had started
link |
was, what if we can take this work people are doing
link |
in research and scale it to what Google has
link |
in terms of the compute power, and also
link |
put that kind of data together?
link |
What does it mean?
link |
And so far, the results had been, if you scale the compute,
link |
scale the data, it does better.
link |
And would that work?
link |
And so that was the first year or two, can we prove that out?
link |
And with this belief, when we started the first year,
link |
we got some early wins, which is always great.
link |
What were the wins like?
link |
What was the wins where you were,
link |
there's some problems to this, this is going to be good?
link |
I think there are two early wins where one was speech,
link |
that we collaborated very closely with the speech research
link |
team, who was also getting interested in this.
link |
And the other one was on images, where the cat paper,
link |
as we call it, that was covered by a lot of folks.
link |
And the birth of Google Brain was around neural networks.
link |
So it was deep learning from the very beginning.
link |
That was the whole mission.
link |
So what would, in terms of scale,
link |
what was the sort of dream of what this could become?
link |
Were there echoes of this open source TensorFlow community
link |
that might be brought in?
link |
Was there a sense of TPUs?
link |
Was there a sense of machine learning is now going to be
link |
at the core of the entire company,
link |
is going to grow into that direction?
link |
Yeah, I think, so that was interesting.
link |
And if I think back to 2012 or 2011,
link |
and first was can we scale it in the year or so,
link |
we had started scaling it to hundreds and thousands
link |
In fact, we had some runs even going to 10,000 machines.
link |
And all of those shows great promise.
link |
In terms of machine learning at Google,
link |
the good thing was Google's been doing machine learning
link |
Deep learning was new, but as we scaled this up,
link |
we showed that, yes, that was possible.
link |
And it was going to impact lots of things.
link |
Like we started seeing real products wanting to use this.
link |
Again, speech was the first, there were image things
link |
that photos came out of and then many other products as well.
link |
So that was exciting.
link |
As we went into that a couple of years,
link |
externally also academia started to,
link |
there was lots of push on, okay,
link |
deep learning is interesting,
link |
we should be doing more and so on.
link |
And so by 2014, we were looking at, okay,
link |
this is a big thing, it's going to grow.
link |
And not just internally, externally as well.
link |
Yes, maybe Google's ahead of where everybody is,
link |
but there's a lot to do.
link |
So a lot of this started to make sense and come together.
link |
So the decision to open source,
link |
I was just chatting with Chris Glatner about this.
link |
The decision to go open source with TensorFlow,
link |
I would say sort of for me personally,
link |
seems to be one of the big seminal moments
link |
in all of software engineering ever.
link |
I think that's when a large company like Google
link |
decides to take a large project that many lawyers
link |
might argue has a lot of IP,
link |
just decide to go open source with it,
link |
and in so doing lead the entire world
link |
and saying, you know what, open innovation
link |
is a pretty powerful thing, and it's okay to do.
link |
That was, I mean, that's an incredible moment in time.
link |
So do you remember those discussions happening?
link |
Whether open source should be happening?
link |
What was that like?
link |
I would say, I think, so the initial idea came from Jeff,
link |
who was a big proponent of this.
link |
I think it came off of two big things.
link |
One was research wise, we were a research group.
link |
We were putting all our research out there.
link |
If you wanted to, we were building on others research
link |
and we wanted to push the state of the art forward.
link |
And part of that was to share the research.
link |
That's how I think deep learning and machine learning
link |
has really grown so fast.
link |
So the next step was, okay, now,
link |
would software help with that?
link |
And it seemed like they were existing
link |
a few libraries out there, Tiano being one,
link |
Torch being another, and a few others,
link |
but they were all done by academia
link |
and so the level was significantly different.
link |
The other one was from a software perspective,
link |
Google had done lots of software
link |
or that we used internally, you know,
link |
and we published papers.
link |
Often there was an open source project
link |
that came out of that that somebody else
link |
picked up that paper and implemented
link |
and they were very successful.
link |
Back then it was like, okay, there's Hadoop,
link |
which has come off of tech that we've built.
link |
We know the tech we've built is way better
link |
for a number of different reasons.
link |
We've invested a lot of effort in that.
link |
And turns out we have Google Cloud
link |
and we are now not really providing our tech,
link |
but we are saying, okay, we have Bigtable,
link |
which is the original thing.
link |
We are going to now provide H base APIs
link |
on top of that, which isn't as good,
link |
but that's what everybody's used to.
link |
So there's like, can we make something
link |
that is better and really just provide,
link |
helps the community in lots of ways,
link |
but also helps push a good standard forward.
link |
So how does Cloud fit into that?
link |
There's a TensorFlow open source library
link |
and how does the fact that you can
link |
use so many of the resources that Google provides
link |
and the Cloud fit into that strategy?
link |
So TensorFlow itself is open
link |
and you can use it anywhere, right?
link |
And we want to make sure that continues to be the case.
link |
On Google Cloud, we do make sure
link |
that there's lots of integrations with everything else
link |
and we want to make sure
link |
that it works really, really well there.
link |
You're leading the TensorFlow effort.
link |
Can you tell me the history
link |
and the timeline of TensorFlow project
link |
in terms of major design decisions,
link |
so like the open source decision,
link |
but really what to include and not?
link |
There's this incredible ecosystem
link |
that I'd like to talk about.
link |
There's all these parts,
link |
but what if just some sample moments
link |
that defined what TensorFlow eventually became
link |
through its, I don't know if you're allowed to say history
link |
when it's just, but in deep learning,
link |
everything moves so fast
link |
and just a few years is already history.
link |
Yes, yes, so looking back, we were building TensorFlow.
link |
I guess we open sourced it in 2015, November 2015.
link |
We started on it in summer of 2014, I guess.
link |
And somewhere like three to six, late 2014,
link |
by then we had decided that, okay,
link |
there's a high likelihood we'll open source it.
link |
So we started thinking about that
link |
and making sure we're heading down that path.
link |
At that point, by that point,
link |
we had seen a few, lots of different use cases at Google.
link |
So there were things like, okay,
link |
yes, you wanna run it at large scale in the data center.
link |
Yes, we need to support different kind of hardware.
link |
We had GPUs at that point.
link |
We had our first GPU at that point
link |
or was about to come out roughly around that time.
link |
So the design sort of included those.
link |
We had started to push on mobile.
link |
So we were running models on mobile.
link |
At that point, people were customizing code.
link |
So we wanted to make sure TensorFlow
link |
could support that as well.
link |
So that sort of became part of that overall design.
link |
When you say mobile,
link |
you mean like a pretty complicated algorithms
link |
running on the phone?
link |
So when you have a model that you deploy on the phone
link |
and run it there, right?
link |
So already at that time,
link |
there was ideas of running machine learning on the phone.
link |
We already had a couple of products
link |
that were doing that by then.
link |
And in those cases,
link |
we had basically customized handcrafted code
link |
or some internal libraries that we're using.
link |
So I was actually at Google during this time
link |
in a parallel, I guess, universe,
link |
but we were using Theano and Caffe.
link |
Was there some degree to which you were bouncing,
link |
like trying to see what Caffe was offering people,
link |
trying to see what Theano was offering
link |
that you want to make sure you're delivering
link |
on whatever that is?
link |
Perhaps the Python part of thing,
link |
maybe did that influence any design decisions?
link |
So when we built this belief
link |
and some of that was in parallel
link |
with some of these libraries coming up,
link |
I mean, Theano itself is older,
link |
but we were building this belief
link |
focused on our internal thing
link |
because our systems were very different.
link |
By the time we got to this,
link |
we looked at a number of libraries that were out there.
link |
Theano, there were folks in the group
link |
who had experience with Torch, with Lua.
link |
There were folks here who had seen Caffe.
link |
I mean, actually, Yang Jing was here as well.
link |
There's what other libraries?
link |
I think we looked at a number of things.
link |
Might even have looked at JNR back then.
link |
I'm trying to remember if it was there.
link |
In fact, yeah, we did discuss ideas around,
link |
okay, should we have a graph or not?
link |
So putting all these together was definitely,
link |
they were key decisions that we wanted.
link |
We had seen limitations in our prior disbelief things.
link |
A few of them were just in terms of research
link |
was moving so fast, we wanted the flexibility.
link |
The hardware was changing fast.
link |
We expected to change that
link |
so that those probably were two things.
link |
And yeah, I think the flexibility
link |
in terms of being able to express
link |
all kinds of crazy things was definitely a big one then.
link |
So what, the graph decisions though,
link |
with moving towards TensorFlow 2.0,
link |
there's more, by default, there'll be eager execution.
link |
So sort of hiding the graph a little bit
link |
because it's less intuitive
link |
in terms of the way people develop and so on.
link |
What was that discussion like in terms of using graphs?
link |
It seemed, it's kind of the Theano way.
link |
Did it seem the obvious choice?
link |
So I think where it came from was our disbelief
link |
had a graph like thing as well.
link |
A much more simple, it wasn't a general graph,
link |
it was more like a straight line thing.
link |
More like what you might think of cafe,
link |
I guess in that sense.
link |
But the graph was,
link |
and we always cared about the production stuff.
link |
Like even with disbelief,
link |
we were deploying a whole bunch of stuff in production.
link |
So graph did come from that when we thought of,
link |
okay, should we do that in Python?
link |
And we experimented with some ideas
link |
where it looked a lot simpler to use,
link |
but not having a graph meant,
link |
okay, how do you deploy now?
link |
So that was probably what tilted the balance for us
link |
and eventually we ended up with a graph.
link |
And I guess the question there is, did you,
link |
I mean, so production seems to be
link |
the really good thing to focus on,
link |
but did you even anticipate the other side of it
link |
where there could be, what is it?
link |
What are the numbers?
link |
It's been crazy, 41 million downloads.
link |
I mean, was that even like a possibility in your mind
link |
that it would be as popular as it became?
link |
So I think we did see a need for this
link |
a lot from the research perspective
link |
and like early days of deep learning in some ways.
link |
41 million, no, I don't think I imagined this number.
link |
Then it seemed like there's a potential future
link |
where lots more people would be doing this
link |
and how do we enable that?
link |
I would say this kind of growth,
link |
I probably started seeing somewhat after the open sourcing
link |
where it was like, okay,
link |
deep learning is actually growing way faster
link |
for a lot of different reasons.
link |
And we are in just the right place to push on that
link |
and leverage that and deliver on lots of things
link |
So what changed once you open sourced?
link |
Like how this incredible amount of attention
link |
from a global population of developers,
link |
how did the project start changing?
link |
I don't even actually remember during those times.
link |
I know looking now, there's really good documentation,
link |
there's an ecosystem of tools,
link |
there's a community, there's a blog,
link |
there's a YouTube channel now, right?
link |
It's very community driven.
link |
Back then, I guess 0.1 version,
link |
is that the version?
link |
I think we call it 0.6 or five,
link |
something like that, I forget.
link |
What changed leading into 1.0?
link |
I think we've gone through a few things there.
link |
When we started out, when we first came out,
link |
people loved the documentation we have
link |
because it was just a huge step up from everything else
link |
because all of those were academic projects,
link |
people doing, who don't think about documentation.
link |
I think what that changed was,
link |
instead of deep learning being a research thing,
link |
some people who were just developers
link |
could now suddenly take this out
link |
and do some interesting things with it, right?
link |
Who had no clue what machine learning was before then.
link |
And that I think really changed
link |
how things started to scale up in some ways
link |
Over the next few months as we looked at
link |
how do we stabilize things,
link |
as we look at not just researchers,
link |
now we want stability, people want to deploy things.
link |
That's how we started planning for 1.0
link |
and there are certain needs for that perspective.
link |
And so again, documentation comes up,
link |
designs, more kinds of things to put that together.
link |
And so that was exciting to get that to a stage
link |
where more and more enterprises wanted to buy in
link |
and really get behind that.
link |
And I think post 1.0 and over the next few releases,
link |
that enterprise adoption also started to take off.
link |
I would say between the initial release and 1.0,
link |
it was, okay, researchers of course,
link |
then a lot of hobbies and early interest,
link |
people excited about this who started to get on board
link |
and then over the 1.x thing, lots of enterprises.
link |
I imagine anything that's below 1.0
link |
gives pressure to be,
link |
the enterprise probably wants something that's stable.
link |
And do you have a sense now that TensorFlow is stable?
link |
Like it feels like deep learning in general
link |
is extremely dynamic field, so much is changing.
link |
And TensorFlow has been growing incredibly.
link |
Do you have a sense of stability at the helm of it?
link |
I mean, I know you're in the midst of it, but.
link |
Yeah, I think in the midst of it,
link |
it's often easy to forget what an enterprise wants
link |
and what some of the people on that side want.
link |
There are still people running models
link |
that are three years old, four years old.
link |
So Inception is still used by tons of people.
link |
Even ResNet 50 is what, couple of years old now or more,
link |
but there are tons of people who use that and they're fine.
link |
They don't need the last couple of bits of performance
link |
or quality, they want some stability
link |
in things that just work.
link |
And so there is value in providing that
link |
with that kind of stability and making it really simpler
link |
because that allows a lot more people to access it.
link |
And then there's the research crowd which wants,
link |
okay, they wanna do these crazy things
link |
exactly like you're saying, right?
link |
Not just deep learning in the straight up models
link |
that used to be there, they want RNNs
link |
and even RNNs are maybe old, they are transformers now.
link |
And now it needs to combine with RL and GANs and so on.
link |
So there's definitely that area that like the boundary
link |
that's shifting and pushing the state of the art.
link |
But I think there's more and more of the past
link |
that's much more stable and even stuff
link |
that was two, three years old is very, very usable
link |
by lots of people.
link |
So that part makes it a lot easier.
link |
So I imagine, maybe you can correct me if I'm wrong,
link |
one of the biggest use cases is essentially
link |
taking something like ResNet 50
link |
and doing some kind of transfer learning
link |
on a very particular problem that you have.
link |
It's basically probably what majority of the world does.
link |
And you wanna make that as easy as possible.
link |
So I would say for the hobbyist perspective,
link |
that's the most common case, right?
link |
In fact, the apps and phones and stuff that you'll see,
link |
the early ones, that's the most common case.
link |
I would say there are a couple of reasons for that.
link |
One is that everybody talks about that.
link |
It looks great on slides.
link |
That's a presentation, yeah, exactly.
link |
What enterprises want is that is part of it,
link |
but that's not the big thing.
link |
Enterprises really have data
link |
that they wanna make predictions on.
link |
This is often what they used to do
link |
with the people who were doing ML
link |
was just regression models,
link |
linear regression, logistic regression, linear models,
link |
or maybe gradient booster trees and so on.
link |
Some of them still benefit from deep learning,
link |
but they want that's the bread and butter,
link |
or like the structured data and so on.
link |
So depending on the audience you look at,
link |
they're a little bit different.
link |
And they just have, I mean, the best of enterprise
link |
probably just has a very large data set,
link |
or deep learning can probably shine.
link |
That's correct, that's right.
link |
And then I think the other pieces that they wanted,
link |
again, with 2.0, the developer summit we put together
link |
is the whole TensorFlow Extended piece,
link |
which is the entire pipeline.
link |
They care about stability across doing their entire thing.
link |
They want simplicity across the entire thing.
link |
I don't need to just train a model.
link |
I need to do that every day again, over and over again.
link |
I wonder to which degree you have a role in,
link |
I don't know, so I teach a course on deep learning.
link |
I have people like lawyers come up to me and say,
link |
when is machine learning gonna enter legal,
link |
The same thing in all kinds of disciplines,
link |
immigration, insurance, often when I see
link |
what it boils down to is these companies
link |
are often a little bit old school
link |
in the way they organize the data.
link |
So the data is just not ready yet, it's not digitized.
link |
Do you also find yourself being in the role
link |
of an evangelist for like, let's get,
link |
organize your data, folks, and then you'll get
link |
the big benefit of TensorFlow.
link |
Do you get those, have those conversations?
link |
Yeah, yeah, you know, I get all kinds of questions there
link |
from, okay, what do I need to make this work, right?
link |
Do we really need deep learning?
link |
I mean, there are all these things,
link |
I already use this linear model, why would this help?
link |
I don't have enough data, let's say,
link |
or I wanna use machine learning,
link |
but I have no clue where to start.
link |
So it varies, that to all the way to the experts
link |
to why support very specific things, it's interesting.
link |
Is there a good answer?
link |
It boils down to oftentimes digitizing data.
link |
So whatever you want automated,
link |
whatever data you want to make prediction based on,
link |
you have to make sure that it's in an organized form.
link |
Like within the TensorFlow ecosystem,
link |
there's now, you're providing more and more data sets
link |
and more and more pre trained models.
link |
Are you finding yourself also the organizer of data sets?
link |
Yes, I think the TensorFlow data sets
link |
that we just released, that's definitely come up
link |
where people want these data sets,
link |
can we organize them and can we make that easier?
link |
So that's definitely one important thing.
link |
The other related thing I would say is I often tell people,
link |
you know what, don't think of the most fanciest thing
link |
that the newest model that you see,
link |
make something very basic work and then you can improve it.
link |
There's just lots of things you can do with it.
link |
Yeah, start with the basics, true.
link |
One of the big things that makes TensorFlow
link |
even more accessible was the appearance
link |
whenever that happened of Keras,
link |
the Keras standard sort of outside of TensorFlow.
link |
I think it was Keras on top of Tiano at first only
link |
and then Keras became on top of TensorFlow.
link |
Do you know when Keras chose to also add TensorFlow
link |
as a backend, who was the,
link |
was it just the community that drove that initially?
link |
Do you know if there was discussions, conversations?
link |
Yeah, so Francois started the Keras project
link |
before he was at Google and the first thing was Tiano.
link |
I don't remember if that was
link |
after TensorFlow was created or way before.
link |
And then at some point,
link |
when TensorFlow started becoming popular,
link |
there were enough similarities
link |
that he decided to create this interface
link |
and put TensorFlow as a backend.
link |
I believe that might still have been
link |
before he joined Google.
link |
So we weren't really talking about that.
link |
He decided on his own and thought that was interesting
link |
and relevant to the community.
link |
In fact, I didn't find out about him being at Google
link |
until a few months after he was here.
link |
He was working on some research ideas
link |
and doing Keras on his nights and weekends project.
link |
He wasn't like part of the TensorFlow.
link |
He didn't join initially.
link |
He joined research and he was doing some amazing research.
link |
He has some papers on that and research,
link |
so he's a great researcher as well.
link |
And at some point we realized,
link |
oh, he's doing this good stuff.
link |
People seem to like the API and he's right here.
link |
So we talked to him and he said,
link |
okay, why don't I come over to your team
link |
and work with you for a quarter
link |
and let's make that integration happen.
link |
And we talked to his manager and he said,
link |
sure, quarter's fine.
link |
And that quarter's been something like two years now.
link |
And so he's fully on this.
link |
So Keras got integrated into TensorFlow in a deep way.
link |
And now with 2.0, TensorFlow 2.0,
link |
sort of Keras is kind of the recommended way
link |
for a beginner to interact with TensorFlow.
link |
Which makes that initial sort of transfer learning
link |
or the basic use cases, even for an enterprise,
link |
super simple, right?
link |
That's correct, that's right.
link |
So what was that decision like?
link |
That seems like it's kind of a bold decision as well.
link |
We did spend a lot of time thinking about that one.
link |
We had a bunch of APIs, some built by us.
link |
There was a parallel layers API that we were building.
link |
And when we decided to do Keras in parallel,
link |
so there were like, okay, two things that we are looking at.
link |
And the first thing we was trying to do
link |
is just have them look similar,
link |
like be as integrated as possible,
link |
share all of that stuff.
link |
There were also like three other APIs
link |
that others had built over time
link |
because we didn't have a standard one.
link |
But one of the messages that we kept hearing
link |
from the community, okay, which one do we use?
link |
And they kept seeing like, okay,
link |
here's a model in this one and here's a model in this one,
link |
which should I pick?
link |
So that's sort of like, okay,
link |
we had to address that straight on with 2.0.
link |
The whole idea was we need to simplify.
link |
We had to pick one.
link |
Based on where we were, we were like,
link |
okay, let's see what are the people like?
link |
And Keras was clearly one that lots of people loved.
link |
There were lots of great things about it.
link |
So we settled on that.
link |
Organically, that's kind of the best way to do it.
link |
It was surprising, nevertheless,
link |
to sort of bring in an outside.
link |
I mean, there was a feeling like Keras
link |
might be almost like a competitor
link |
in a certain kind of, to TensorFlow.
link |
And in a sense, it became an empowering element
link |
Yeah, it's interesting how you can put two things together,
link |
In this case, I think Francois, the team,
link |
and a bunch of us have chatted,
link |
and I think we all want to see the same kind of things.
link |
We all care about making it easier
link |
for the huge set of developers out there,
link |
and that makes a difference.
link |
So Python has Guido van Rossum,
link |
who until recently held the position
link |
of benevolent dictator for life.
link |
All right, so there's a huge successful open source project
link |
like TensorFlow need one person who makes a final decision.
link |
So you've did a pretty successful TensorFlow Dev Summit
link |
just now, last couple of days.
link |
There's clearly a lot of different new features
link |
being incorporated, an amazing ecosystem, so on.
link |
Who's, how are those design decisions made?
link |
Is there a BDFL in TensorFlow,
link |
or is it more distributed and organic?
link |
I think it's somewhat different, I would say.
link |
I've always been involved in the key design directions,
link |
but there are lots of things that are distributed
link |
where there are a number of people, Martin Wick being one,
link |
who has really driven a lot of our open source stuff,
link |
a lot of the APIs,
link |
and there are a number of other people who've been,
link |
you know, pushed and been responsible
link |
for different parts of it.
link |
We do have regular design reviews.
link |
Over the last year, we've had a lot of
link |
we've really spent a lot of time opening up to the community
link |
and adding transparency.
link |
We're setting more processes in place,
link |
so RFCs, special interest groups,
link |
to really grow that community and scale that.
link |
I think the kind of scale that ecosystem is in,
link |
I don't think we could scale with having me
link |
as the lone point of decision maker.
link |
I got it. So, yeah, the growth of that ecosystem,
link |
maybe you can talk about it a little bit.
link |
First of all, it started with Andrej Karpathy
link |
when he first did ComNetJS.
link |
The fact that you can train and you'll network
link |
in the browser was, in JavaScript, was incredible.
link |
So now TensorFlow.js is really making that
link |
a serious, like a legit thing,
link |
a way to operate, whether it's in the backend
link |
Then there's the TensorFlow Extended, like you mentioned.
link |
There's TensorFlow Lite for mobile.
link |
And all of it, as far as I can tell,
link |
it's really converging towards being able to
link |
save models in the same kind of way.
link |
You can move around, you can train on the desktop
link |
and then move it to mobile and so on.
link |
So there's that cohesiveness.
link |
So can you maybe give me, whatever I missed,
link |
a bigger overview of the mission of the ecosystem
link |
that's trying to be built and where is it moving forward?
link |
Yeah. So in short, the way I like to think of this is
link |
our goals to enable machine learning.
link |
And in a couple of ways, you know, one is
link |
we have lots of exciting things going on in ML today.
link |
We started with deep learning,
link |
but we now support a bunch of other algorithms too.
link |
So one is to, on the research side,
link |
keep pushing on the state of the art.
link |
Can we, you know, how do we enable researchers
link |
to build the next amazing thing?
link |
So BERT came out recently, you know,
link |
it's great that people are able to do new kinds of research.
link |
And there are lots of amazing research
link |
that happens across the world.
link |
So that's one direction.
link |
The other is how do you take that across
link |
all the people outside who want to take that research
link |
and do some great things with it
link |
and integrate it to build real products,
link |
to have a real impact on people.
link |
And so if that's the other axes in some ways,
link |
you know, at a high level, one way I think about it is
link |
there are a crazy number of compute devices
link |
And we often used to think of ML and training
link |
and all of this as, okay, something you do
link |
either in the workstation or the data center or cloud.
link |
But we see things running on the phones.
link |
We see things running on really tiny chips.
link |
I mean, we had some demos at the developer summit.
link |
And so the way I think about this ecosystem is
link |
how do we help get machine learning on every device
link |
that has a compute capability?
link |
And that continues to grow and so in some ways
link |
this ecosystem is looked at, you know,
link |
various aspects of that and grown over time
link |
to cover more of those.
link |
And we continue to push the boundaries.
link |
In some areas we've built more tooling
link |
and things around that to help you.
link |
I mean, the first tool we started was TensorBoard.
link |
You wanted to learn just the training piece,
link |
the effects or TensorFlow extended
link |
to really do your entire ML pipelines.
link |
If you're, you know, care about all that production stuff,
link |
but then going to the edge,
link |
going to different kinds of things.
link |
And it's not just us now.
link |
We are a place where there are lots of libraries
link |
being built on top.
link |
So there are some for research,
link |
maybe things like TensorFlow agents
link |
or TensorFlow probability that started as research things
link |
or for researchers for focusing
link |
on certain kinds of algorithms,
link |
but they're also being deployed
link |
or used by, you know, production folks.
link |
And some have come from within Google,
link |
just teams across Google
link |
who wanted to build these things.
link |
Others have come from just the community
link |
because there are different pieces
link |
that different parts of the community care about.
link |
And I see our goal as enabling even that, right?
link |
It's not, we cannot and won't build every single thing.
link |
That just doesn't make sense.
link |
But if we can enable others to build the things
link |
that they care about, and there's a broader community
link |
that cares about that, and we can help encourage that,
link |
That really helps the entire ecosystem, not just those.
link |
One of the big things about 2.0 that we're pushing on is,
link |
okay, we have these so many different pieces, right?
link |
How do we help make all of them work well together?
link |
So there are a few key pieces there that we're pushing on,
link |
one being the core format in there
link |
and how we share the models themselves
link |
through save model and TensorFlow hub and so on.
link |
And a few of the pieces that we really put this together.
link |
I was very skeptical that that's,
link |
you know, when TensorFlow.js came out,
link |
it didn't seem, or deep learning JS as it was earlier.
link |
Yeah, that was the first.
link |
It seems like technically very difficult project.
link |
As a standalone, it's not as difficult,
link |
but as a thing that integrates into the ecosystem,
link |
it seems very difficult.
link |
So, I mean, there's a lot of aspects of this
link |
you're making look easy, but,
link |
and the technical side,
link |
how many challenges have to be overcome here?
link |
And still have to be overcome.
link |
That's the question here too.
link |
There are lots of steps to it, right?
link |
And we've iterated over the last few years,
link |
so there's a lot we've learned.
link |
I, yeah, and often when things come together well,
link |
things look easy and that's exactly the point.
link |
It should be easy for the end user,
link |
but there are lots of things that go behind that.
link |
If I think about still challenges ahead,
link |
you know, we have a lot more devices coming on board,
link |
for example, from the hardware perspective.
link |
How do we make it really easy for these vendors
link |
to integrate with something like TensorFlow, right?
link |
So there's a lot of compiler stuff
link |
that others are working on.
link |
There are things we can do in terms of our APIs
link |
and so on that we can do.
link |
TensorFlow started as a very monolithic system
link |
and to some extent it still is.
link |
There are less, lots of tools around it,
link |
but the core is still pretty large and monolithic.
link |
One of the key challenges for us to scale that out
link |
is how do we break that apart with clearer interfaces?
link |
It's, you know, in some ways it's software engineering 101,
link |
but for a system that's now four years old, I guess,
link |
or more, and that's still rapidly evolving
link |
and that we're not slowing down with,
link |
it's hard to change and modify and really break apart.
link |
It's sort of like, as people say, right,
link |
it's like changing the engine with a car running
link |
or trying to fix that.
link |
That's exactly what we're trying to do.
link |
So there's a challenge here
link |
because the downside of so many people
link |
being excited about TensorFlow
link |
and coming to rely on it in many of their applications
link |
is that you're kind of responsible,
link |
like it's the technical debt.
link |
You're responsible for previous versions
link |
to some degree still working.
link |
So when you're trying to innovate,
link |
I mean, it's probably easier
link |
to just start from scratch every few months.
link |
So do you feel the pain of that?
link |
2.0 does break some back compatibility,
link |
It seems like the conversion is pretty straightforward.
link |
Do you think that's still important
link |
given how quickly deep learning is changing?
link |
Can you just, the things that you've learned,
link |
can you just start over or is there pressure to not?
link |
It's a tricky balance.
link |
So if it was just a researcher writing a paper
link |
who a year later will not look at that code again,
link |
sure, it doesn't matter.
link |
There are a lot of production systems
link |
that rely on TensorFlow,
link |
both at Google and across the world.
link |
And people worry about this.
link |
I mean, these systems run for a long time.
link |
So it is important to keep that compatibility and so on.
link |
And yes, it does come with a huge cost.
link |
There's, we have to think about a lot of things
link |
as we do new things and make new changes.
link |
I think it's a trade off, right?
link |
You can, you might slow certain kinds of things down,
link |
but the overall value you're bringing
link |
because of that is much bigger
link |
because it's not just about breaking the person yesterday.
link |
It's also about telling the person tomorrow
link |
that, you know what, this is how we do things.
link |
We're not gonna break you when you come on board
link |
because there are lots of new people
link |
who are also gonna come on board.
link |
And, you know, one way I like to think about this,
link |
and I always push the team to think about it as well,
link |
when you wanna do new things,
link |
you wanna start with a clean slate.
link |
Design with a clean slate in mind,
link |
and then we'll figure out
link |
how to make sure all the other things work.
link |
And yes, we do make compromises occasionally,
link |
but unless you design with the clean slate
link |
and not worry about that,
link |
you'll never get to a good place.
link |
Oh, that's brilliant, so even if you are responsible
link |
when you're in the idea stage,
link |
when you're thinking of new,
link |
just put all that behind you.
link |
Okay, that's really, really well put.
link |
So I have to ask this
link |
because a lot of students, developers ask me
link |
how I feel about PyTorch versus TensorFlow.
link |
So I've recently completely switched
link |
my research group to TensorFlow.
link |
I wish everybody would just use the same thing,
link |
and TensorFlow is as close to that, I believe, as we have.
link |
But do you enjoy competition?
link |
So TensorFlow is leading in many ways,
link |
on many dimensions in terms of ecosystem,
link |
in terms of number of users,
link |
momentum, power, production levels, so on,
link |
but a lot of researchers are now also using PyTorch.
link |
Do you enjoy that kind of competition
link |
or do you just ignore it
link |
and focus on making TensorFlow the best that it can be?
link |
So just like research or anything people are doing,
link |
it's great to get different kinds of ideas.
link |
And when we started with TensorFlow,
link |
like I was saying earlier,
link |
one, it was very important
link |
for us to also have production in mind.
link |
We didn't want just research, right?
link |
And that's why we chose certain things.
link |
Now PyTorch came along and said,
link |
you know what, I only care about research.
link |
This is what I'm trying to do.
link |
What's the best thing I can do for this?
link |
And it started iterating and said,
link |
okay, I don't need to worry about graphs.
link |
Let me just run things.
link |
And I don't care if it's not as fast as it can be,
link |
but let me just make this part easy.
link |
And there are things you can learn from that, right?
link |
They, again, had the benefit of seeing what had come before,
link |
but also exploring certain different kinds of spaces.
link |
And they had some good things there,
link |
building on say things like JNR and so on before that.
link |
So competition is definitely interesting.
link |
It made us, you know,
link |
this is an area that we had thought about,
link |
like I said, way early on.
link |
Over time we had revisited this a couple of times,
link |
should we add this again?
link |
At some point we said, you know what,
link |
it seems like this can be done well,
link |
so let's try it again.
link |
And that's how we started pushing on eager execution.
link |
How do we combine those two together?
link |
Which has finally come very well together in 2.0,
link |
but it took us a while to get all the things together
link |
So let me ask, put another way,
link |
I think eager execution is a really powerful thing
link |
Do you think it wouldn't have been,
link |
you know, Muhammad Ali versus Frasier, right?
link |
Do you think it wouldn't have been added as quickly
link |
if PyTorch wasn't there?
link |
It might have taken longer.
link |
Yeah, it was, I mean,
link |
we had tried some variants of that before,
link |
so I'm sure it would have happened,
link |
but it might have taken longer.
link |
I'm grateful that TensorFlow is finally
link |
in the way they did.
link |
It's doing some incredible work last couple years.
link |
What other things that we didn't talk about
link |
are you looking forward in 2.0?
link |
That comes to mind.
link |
So we talked about some of the ecosystem stuff,
link |
making it easily accessible to Keras,
link |
Is there other things that we missed?
link |
Yeah, so I would say one is just where 2.0 is,
link |
and you know, with all the things that we've talked about,
link |
I think as we think beyond that,
link |
there are lots of other things that it enables us to do
link |
and that we're excited about.
link |
So what it's setting us up for,
link |
okay, here are these really clean APIs.
link |
We've cleaned up the surface for what the users want.
link |
What it also allows us to do a whole bunch of stuff
link |
behind the scenes once we are ready with 2.0.
link |
So for example, in TensorFlow with graphs
link |
and all the things you could do,
link |
you could always get a lot of good performance
link |
if you spent the time to tune it, right?
link |
And we've clearly shown that, lots of people do that.
link |
With 2.0, with these APIs, where we are,
link |
we can give you a lot of performance
link |
just with whatever you do.
link |
You know, because we see these, it's much cleaner.
link |
We know most people are gonna do things this way.
link |
We can really optimize for that
link |
and get a lot of those things out of the box.
link |
And it really allows us, you know,
link |
both for single machine and distributed and so on,
link |
to really explore other spaces behind the scenes
link |
after 2.0 in the future versions as well.
link |
So right now the team's really excited about that,
link |
that over time I think we'll see that.
link |
The other piece that I was talking about
link |
in terms of just restructuring the monolithic thing
link |
into more pieces and making it more modular,
link |
I think that's gonna be really important
link |
for a lot of the other people in the ecosystem,
link |
other organizations and so on that wanted to build things.
link |
Can you elaborate a little bit what you mean
link |
by making TensorFlow ecosystem more modular?
link |
So the way it's organized today is there's one,
link |
there are lots of repositories
link |
in the TensorFlow organization at GitHub.
link |
The core one where we have TensorFlow,
link |
it has the execution engine,
link |
it has the key backends for CPUs and GPUs,
link |
it has the work to do distributed stuff.
link |
And all of these just work together
link |
in a single library or binary.
link |
There's no way to split them apart easily.
link |
I mean, there are some interfaces,
link |
but they're not very clean.
link |
In a perfect world, you would have clean interfaces where,
link |
okay, I wanna run it on my fancy cluster
link |
with some custom networking,
link |
just implement this and do that.
link |
I mean, we kind of support that,
link |
but it's hard for people today.
link |
I think as we are starting to see more interesting things
link |
in some of these spaces,
link |
having that clean separation will really start to help.
link |
And again, going to the large size of the ecosystem
link |
and the different groups involved there,
link |
enabling people to evolve
link |
and push on things more independently
link |
just allows it to scale better.
link |
And by people, you mean individual developers and?
link |
And organizations.
link |
And organizations.
link |
So the hope is that everybody sort of major,
link |
I don't know, Pepsi or something uses,
link |
like major corporations go to TensorFlow to this kind of.
link |
Yeah, if you look at enterprises like Pepsi or these,
link |
I mean, a lot of them are already using TensorFlow.
link |
They are not the ones that do the development
link |
or changes in the core.
link |
Some of them do, but a lot of them don't.
link |
I mean, they touch small pieces.
link |
There are lots of these,
link |
some of them being, let's say, hardware vendors
link |
who are building their custom hardware
link |
and they want their own pieces.
link |
Or some of them being bigger companies, say, IBM.
link |
I mean, they're involved in some of our
link |
special interest groups,
link |
and they see a lot of users
link |
who want certain things and they want to optimize for that.
link |
So folks like that often.
link |
Autonomous vehicle companies, perhaps.
link |
So, yeah, like I mentioned,
link |
TensorFlow has been downloaded 41 million times,
link |
50,000 commits, almost 10,000 pull requests,
link |
and 1,800 contributors.
link |
So I'm not sure if you can explain it,
link |
but what does it take to build a community like that?
link |
In retrospect, what do you think,
link |
what is the critical thing that allowed
link |
for this growth to happen,
link |
and how does that growth continue?
link |
Yeah, yeah, that's an interesting question.
link |
I wish I had all the answers there, I guess,
link |
so you could replicate it.
link |
I think there are a number of things
link |
that need to come together, right?
link |
One, just like any new thing,
link |
it is about, there's a sweet spot of timing,
link |
what's needed, does it grow with,
link |
what's needed, so in this case, for example,
link |
TensorFlow's not just grown because it was a good tool,
link |
it's also grown with the growth of deep learning itself.
link |
So those factors come into play.
link |
Other than that, though,
link |
I think just hearing, listening to the community,
link |
what they do, what they need,
link |
being open to, like in terms of external contributions,
link |
we've spent a lot of time in making sure
link |
we can accept those contributions well,
link |
we can help the contributors in adding those,
link |
putting the right process in place,
link |
getting the right kind of community,
link |
welcoming them and so on.
link |
Like over the last year, we've really pushed on transparency,
link |
that's important for an open source project.
link |
People wanna know where things are going,
link |
and we're like, okay, here's a process
link |
where you can do that, here are our RFCs and so on.
link |
So thinking through, there are lots of community aspects
link |
that come into that you can really work on.
link |
As a small project, it's maybe easy to do
link |
because there's like two developers and you can do those.
link |
As you grow, putting more of these processes in place,
link |
thinking about the documentation,
link |
thinking about what two developers care about,
link |
what kind of tools would they want to use,
link |
all of these come into play, I think.
link |
So one of the big things I think
link |
that feeds the TensorFlow fire
link |
is people building something on TensorFlow,
link |
and implement a particular architecture
link |
that does something cool and useful,
link |
and they put that on GitHub.
link |
And so it just feeds this growth.
link |
Do you have a sense that with 2.0 and 1.0
link |
that there may be a little bit of a partitioning
link |
like there is with Python 2 and 3,
link |
that there'll be a code base
link |
and in the older versions of TensorFlow,
link |
they will not be as compatible easily?
link |
Or are you pretty confident that this kind of conversion
link |
is pretty natural and easy to do?
link |
So we're definitely working hard
link |
to make that very easy to do.
link |
There's lots of tooling that we talked about
link |
at the developer summit this week,
link |
and we'll continue to invest in that tooling.
link |
It's, you know, when you think
link |
of these significant version changes,
link |
that's always a risk,
link |
and we are really pushing hard
link |
to make that transition very, very smooth.
link |
So I think, so at some level,
link |
people wanna move and they see the value in the new thing.
link |
They don't wanna move just because it's a new thing,
link |
and some people do,
link |
but most people want a really good thing.
link |
And I think over the next few months,
link |
as people start to see the value,
link |
we'll definitely see that shift happening.
link |
So I'm pretty excited and confident
link |
that we will see people moving.
link |
As you said earlier, this field is also moving rapidly,
link |
so that'll help because we can do more things
link |
and all the new things will clearly happen in 2.x,
link |
so people will have lots of good reasons to move.
link |
So what do you think TensorFlow 3.0 looks like?
link |
Is there, are things happening so crazily
link |
that even at the end of this year
link |
seems impossible to plan for?
link |
Or is it possible to plan for the next five years?
link |
I think it's tricky.
link |
There are some things that we can expect
link |
in terms of, okay, change, yes, change is gonna happen.
link |
Are there some things gonna stick around
link |
and some things not gonna stick around?
link |
I would say the basics of deep learning,
link |
the, you know, say convolution models
link |
or the basic kind of things,
link |
they'll probably be around in some form still in five years.
link |
Will RL and GAN stay?
link |
Very likely, based on where they are.
link |
Will we have new things?
link |
Probably, but those are hard to predict.
link |
And some directionally, some things that we can see is,
link |
you know, in things that we're starting to do, right,
link |
with some of our projects right now
link |
is just 2.0 combining eager execution and graphs
link |
where we're starting to make it more like
link |
just your natural programming language.
link |
You're not trying to program something else.
link |
Similarly, with Swift for TensorFlow,
link |
we're taking that approach.
link |
Can you do something ground up, right?
link |
So some of those ideas seem like, okay,
link |
that's the right direction.
link |
In five years, we expect to see more in that area.
link |
Other things we don't know is,
link |
will hardware accelerators be the same?
link |
Will we be able to train with four bits
link |
instead of 32 bits?
link |
And I think the TPU side of things is exploring that.
link |
I mean, TPU is already on version three.
link |
It seems that the evolution of TPU and TensorFlow
link |
are sort of, they're coevolving almost in terms of
link |
both are learning from each other and from the community
link |
and from the applications
link |
where the biggest benefit is achieved.
link |
You've been trying to sort of, with Eager, with Keras,
link |
to make TensorFlow as accessible
link |
and easy to use as possible.
link |
What do you think, for beginners,
link |
is the biggest thing they struggle with?
link |
Have you encountered that?
link |
Or is basically what Keras is solving is that Eager,
link |
like we talked about?
link |
Yeah, for some of them, like you said, right,
link |
the beginners want to just be able to take
link |
they don't care if it's Inception or ResNet
link |
or something else,
link |
and do some training or transfer learning
link |
on their kind of model.
link |
Being able to make that easy is important.
link |
if you do that by providing them simple models
link |
with say, in hub or so on,
link |
they don't care about what's inside that box,
link |
but they want to be able to use it.
link |
So we're pushing on, I think, different levels.
link |
If you look at just a component that you get,
link |
which has the layers already smooshed in,
link |
the beginners probably just want that.
link |
Then the next step is, okay,
link |
look at building layers with Keras.
link |
If you go out to research,
link |
then they are probably writing custom layers themselves
link |
or doing their own loops.
link |
So there's a whole spectrum there.
link |
And then providing the pre trained models
link |
seems to really decrease the time from you trying to start.
link |
You could basically in a Colab notebook
link |
achieve what you need.
link |
So I'm basically answering my own question
link |
because I think what TensorFlow delivered on recently
link |
is trivial for beginners.
link |
So I was just wondering if there was other pain points
link |
you're trying to ease,
link |
but I'm not sure there would.
link |
No, those are probably the big ones.
link |
I see high schoolers doing a whole bunch of things now,
link |
which is pretty amazing.
link |
It's both amazing and terrifying.
link |
In a sense that when they grow up,
link |
it's some incredible ideas will be coming from them.
link |
So there's certainly a technical aspect to your work,
link |
but you also have a management aspect to your role
link |
with TensorFlow leading the project,
link |
a large number of developers and people.
link |
So what do you look for in a good team?
link |
What do you think?
link |
Google has been at the forefront of exploring
link |
what it takes to build a good team
link |
and TensorFlow is one of the most cutting edge technologies
link |
So in this context, what do you think makes for a good team?
link |
It's definitely something I think a favorite about.
link |
I think in terms of the team being able
link |
to deliver something well,
link |
one of the things that's important is a cohesion
link |
So being able to execute together in doing things
link |
that's not an end, like at this scale,
link |
an individual engineer can only do so much.
link |
There's a lot more that they can do together,
link |
even though we have some amazing superstars across Google
link |
and in the team, but there's, you know,
link |
often the way I see it as the product
link |
of what the team generates is way larger
link |
than the whole or the individual put together.
link |
And so how do we have all of them work together,
link |
the culture of the team itself,
link |
hiring good people is important.
link |
But part of that is it's not just that,
link |
okay, we hire a bunch of smart people
link |
and throw them together and let them do things.
link |
It's also people have to care about what they're building,
link |
people have to be motivated for the right kind of things.
link |
That's often an important factor.
link |
And, you know, finally, how do you put that together
link |
with a somewhat unified vision of where we wanna go?
link |
So are we all looking in the same direction
link |
or each of us going all over?
link |
And sometimes it's a mix.
link |
Google's a very bottom up organization in some sense,
link |
also research even more so, and that's how we started.
link |
But as we've become this larger product and ecosystem,
link |
I think it's also important to combine that well
link |
with a mix of, okay, here's the direction we wanna go in.
link |
There is exploration we'll do around that,
link |
but let's keep staying in that direction,
link |
not just all over the place.
link |
And is there a way you monitor the health of the team?
link |
Sort of like, is there a way you know you did a good job?
link |
Like, I mean, you're sort of, you're saying nice things,
link |
but it's sometimes difficult to determine how aligned.
link |
Because it's not binary.
link |
It's not like there's tensions and complexities and so on.
link |
And the other element of the mission of superstars,
link |
there's so much, even at Google,
link |
such a large percentage of work
link |
is done by individual superstars too.
link |
So there's a, and sometimes those superstars
link |
can be against the dynamic of a team and those tensions.
link |
I mean, I'm sure in TensorFlow it might be
link |
a little bit easier because the mission of the project
link |
is so sort of beautiful.
link |
You're at the cutting edge, so it's exciting.
link |
But have you had struggle with that?
link |
Has there been challenges?
link |
There are always people challenges
link |
in different kinds of ways.
link |
That said, I think we've been what's good
link |
about getting people who care and are, you know,
link |
have the same kind of culture,
link |
and that's Google in general to a large extent.
link |
But also, like you said, given that the project
link |
has had so many exciting things to do,
link |
there's been room for lots of people
link |
to do different kinds of things and grow,
link |
which does make the problem a bit easier, I guess.
link |
And it allows people, depending on what they're doing,
link |
if there's room around them, then that's fine.
link |
But yes, we do care about whether a superstar or not,
link |
that they need to work well with the team across Google.
link |
That's interesting to hear.
link |
So it's like superstar or not,
link |
the productivity broadly is about the team.
link |
I mean, they might add a lot of value,
link |
but if they're hurting the team, then that's a problem.
link |
So in hiring engineers, it's so interesting, right,
link |
the hiring process.
link |
What do you look for?
link |
How do you determine a good developer
link |
or a good member of a team
link |
from just a few minutes or hours together?
link |
Again, no magic answers, I'm sure.
link |
Yeah, I mean, Google has a hiring process
link |
that we've refined over the last 20 years, I guess,
link |
and that you've probably heard and seen a lot about.
link |
So we do work with the same hiring process
link |
and that's really helped.
link |
For me in particular, I would say,
link |
in addition to the core technical skills,
link |
what does matter is their motivation
link |
in what they wanna do.
link |
Because if that doesn't align well
link |
with where we wanna go,
link |
that's not gonna lead to long term success
link |
for either them or the team.
link |
And I think that becomes more important
link |
the more senior the person is,
link |
but it's important at every level.
link |
Like even the junior most engineer,
link |
if they're not motivated to do well
link |
at what they're trying to do,
link |
however smart they are,
link |
it's gonna be hard for them to succeed.
link |
Does the Google hiring process touch on that passion?
link |
So like trying to determine,
link |
because I think as far as I understand,
link |
maybe you can speak to it,
link |
that the Google hiring process sort of helps
link |
in the initial like determines the skill set there,
link |
is your puzzle solving ability,
link |
problem solving ability good?
link |
But like, I'm not sure,
link |
but it seems that the determining
link |
whether the person is like fire inside them,
link |
that burns to do anything really,
link |
it doesn't really matter.
link |
It's just some cool stuff,
link |
Is that something that ultimately ends up
link |
when they have a conversation with you
link |
or once it gets closer to the team?
link |
So one of the things we do have as part of the process
link |
is just a culture fit,
link |
like part of the interview process itself,
link |
in addition to just the technical skills
link |
and each engineer or whoever the interviewer is,
link |
is supposed to rate the person on the culture
link |
and the culture fit with Google and so on.
link |
So that is definitely part of the process.
link |
Now, there are various kinds of projects
link |
and different kinds of things.
link |
So there might be variants
link |
and of the kind of culture you want there and so on.
link |
And yes, that does vary.
link |
TensorFlow has always been a fast moving project
link |
and we want people who are comfortable with that.
link |
But at the same time now, for example,
link |
we are at a place where we are also very full fledged product
link |
and we wanna make sure things that work
link |
really, really work, right?
link |
You can't cut corners all the time.
link |
So balancing that out and finding the people
link |
who are the right fit for those is important.
link |
And I think those kinds of things do vary a bit
link |
across projects and teams and product areas across Google.
link |
And so you'll see some differences there
link |
in the final checklist.
link |
But a lot of the core culture,
link |
it comes along with just the engineering excellence
link |
What is the hardest part of your job?
link |
I'll take your pick, I guess.
link |
It's fun, I would say, right?
link |
I mean, lots of things at different times.
link |
I think that does vary.
link |
So let me clarify that difficult things are fun
link |
when you solve them, right?
link |
So it's fun in that sense.
link |
I think the key to a successful thing across the board
link |
and in this case, it's a large ecosystem now,
link |
but even a small product,
link |
is striking that fine balance
link |
across different aspects of it.
link |
Sometimes it's how fast do you go
link |
versus how perfect it is.
link |
Sometimes it's how do you involve this huge community?
link |
Who do you involve or do you decide,
link |
okay, now is not a good time to involve them
link |
because it's not the right fit.
link |
Sometimes it's saying no to certain kinds of things.
link |
Those are often the hard decisions.
link |
Some of them you make quickly
link |
because you don't have the time.
link |
Some of them you get time to think about them,
link |
but they're always hard.
link |
So both choices are pretty good, those decisions.
link |
What about deadlines?
link |
Is this, do you find TensorFlow,
link |
to be driven by deadlines
link |
to a degree that a product might?
link |
Or is there still a balance to where it's less deadline?
link |
You had the Dev Summit today
link |
that came together incredibly.
link |
Looked like there's a lot of moving pieces and so on.
link |
So did that deadline make people rise to the occasion
link |
releasing TensorFlow 2.0 alpha?
link |
I'm sure that was done last minute as well.
link |
I mean, up to the last point.
link |
Again, it's one of those things
link |
that you need to strike the good balance.
link |
There's some value that deadlines bring
link |
that does bring a sense of urgency
link |
to get the right things together.
link |
Instead of getting the perfect thing out,
link |
you need something that's good and works well.
link |
And the team definitely did a great job
link |
in putting that together.
link |
So I was very amazed and excited
link |
by everything how that came together.
link |
That said, across the year,
link |
we try not to put out official deadlines.
link |
We focus on key things that are important,
link |
figure out how much of it's important.
link |
And we are developing in the open,
link |
both internally and externally,
link |
everything's available to everybody.
link |
So you can pick and look at where things are.
link |
We do releases at a regular cadence.
link |
So fine, if something doesn't necessarily end up
link |
this month, it'll end up in the next release
link |
in a month or two.
link |
And that's okay, but we want to keep moving
link |
as fast as we can in these different areas.
link |
Because we can iterate and improve on things,
link |
sometimes it's okay to put things out
link |
that aren't fully ready.
link |
We'll make sure it's clear that okay,
link |
this is experimental, but it's out there
link |
if you want to try and give feedback.
link |
That's very, very useful.
link |
I think that quick cycle and quick iteration is important.
link |
That's what we often focus on rather than
link |
here's a deadline where you get everything else.
link |
Is 2.0, is there pressure to make that stable?
link |
Or like, for example, WordPress 5.0 just came out
link |
and there was no pressure to,
link |
it was a lot of build updates delivered way too late,
link |
but, and they said, okay, well,
link |
but we're gonna release a lot of updates
link |
really quickly to improve it.
link |
Do you see TensorFlow 2.0 in that same kind of way
link |
or is there this pressure to once it hits 2.0,
link |
once you get to the release candidate
link |
and then you get to the final,
link |
that's gonna be the stable thing?
link |
So it's gonna be stable in,
link |
just like when NodeX was where every API that's there
link |
is gonna remain in work.
link |
It doesn't mean we can't change things under the covers.
link |
It doesn't mean we can't add things.
link |
So there's still a lot more for us to do
link |
and we'll continue to have more releases.
link |
So in that sense, there's still,
link |
I don't think we'll be done in like two months
link |
when we release this.
link |
I don't know if you can say, but is there,
link |
there's not external deadlines for TensorFlow 2.0,
link |
but is there internal deadlines,
link |
the artificial or otherwise,
link |
that you're trying to set for yourself
link |
or is it whenever it's ready?
link |
So we want it to be a great product, right?
link |
And that's a big important piece for us.
link |
TensorFlow's already out there.
link |
We have 41 million downloads for 1.0 X.
link |
So it's not like we have to have this.
link |
So it's not like, a lot of the features
link |
that we've really polishing
link |
and putting them together are there.
link |
We don't have to rush that just because.
link |
So in that sense, we wanna get it right
link |
and really focus on that.
link |
That said, we have said that we are looking
link |
to get this out in the next few months,
link |
in the next quarter.
link |
And as far as possible,
link |
we'll definitely try to make that happen.
link |
Yeah, my favorite line was, spring is a relative concept.
link |
Spoken like a true developer.
link |
So something I'm really interested in
link |
and your previous line of work is,
link |
before TensorFlow, you led a team at Google on search ads.
link |
I think this is a very interesting topic
link |
on every level, on a technical level,
link |
because at their best, ads connect people
link |
to the things they want and need.
link |
So, and at their worst, they're just these things
link |
that annoy the heck out of you
link |
to the point of ruining the entire user experience
link |
of whatever you're actually doing.
link |
So they have a bad rep, I guess.
link |
And on the other end, so that this connecting users
link |
to the thing they need and want
link |
is a beautiful opportunity for machine learning to shine.
link |
Like huge amounts of data that's personalized
link |
and you kind of map to the thing
link |
they actually want won't get annoyed.
link |
So what have you learned from this,
link |
Google that's leading the world in this aspect,
link |
what have you learned from that experience
link |
and what do you think is the future of ads?
link |
Take you back to that.
link |
Yeah, yes, it's been a while,
link |
but I totally agree with what you said.
link |
I think the search ads, the way it was always looked at
link |
and I believe it still is,
link |
is it's an extension of what search is trying to do.
link |
And the goal is to make the information
link |
and make the world's information accessible.
link |
That's it's not just information,
link |
but maybe products or other things that people care about.
link |
And so it's really important for them to align
link |
with what the users need.
link |
And in search ads, there's a minimum quality level
link |
before that ad would be shown.
link |
If you don't have an ad that hits that quality,
link |
but it will not be shown even if we have it
link |
and okay, maybe we lose some money there, that's fine.
link |
That is really, really important.
link |
And I think that that is something I really liked
link |
about being there.
link |
Advertising is a key part.
link |
I mean, as a model, it's been around for ages, right?
link |
It's not a new model, it's been adapted to the web
link |
and became a core part of search
link |
and many other search engines across the world.
link |
And I do hope, like you said,
link |
there are aspects of ads that are annoying
link |
and I go to a website and if it just keeps popping
link |
an ad in my face not to let me read,
link |
that's gonna be annoying clearly.
link |
So I hope we can strike that balance
link |
between showing a good ad where it's valuable to the user
link |
and provides the monetization to the service.
link |
And this might be search, this might be a website,
link |
all of these, they do need the monetization
link |
for them to provide that service.
link |
But if it's done in a good balance between
link |
showing just some random stuff that's distracting
link |
versus showing something that's actually valuable.
link |
So do you see it moving forward as to continue
link |
being a model that funds businesses like Google,
link |
that's a significant revenue stream?
link |
Because that's one of the most exciting things
link |
but also limiting things in the internet
link |
is nobody wants to pay for anything.
link |
And advertisements, again, coupled at their best,
link |
are actually really useful and not annoying.
link |
Do you see that continuing and growing and improving
link |
or is there, do you see sort of more Netflix type models
link |
where you have to start to pay for content?
link |
I think it's a mix.
link |
I think it's gonna take a long while for everything
link |
to be paid on the internet, if at all, probably not.
link |
I mean, I think there's always gonna be things
link |
that are sort of monetized with things like ads.
link |
But over the last few years, I would say
link |
we've definitely seen that transition towards
link |
more paid services across the web
link |
and people are willing to pay for them
link |
because they do see the value.
link |
I mean, Netflix is a great example.
link |
I mean, we have YouTube doing things.
link |
People pay for the apps they buy.
link |
More people I find are willing to pay for newspaper content
link |
for the good news websites across the web.
link |
That wasn't the case a few years,
link |
even a few years ago, I would say.
link |
And I just see that change in myself as well
link |
and just lots of people around me.
link |
So definitely hopeful that we'll transition
link |
to that mix model where maybe you get
link |
to try something out for free, maybe with ads,
link |
but then there's a more clear revenue model
link |
that sort of helps go beyond that.
link |
So speaking of revenue, how is it that a person
link |
can use the TPU in a Google call app for free?
link |
So what's the, I guess the question is,
link |
what's the future of TensorFlow in terms of empowering,
link |
say, a class of 300 students?
link |
And I'm asked by MIT, what is going to be the future
link |
of them being able to do their homework in TensorFlow?
link |
Like, where are they going to train these networks, right?
link |
What's that future look like with TPUs,
link |
with cloud services, and so on?
link |
I think a number of things there.
link |
I mean, any TensorFlow open source,
link |
you can run it wherever, you can run it on your desktop
link |
and your desktops always keep getting more powerful,
link |
so maybe you can do more.
link |
My phone is like, I don't know how many times
link |
more powerful than my first desktop.
link |
You'll probably train it on your phone though,
link |
yeah, that's true.
link |
Right, so in that sense, the power you have
link |
in your hands is a lot more.
link |
Clouds are actually very interesting from, say,
link |
students or courses perspective,
link |
because they make it very easy to get started.
link |
I mean, Colab, the great thing about it is,
link |
go to a website and it just works.
link |
No installation needed, nothing to,
link |
you're just there and things are working.
link |
That's really the power of cloud as well.
link |
And so I do expect that to grow.
link |
Again, Colab is a free service.
link |
It's great to get started, to play with things,
link |
to explore things.
link |
That said, with free, you can only get so much.
link |
So just like we were talking about,
link |
free versus paid, yeah, there are services
link |
you can pay for and get a lot more.
link |
Great, so if I'm a complete beginner
link |
interested in machine learning and TensorFlow,
link |
Probably start with going to our website
link |
and playing there.
link |
So just go to TensorFlow.org and start clicking on things.
link |
Yep, check out tutorials and guides.
link |
There's stuff you can just click there
link |
and go to a Colab and do things.
link |
No installation needed, you can get started right there.
link |
Okay, awesome, Rajit, thank you so much for talking today.
link |
Thank you, Lex, it was great.