back to indexRajat Monga: TensorFlow | Lex Fridman Podcast #22
link |
The following is a conversation with Rajat Manga.
link |
He's an engineering director at Google,
link |
leading the TensorFlow team.
link |
TensorFlow is an open source library
link |
at the center of much of the work going on in the world
link |
in deep learning, both the cutting edge research
link |
and the large scale application of learning based approaches.
link |
But it's quickly becoming much more
link |
than a software library.
link |
It's now an ecosystem of tools for the deployment
link |
of machine learning in the cloud, on the phone,
link |
in the browser, on both generic and specialized hardware.
link |
TPU, GPU, and so on.
link |
Plus, there's a big emphasis on growing
link |
a passionate community of developers.
link |
Rajat, Jeff Dean, and a large team of engineers at Google
link |
Brain are working to define the future of machine learning
link |
with TensorFlow 2.0, which is now in alpha.
link |
I think the decision to open source TensorFlow
link |
is a definitive moment in the tech industry.
link |
It showed that open innovation can be successful
link |
and inspire many companies to open source their code,
link |
to publish, and in general engage in the open exchange
link |
This conversation is part of the artificial intelligence
link |
If you enjoy it, subscribe on YouTube, iTunes,
link |
or simply connect with me on Twitter
link |
at Lex Friedman, spelled FRID.
link |
And now, here's my conversation with Rajat Manga.
link |
You were involved with Google Brain since its start in 2011
link |
It started with disbelief, the proprietary machine learning
link |
library, and turned into TensorFlow 2014,
link |
the open source library.
link |
So what were the early days of Google Brain like?
link |
What were the goals, the missions?
link |
How do you even proceed forward once there's
link |
so much possibilities before you?
link |
It was interesting back then when I started,
link |
or when you were even just talking about it.
link |
The idea of deep learning was interesting
link |
and intriguing in some ways.
link |
It hadn't yet taken off, but it held some promise.
link |
It had shown some very promising and early results.
link |
I think the idea where Andrew and Jeff had started
link |
was what if we can take this, what people are doing in research,
link |
and scale it to what Google has in terms of the compute power,
link |
and also put that kind of data together, what does it mean?
link |
And so far, the results had been if you scale the computer,
link |
scale the data, it does better, and would that work?
link |
And so that was the first year or two.
link |
Can we prove that outright?
link |
And with disbelief, when we started the first year,
link |
we got some early wins, which is always great.
link |
What were the wins like?
link |
What was the wins where there are some problems to this?
link |
This is going to be good.
link |
I think the two early wins were one was speech
link |
that we collaborated very closely with the speech research
link |
team, who was also getting interested in this.
link |
And the other one was on images where
link |
the cat paper, as we call it, that was covered by a lot of folks.
link |
And the birth of Google Brain was around neural networks.
link |
So it was deep learning from the very beginning.
link |
That was the whole mission.
link |
So in terms of scale, what was the dream
link |
of what this could become?
link |
Were there echoes of this open source TensorFlow community
link |
that might be brought in?
link |
Was there a sense of TPUs?
link |
Was there a sense of machine learning
link |
is now going to be at the core of the entire company?
link |
Is going to grow into that direction?
link |
Yeah, I think so that was interesting.
link |
And if I think back to 2012 or 2011,
link |
and first was can we scale it in the year or so,
link |
we had started scaling it to hundreds and thousands
link |
In fact, we had some runs even going to 10,000 machines.
link |
And all of those shows great promise.
link |
In terms of machine learning at Google,
link |
the good thing was Google's been doing machine learning
link |
Deep learning was new.
link |
But as we scale this up, we showed that, yes, that was
link |
possible, and it was going to impact lots of things.
link |
Like, we started seeing real products wanting to use this.
link |
Again, speech was the first.
link |
There were image things that photos came out of
link |
in many other products as well.
link |
So that was exciting.
link |
As we went into with that a couple of years,
link |
externally also academia started to,
link |
there was lots of push on, OK, deep learning's
link |
interesting, we should be doing more, and so on.
link |
And so by 2014, we were looking at, OK, this is a big thing.
link |
It's going to grow.
link |
And not just internally, externally as well.
link |
Yes, maybe Google's ahead of where everybody is,
link |
but there's a lot to do.
link |
So a lot of this start to make sense and come together.
link |
So the decision to open source, I was just chatting with Chris
link |
Flattner about this, the decision to go open source
link |
with TensorFlow, I would say for me personally,
link |
seems to be one of the big seminal moments in all
link |
of software engineering ever.
link |
I think that when a large company like Google
link |
decides to take a large project that many lawyers might argue
link |
has a lot of IP, just decide to go open source with it.
link |
And in so doing, lead the entire world in saying,
link |
you know what, open innovation is a pretty powerful thing.
link |
And it's OK to do.
link |
That was, I mean, that's an incredible moment in time.
link |
So do you remember those discussions happening?
link |
Are there open source should be happening?
link |
What was that like?
link |
I would say, I think, so the initial idea came from Jeff,
link |
who was a big proponent of this.
link |
I think it came off of two big things.
link |
One was research wise, we were a research group.
link |
We were putting all our research out there if you wanted to.
link |
We were building on other's research,
link |
and we wanted to push the state of the art forward.
link |
And part of that was to share the research.
link |
That's how I think deep learning and machine learning
link |
has really grown so fast.
link |
So the next step was, OK, now word software
link |
And it seemed like they were existing a few libraries
link |
out there, Tiano being one, Torch being another,
link |
But they were all done by academia,
link |
and so the level was significantly different.
link |
The other one was, from a software perspective,
link |
Google had done lots of software that we used internally.
link |
And we published papers.
link |
Often there was an open source project
link |
that came out of that, that somebody else
link |
picked up that paper and implemented,
link |
and they were very successful.
link |
Back then, it was like, OK, there's
link |
Hadoop, which has come off of tech that we've built.
link |
We know that tech we've built is way better
link |
for a number of different reasons.
link |
We've invested a lot of effort in that.
link |
And turns out, we have Google Cloud,
link |
and we are now not really providing our tech,
link |
but we are saying, OK, we have Bigtable, which
link |
is the original thing.
link |
We are going to now provide HBase APIs on top of that, which
link |
isn't as good, but that's what everybody's used to.
link |
So there's like, can we make something that is better
link |
and really just provide?
link |
Helps the community in lots of ways,
link |
but it also helps push the right, a good standard forward.
link |
So how does Cloud fit into that?
link |
There's a TensorFlow open source library.
link |
And how does the fact that you can
link |
use so many of the resources that Google provides
link |
and the Cloud fit into that strategy?
link |
So TensorFlow itself is open, and you can use it anywhere.
link |
And we want to make sure that continues to be the case.
link |
On Google Cloud, we do make sure that there's
link |
lots of integrations with everything else,
link |
and we want to make sure that it works really, really well there.
link |
You're leading the TensorFlow effort.
link |
Can you tell me the history and the timeline of TensorFlow
link |
project in terms of major design decisions,
link |
like the open source decision, but really, what to include
link |
There's this incredible ecosystem that I'd
link |
like to talk about, there's all these parts.
link |
But if you just some sample moments that
link |
defined what TensorFlow eventually became through its,
link |
I don't know if you were allowed to say history when it's just,
link |
but in deep learning, everything moves so fast
link |
in just a few years, it's already history.
link |
So looking back, we were building TensorFlow.
link |
I guess we open sourced it in 2015, November 2015.
link |
We started on it in summer of 2014, I guess.
link |
And somewhere like three to six late 2014,
link |
by then we had decided that, OK, there's
link |
a high likelihood we'll open source it.
link |
So we started thinking about that and making sure
link |
that we're heading down that path.
link |
At that point, by that point, we'd
link |
seen a few lots of different use cases at Google.
link |
So there were things like, OK, yes,
link |
you want to run in at large scale in the data center.
link |
Yes, we need to support different kind of hardware.
link |
We had GPUs at that point.
link |
We had our first GPU at that point
link |
or was about to come out roughly around that time.
link |
So the design included those.
link |
We had started to push on mobile.
link |
So we were running models on mobile.
link |
At that point, people were customizing code.
link |
So we wanted to make sure TensorFlow could support that
link |
as well so that that became part of that overall
link |
When you say mobile, you mean like pretty complicated
link |
algorithms of running on the phone?
link |
So when you have a model that you
link |
deploy on the phone and run it there, right?
link |
So already at that time, there was ideas of running machine
link |
learning on the phone.
link |
We already had a couple of products
link |
that were doing that by then.
link |
And in those cases, we had basically
link |
customized handcrafted code or some internal libraries
link |
So I was actually at Google during this time in a parallel,
link |
I guess, universe.
link |
But we were using Theano and CAFE.
link |
Was there some degree to which you were bouncing,
link |
like trying to see what CAFE was offering people,
link |
trying to see what Theano was offering
link |
that you want to make sure you're delivering on whatever that
link |
is, perhaps the Python part of thing.
link |
Maybe did that influence any design decisions?
link |
So when we built this belief, and some of that
link |
was in parallel with some of these libraries
link |
coming up, I mean, Theano itself is older.
link |
But we were building this belief focused on our internal thing
link |
because our systems were very different.
link |
By the time we got to this, we looked
link |
at a number of libraries that were out there.
link |
Theano, there were folks in the group
link |
who had experience with Torch, with Lua.
link |
There were folks here who had seen CAFE.
link |
I mean, actually, Yang Cheng was here as well.
link |
There's what other libraries?
link |
I think we looked at a number of things.
link |
Might even have looked at Jane and her back then.
link |
I'm trying to remember if it was there.
link |
In fact, yeah, we did discuss ideas around, OK,
link |
should we have a graph or not?
link |
And they were supporting all these together
link |
was definitely, you know, there were key decisions
link |
We had seen limitations in our prior disbelief things.
link |
A few of them were just in terms of research
link |
was moving so fast.
link |
We wanted the flexibility.
link |
We want the hardware was changing fast.
link |
We expected to change that so that those probably were two
link |
And yeah, I think the flexibility in terms
link |
of being able to express all kinds of crazy things
link |
was definitely a big one then.
link |
So what the graph decisions, though,
link |
with moving towards TensorFlow 2.0, there's more,
link |
by default, there'll be eager execution.
link |
So sort of hiding the graph a little bit
link |
because it's less intuitive in terms of the way
link |
people develop and so on.
link |
What was that discussion like with in terms of using graphs?
link |
It seemed it's kind of the theano way.
link |
Did it seem the obvious choice?
link |
So I think where it came from was our disbelief,
link |
had a graph like thing as well.
link |
It wasn't a general graph.
link |
It was more like a straight line thing.
link |
More like what you might think of Cafe,
link |
I guess, in that sense.
link |
And we always cared about the production stuff.
link |
Even with disbelief, we were deploying a whole bunch of stuff
link |
So graph did come from that when we thought of, OK,
link |
should we do that in Python and we experimented with some ideas
link |
where it looked a lot simpler to use,
link |
but not having a graph meant, OK, how do you deploy now?
link |
So that was probably what tilted the balance for us.
link |
And eventually, we ended up with the graph.
link |
And I guess the question there is, did you?
link |
I mean, production seems to be the really good thing to focus on.
link |
But did you even anticipate the other side of it
link |
where there could be, what is it?
link |
What are the numbers?
link |
Something crazy, 41 million downloads?
link |
I mean, was that even like a possibility in your mind
link |
that it would be as popular as it became?
link |
So I think we did see a need for this a lot
link |
from the research perspective and early days of deep learning
link |
No, I don't think I imagine this number then.
link |
It seemed like there's a potential future where lots more people
link |
would be doing this.
link |
And how do we enable that?
link |
I would say this kind of growth, I probably
link |
started seeing somewhat after the open sourcing where it was
link |
like, OK, deep learning is actually
link |
growing way faster for a lot of different reasons.
link |
And we are in just the right place to push on that
link |
and leverage that and deliver on lots of things
link |
So what changed once the open source?
link |
Like how this incredible amount of attention
link |
from a global population of developers,
link |
how did the projects start changing?
link |
I don't even actually remember it during those times.
link |
I know looking now, there's really good documentation.
link |
There's an ecosystem of tools.
link |
There's a YouTube channel now.
link |
It's very community driven.
link |
Back then, I guess 0.1 version.
link |
Is that the version?
link |
I think we called it 0.6 or 5, something like that.
link |
Something like that.
link |
What changed leading into 1.0?
link |
I think we've gone through a few things there.
link |
When we started out, when we first came out,
link |
people loved the documentation we have.
link |
Because it was just a huge step up from everything else.
link |
Because all of those were academic projects, people
link |
don't think about documentation.
link |
I think what that changed was instead of deep learning
link |
being a research thing, some people who were just developers
link |
could now suddenly take this out and do
link |
some interesting things with it.
link |
Who had no clue what machine learning was before then.
link |
And that, I think, really changed
link |
how things started to scale up in some ways and pushed on it.
link |
Over the next few months, as we looked at,
link |
how do we stabilize things?
link |
As we look at not just researchers,
link |
now we want stability.
link |
People want to deploy things.
link |
That's how we started planning for 1.0.
link |
And there are certain needs for that perspective.
link |
And so, again, documentation comes up,
link |
designs, more kinds of things to put that together.
link |
And so that was exciting to get that to a stage where
link |
more and more enterprises wanted to buy in and really
link |
And I think post 1.0 and with the next few releases,
link |
their enterprise adoption also started to take off.
link |
I would say between the initial release and 1.0,
link |
it was, OK, researchers, of course.
link |
Then a lot of hobbies and early interest,
link |
people excited about this who started to get on board.
link |
And then over the 1.x thing, lots of enterprises.
link |
I imagine anything that's below 1.0
link |
gets pressured to be enterprise problem or something
link |
And do you have a sense now that TensorFlow is stable?
link |
It feels like deep learning, in general,
link |
is extremely dynamic field.
link |
So much is changing.
link |
Do you have a, and TensorFlow has been growing incredibly.
link |
Do you have a sense of stability at the helm of this?
link |
I mean, I know you're in the midst of it.
link |
I think in the midst of it, it's often easy to forget what
link |
an enterprise wants and what some of the people on that side
link |
There are still people running models
link |
that are three years old, four years old.
link |
So inception is still used by tons of people.
link |
Even less than 50 is what, a couple of years old now or more.
link |
But there are tons of people who use that, and they're fine.
link |
They don't need the last couple of bits of performance or quality.
link |
They want some stability in things that just work.
link |
And so there is value in providing that with that kind
link |
of stability and making it really simpler,
link |
because that allows a lot more people to access it.
link |
And then there's the research crowd, which wants, OK,
link |
they want to do these crazy things exactly like you're
link |
saying, not just deep learning in the straight up models
link |
that used to be there.
link |
They want RNNs, and even RNNs are maybe old.
link |
They are transformers now, and now it
link |
needs to combine with RL and GANs and so on.
link |
So there's definitely that area, the boundary that's
link |
shifting and pushing the state of the art.
link |
But I think there's more and more of the past
link |
that's much more stable.
link |
And even stuff that was two, three years old
link |
is very, very usable by lots of people.
link |
So that part makes it a lot easier.
link |
So I imagine maybe you can correct me if I'm wrong.
link |
One of the biggest use cases is essentially
link |
taking something like ResNet 50 and doing
link |
some kind of transfer learning on a very particular problem
link |
It's basically probably what majority of the world does.
link |
And you want to make that as easy as possible.
link |
So I would say, for the hobbyist perspective,
link |
that's the most common case.
link |
In fact, the apps on phones and stuff
link |
that you'll see, the early ones, that's the most common case.
link |
I would say there are a couple of reasons for that.
link |
One is that everybody talks about that.
link |
It looks great on slides.
link |
That's a great presentation.
link |
What enterprises want is that is part of it,
link |
but that's not the big thing.
link |
Enterprises really have data that they
link |
want to make predictions on.
link |
This is often what they used to do with the people who
link |
were doing ML was just regression models,
link |
linear regression, logistic regression, linear models,
link |
or maybe gradient booster trees and so on.
link |
Some of them still benefit from deep learning,
link |
but they weren't that that's the bread and butter,
link |
like the structured data and so on.
link |
So depending on the audience you look at,
link |
they're a little bit different.
link |
And they just have, I mean, the best of enterprise
link |
probably just has a very large data set
link |
where deep learning can probably shine.
link |
And then I think the other pieces
link |
that they wanted, again, to point out
link |
that the developer summit we put together
link |
is that the whole TensorFlow Extended
link |
piece, which is the entire pipeline,
link |
they care about stability across doing their entire thing.
link |
They want simplicity across the entire thing.
link |
I don't need to just train a model.
link |
I need to do that every day again, over and over again.
link |
I wonder to which degree you have a role in, I don't know.
link |
So I teach a course on deep learning.
link |
I have people like lawyers come up to me and say,
link |
when is machine learning going to enter legal,
link |
The same thing in all kinds of disciplines, immigration,
link |
Often when I see what it boils down to is these companies
link |
are often a little bit old school in the way
link |
they organize the data.
link |
So the data is just not ready yet.
link |
It's not digitized.
link |
Do you also find yourself being in the role of an evangelist
link |
for let's organize your data, folks,
link |
and then you'll get the big benefit of TensorFlow?
link |
Do you have those conversations?
link |
Yeah, I get all kinds of questions there from, OK,
link |
what do I need to make this work, right?
link |
Do we really need deep learning?
link |
I mean, there are all these things.
link |
I already used this linear model.
link |
Why would this help?
link |
I don't have enough data, let's say.
link |
Or I want to use machine learning,
link |
but I have no clue where to start.
link |
So it's a great start to all the way to the experts
link |
who wise were very specific things, so it's interesting.
link |
Is there a good answer?
link |
It boils down to oftentimes digitizing data.
link |
So whatever you want automated, whatever data
link |
you want to make prediction based on,
link |
you have to make sure that it's in an organized form.
link |
Like with an intensive flow ecosystem,
link |
there's now you're providing more and more data
link |
sets and more and more pretrained models.
link |
Are you finding yourself also the organizer of data sets?
link |
Yes, I think with TensorFlow data sets
link |
that we just released, that's definitely come up where people
link |
want these data sets.
link |
Can we organize them and can we make that easier?
link |
So that's definitely one important thing.
link |
The other related thing I would say is I often tell people,
link |
you know what, don't think of the most fanciest thing
link |
that the newest model that you see.
link |
Make something very basic work, and then
link |
you can improve it.
link |
There's just lots of things you can do with it.
link |
Yeah, start with the basics.
link |
One of the big things that makes TensorFlow even more
link |
accessible was the appearance, whenever
link |
that happened, of Keras, the Keras standard outside of TensorFlow.
link |
I think it was Keras on top of Tiano at first only,
link |
and then Keras became on top of TensorFlow.
link |
Do you know when Keras chose to also add TensorFlow as a back end,
link |
who was it just the community that drove that initially?
link |
Do you know if there was discussions, conversations?
link |
Yeah, so Franco started the Keras project
link |
before he was at Google, and the first thing was Tiano.
link |
I don't remember if that was after TensorFlow
link |
was created or way before.
link |
And then at some point, when TensorFlow
link |
started becoming popular, there were enough similarities
link |
that he decided to create this interface
link |
and put TensorFlow as a back end.
link |
I believe that might still have been before he joined Google.
link |
So we weren't really talking about that.
link |
He decided on his own and thought that was interesting
link |
and relevant to the community.
link |
In fact, I didn't find out about him being at Google
link |
until a few months after he was here.
link |
He was working on some research ideas.
link |
And doing Keras and his nights and weekends project and stuff.
link |
I wish this thing.
link |
So he wasn't part of the TensorFlow.
link |
He didn't join initially.
link |
He joined research, and he was doing some amazing research.
link |
He has some papers on that and research.
link |
He's a great researcher as well.
link |
And at some point, we realized, oh, he's doing this good stuff.
link |
People seem to like the API, and he's right here.
link |
So we talked to him, and he said, OK,
link |
why don't I come over to your team
link |
and work with you for a quarter?
link |
And let's make that integration happen.
link |
And we talked to his manager, and he said, sure,
link |
what, quarter's fine.
link |
And that quarter's been something like two years now.
link |
So he's fully on this.
link |
So Keras got integrated into TensorFlow in a deep way.
link |
And now with TensorFlow 2.0, Keras
link |
is kind of the recommended way for a beginner
link |
to interact with TensorFlow, which
link |
makes that initial sort of transfer learning
link |
or the basic use cases, even for an enterprise,
link |
super simple, right?
link |
So what was that decision like?
link |
That seems like it's kind of a bold decision as well.
link |
We did spend a lot of time thinking about that one.
link |
We had a bunch of APIs some bit by us.
link |
There was a parallel layers API that we were building
link |
and when we decided to do Keras in parallel,
link |
so they were like, OK, two things that we are looking at.
link |
And the first thing we was trying to do
link |
is just have them look similar, be as integrated as possible,
link |
share all of that stuff.
link |
There were also three other APIs that others had built over time
link |
because we didn't have a standard one.
link |
But one of the messages that we kept hearing from the community,
link |
OK, which one do we use?
link |
And they kept seeing, OK, here's a model in this one,
link |
and here's a model in this one, which should I pick?
link |
So that's sort of like, OK, we had to address that
link |
straight on with 2.0.
link |
The whole idea was we need to simplify.
link |
We had to pick one.
link |
Based on where we were, we were like, OK, let's see what
link |
are the people like.
link |
And Keras was clearly one that lots of people loved.
link |
There were lots of great things about it.
link |
So we settled on that.
link |
That's kind of the best way to do it.
link |
But it was surprising, nevertheless,
link |
to sort of bring in and outside.
link |
I mean, there was a feeling like Keras might be almost
link |
like a competitor in a certain kind of a two tensor flow.
link |
And in a sense, it became an empowering element
link |
Yeah, it's interesting how you can put two things together
link |
which can align right.
link |
And in this case, I think Francois, the team,
link |
and a bunch of us have chatted and I think we all
link |
want to see the same kind of things.
link |
We all care about making it easier for the huge set
link |
of developers out there.
link |
And that makes a difference.
link |
So Python has Guido van Rossum, who
link |
until recently held the position of benevolent
link |
dictator for life.
link |
Right, so there's a huge successful open source
link |
project like tensor flow.
link |
Need one person who makes a final decision.
link |
So you did a pretty successful tensor flow Dev Summit
link |
just now, last couple of days.
link |
There's clearly a lot of different new features
link |
being incorporated in amazing ecosystem, so on.
link |
How are those design decisions made?
link |
Is there a BDFL in tensor flow?
link |
And or is it more distributed and organic?
link |
I think it's somewhat different, I would say.
link |
I've always been involved in the key design directions.
link |
But there are lots of things that
link |
are distributed where their number of people, Martin
link |
Wick being one who has really driven a lot of our open source
link |
stuff, a lot of the APIs.
link |
And there are a number of other people
link |
who have been pushed and been responsible
link |
for different parts of it.
link |
We do have regular design reviews.
link |
Over the last year, we've really spent a lot of time opening up
link |
to the community and adding transparency.
link |
We're setting more processes in place,
link |
so RFCs, special interest groups, really
link |
grow that community and scale that.
link |
I think the kind of scale that ecosystem is in,
link |
I don't think we could scale with having me as the lone
link |
point of decision maker.
link |
So yeah, the growth of that ecosystem,
link |
maybe you can talk about it a little bit.
link |
First of all, when I started with Andre Karpathi
link |
when he first did ComNet.js, the fact
link |
that you can train in your own network
link |
and the browser in JavaScript was incredible.
link |
So now TensorFlow.js is really making
link |
that a serious, a legit thing, a way
link |
to operate, whether it's in the back end or the front end.
link |
Then there's the TensorFlow Extended, like you mentioned.
link |
There's TensorFlow Lite for mobile.
link |
And all of it, as far as I can tell,
link |
it's really converging towards being
link |
able to save models in the same kind of way.
link |
You can move around, you can train on the desktop,
link |
and then move it to mobile, and so on.
link |
So this is that cohesiveness.
link |
So can you maybe give me whatever
link |
I missed, a bigger overview of the mission of the ecosystem
link |
that's trying to be built, and where is it moving forward?
link |
So in short, the way I like to think of this
link |
is our goals to enable machine learning.
link |
And in a couple of ways, one is we
link |
have lots of exciting things going on in ML today.
link |
We started with deep learning, but we now
link |
support a bunch of other algorithms too.
link |
So one is to, on the research side,
link |
keep pushing on the state of the art.
link |
Can we, how do we enable researchers
link |
to build the next amazing thing?
link |
So BERT came out recently.
link |
It's great that people are able to do new kinds of research.
link |
There are lots of amazing research
link |
that happens across the world.
link |
So that's one direction.
link |
The other is, how do you take that
link |
across all the people outside who want to take that research
link |
and do some great things with it and integrate it
link |
to build real products, to have a real impact on people?
link |
And so if that's the other axes in some ways.
link |
And a high level, one way I think about it
link |
is there are a crazy number of computer devices
link |
And we often used to think of ML and training and all of this
link |
as, OK, something you do either in the workstation
link |
or the data center or cloud.
link |
But we see things running on the phones.
link |
We see things running on really tiny chips.
link |
And we had some demos at the developer summit.
link |
And so the way I think about this ecosystem
link |
is, how do we help get machine learning on every device that
link |
has a compute capability?
link |
And that continues to grow.
link |
And so in some ways, this ecosystem
link |
has looked at various aspects of that
link |
and grown over time to cover more of those.
link |
And we continue to push the boundaries.
link |
In some areas, we've built more tooling and things
link |
around that to help you.
link |
I mean, the first tool we started was TensorBoard.
link |
You want to learn just the training piece, the effects
link |
for TensorFlow Extended to really do your entire ML
link |
pipelines if you care about all that production stuff,
link |
but then going to the edge, going to different kinds of things.
link |
And it's not just us now.
link |
We are a place where there are lots of libraries being built
link |
So there are some for research, maybe things
link |
like TensorFlow Agents or TensorFlow Probability that
link |
started as research things or for researchers
link |
for focusing on certain kinds of algorithms,
link |
but they're also being deployed or reduced by production folks.
link |
And some have come from within Google, just teams
link |
across Google who wanted to do the build these things.
link |
Others have come from just the community
link |
because there are different pieces
link |
that different parts of the community care about.
link |
And I see our goal as enabling even that.
link |
It's not we cannot and won't build every single thing.
link |
That just doesn't make sense.
link |
But if we can enable others to build the things
link |
that they care about, and there's a broader community that
link |
cares about that, and we can help encourage that,
link |
That really helps the entire ecosystem, not just those.
link |
One of the big things about 2.0 that we're pushing on
link |
is, OK, we have these so many different pieces, right?
link |
How do we help make all of them work well together?
link |
There are a few key pieces there that we're pushing on,
link |
one being the core format in there
link |
and how we share the models themselves through SAVE model
link |
and what TensorFlow Hub and so on.
link |
And a few of the pieces that we really put this together.
link |
I was very skeptical that that's, when TensorFlow.js came out,
link |
it didn't seem or deep learning.js.
link |
Yeah, that was the first.
link |
It seems like technically very difficult project.
link |
As a standalone, it's not as difficult.
link |
But as a thing that integrates into the ecosystem,
link |
it seems very difficult.
link |
So I mean, there's a lot of aspects of this
link |
you're making look easy.
link |
But on the technical side, how many challenges
link |
have to be overcome here?
link |
And still have to be overcome.
link |
That's the question here, too.
link |
There are lots of steps to it.
link |
I think we've iterated over the last few years,
link |
so there's a lot we've learned.
link |
I, yeah, and often when things come together well,
link |
And that's exactly the point.
link |
It should be easy for the end user.
link |
But there are lots of things that go behind that.
link |
If I think about still challenges ahead,
link |
there are we have a lot more devices coming on board,
link |
for example, from the hardware perspective.
link |
How do we make it really easy for these vendors
link |
to integrate with something like TensorFlow?
link |
So there's a lot of compiler stuff
link |
that others are working on.
link |
There are things we can do in terms of our APIs
link |
and so on that we can do.
link |
As we, TensorFlow started as a very monolithic system.
link |
And to some extent, it still is.
link |
There are less lots of tools around it,
link |
but the core is still pretty large and monolithic.
link |
One of the key challenges for us to scale that out
link |
is how do we break that apart with clear interfaces?
link |
It's, in some ways, it's software engineering one
link |
one, but for a system that's now four years old, I guess,
link |
or more, and that's still rapidly evolving
link |
and that we're not slowing down with,
link |
it's hard to change and modify and really break apart.
link |
It's sort of like, as people say, right,
link |
it's like changing the engine with a car running
link |
or fixed benefits.
link |
That's exactly what we're trying to do.
link |
So there's a challenge here, because the downside
link |
of so many people being excited about TensorFlow
link |
and becoming to rely on it in many other applications
link |
is that you're kind of responsible.
link |
It's the technical debt.
link |
You're responsible for previous versions
link |
to some degree still working.
link |
So when you're trying to innovate,
link |
I mean, it's probably easier to just start from scratch
link |
So do you feel the pain of that?
link |
2.0 does break some back compatibility, but not too much.
link |
It seems like the conversion is pretty straightforward.
link |
Do you think that's still important,
link |
given how quickly deep learning is changing?
link |
Can you just, the things that you've learned,
link |
can you just start over?
link |
Or is there pressure to not?
link |
It's a tricky balance.
link |
So if it was just a researcher writing a paper who
link |
a year later will not look at that code again,
link |
sure, it doesn't matter.
link |
There are a lot of production systems
link |
that rely on TensorFlow, both at Google
link |
and across the world.
link |
And people worry about this.
link |
I mean, these systems run for a long time.
link |
So it is important to keep that compatibility and so on.
link |
And yes, it does come with a huge cost.
link |
We have to think about a lot of things
link |
as we do new things and make new changes.
link |
I think it's a trade off, right?
link |
You can, you might slow certain kinds of things down,
link |
but the overall value you're bringing because of that
link |
is much bigger because it's not just
link |
about breaking the person yesterday.
link |
It's also about telling the person tomorrow that, you know what?
link |
This is how we do things.
link |
We're not going to break you when you come on board
link |
because there are lots of new people who are also
link |
going to come on board.
link |
So one way I like to think about this,
link |
and I always push the team to think about it as well,
link |
when you want to do new things, you
link |
want to start with a clean slate,
link |
design with a clean slate in mind,
link |
and then we'll figure out how to make sure all the other things
link |
And yes, we do make compromises occasionally.
link |
But unless you design with the clean slate
link |
and not worry about that, you'll never get to a good place.
link |
So even if you are responsible in the idea stage,
link |
when you're thinking of new, just put all that behind you.
link |
OK, that's really well put.
link |
So I have to ask this because a lot of students, developers,
link |
asked me how I feel about PyTorch versus TensorFlow.
link |
So I've recently completely switched my research group
link |
I wish everybody would just use the same thing.
link |
And TensorFlow is as close to that, I believe, as we have.
link |
But do you enjoy competition?
link |
So TensorFlow is leading in many ways, many dimensions
link |
in terms of the ecosystem, in terms of the number of users,
link |
momentum power, production level, so on.
link |
But a lot of researchers are now also using PyTorch.
link |
Do you enjoy that kind of competition,
link |
or do you just ignore it and focus
link |
on making TensorFlow the best that it can be?
link |
So just like research or anything people are doing,
link |
it's great to get different kinds of ideas.
link |
And when we started with TensorFlow,
link |
like I was saying earlier, it was very important for us
link |
to also have production in mind.
link |
We didn't want just research, right?
link |
And that's why we chose certain things.
link |
Now PyTorch came along and said, you know what?
link |
I only care about research.
link |
This is what I'm trying to do.
link |
What's the best thing I can do for this?
link |
And it started iterating and said, OK,
link |
I don't need to worry about graphs.
link |
Let me just run things.
link |
I don't care if it's not as fast as it can be,
link |
but let me just make this part easy.
link |
And there are things you can learn from that, right?
link |
They, again, had the benefit of seeing what had come before,
link |
but also exploring certain different kinds of spaces.
link |
And they had some good things there,
link |
building on, say, things like Jainer and so on before that.
link |
So competition is definitely interesting.
link |
It made us, you know, this is an area
link |
that we had thought about, like I said, very early on.
link |
Over time, we had revisited this a couple of times.
link |
Should we add this again?
link |
At some point, we said, you know what,
link |
here's it seems like this can be done well.
link |
So let's try it again.
link |
And that's how we started pushing on eager execution.
link |
How do we combine those two together,
link |
which has finally come very well together in 2.0,
link |
but it took us a while to get all the things together
link |
So let me, I mean, ask, put another way.
link |
I think eager execution is a really powerful thing,
link |
Do you think he wouldn't have been,
link |
you know, Muhammad Ali versus Frazier, right?
link |
Do you think it wouldn't have been added as quickly
link |
if PyTorch wasn't there?
link |
It might have taken longer.
link |
It was, I mean, we had tried some variants of that before.
link |
So I'm sure it would have happened,
link |
but it might have taken longer.
link |
I'm grateful that TensorFlow is part of the way they did.
link |
That's doing some incredible work last couple of years.
link |
What other things that we didn't talk about?
link |
Are you looking forward in 2.0?
link |
That comes to mind.
link |
So we talked about some of the ecosystem stuff,
link |
making it easily accessible to Keras, eager execution.
link |
Is there other things that we missed?
link |
Yeah, so I would say one is just where 2.0 is,
link |
and, you know, with all the things that we've talked about,
link |
I think as we think beyond that,
link |
there are lots of other things that it enables us to do
link |
and that we're excited about.
link |
So what it's setting us up for,
link |
okay, here are these really clean APIs.
link |
We've cleaned up the surface for what the users want.
link |
What it also allows us to do a whole bunch of stuff
link |
behind the scenes once we are ready with 2.0.
link |
So for example, in TensorFlow with graphs
link |
and all the things you could do,
link |
you could always get a lot of good performance
link |
if you spent the time to tune it, right?
link |
And we've clearly shown that, lots of people do that.
link |
With 2.0, with these APIs where we are,
link |
we can give you a lot of performance
link |
just with whatever you do.
link |
You know, because we see these, it's much cleaner.
link |
We know most people are gonna do things this way.
link |
We can really optimize for that
link |
and get a lot of those things out of the box.
link |
And it really allows us, you know,
link |
both for single machine and distributed and so on,
link |
to really explore other spaces behind the scenes
link |
after 2.0 in the future versions as well.
link |
So right now, the team's really excited about that,
link |
that over time, I think we'll see that.
link |
The other piece that I was talking about
link |
in terms of just restructuring the monolithic thing
link |
into more pieces and making it more modular,
link |
I think that's gonna be really important
link |
for a lot of the other people in the ecosystem,
link |
other organizations and so on that wanted to build things.
link |
Can you elaborate a little bit what you mean
link |
by making TensorFlow more ecosystem or modular?
link |
So the way it's organized today is there's one,
link |
there are lots of repositories
link |
in the TensorFlow organization at GitHub,
link |
the core one where we have TensorFlow,
link |
it has the execution engine,
link |
it has, you know, the key backends for CPUs and GPUs,
link |
it has the work to do distributed stuff.
link |
And all of these just work together
link |
in a single library or binary,
link |
there's no way to split them apart easily.
link |
I mean, there are some interfaces,
link |
but they're not very clean.
link |
In a perfect world, you would have clean interfaces where,
link |
okay, I wanna run it on my fancy cluster
link |
with some custom networking,
link |
just implement this and do that.
link |
I mean, we kind of support that,
link |
but it's hard for people today.
link |
I think as we are starting to see more interesting things
link |
in some of these spaces,
link |
having that clean separation will really start to help.
link |
And again, going to the large size of the ecosystem
link |
and the different groups involved there,
link |
enabling people to evolve and push on things
link |
more independently just allows it to scale better.
link |
And by people, you mean individual developers and?
link |
And organizations.
link |
And organizations.
link |
So the hope is that everybody sort of major,
link |
I don't know, Pepsi or something uses,
link |
like major corporations go to TensorFlow to this kind of.
link |
Yeah, if you look at enterprise like Pepsi or these,
link |
I mean, a lot of them are already using TensorFlow.
link |
They are not the ones that do the development
link |
or changes in the core.
link |
Some of them do, but a lot of them don't.
link |
I mean, they touch small pieces.
link |
There are lots of these, some of them being,
link |
let's say hardware vendors who are building
link |
their custom hardware and they want their own pieces.
link |
Or some of them being bigger companies, say IBM.
link |
I mean, they're involved in some of our special interest
link |
groups and they see a lot of users
link |
who want certain things and they want to optimize for that.
link |
So folks like that often.
link |
Autonomous vehicle companies, perhaps.
link |
So yeah, like I mentioned, TensorFlow
link |
has been down on it 41 million times, 50,000 commits,
link |
almost 10,000 pull requests, 1,800 contributors.
link |
So I'm not sure if you can explain it,
link |
but what does it take to build a community like that?
link |
In retrospect, what do you think?
link |
What is the critical thing that allowed for this growth
link |
to happen and how does that growth continue?
link |
Yeah, that's an interesting question.
link |
I wish I had all the answers there, I guess,
link |
so you could replicate it.
link |
I think there are a number of things
link |
that need to come together, right?
link |
One, just like any new thing, there's
link |
a sweet spot of timing, what's needed,
link |
does it grow with what's needed.
link |
So in this case, for example, TensorFlow
link |
is not just grown because it has a good tool,
link |
it's also grown with the growth of deep learning itself.
link |
So those factors come into play.
link |
Other than that, though, I think just
link |
hearing, listening to the community, what they're
link |
doing, what they need, being open to,
link |
like in terms of external contributions,
link |
we've spent a lot of time in making sure
link |
we can accept those contributions well,
link |
we can help the contributors in adding those,
link |
putting the right process in place,
link |
getting the right kind of community,
link |
welcoming them, and so on.
link |
Like over the last year, we've really pushed on transparency.
link |
That's important for an open source project.
link |
People want to know where things are going,
link |
and we're like, OK, here's a process for you.
link |
You can do that, here are our seasons, and so on.
link |
So thinking through, there are lots of community aspects
link |
that come into that you can really work on.
link |
As a small project, it's maybe easy to do,
link |
because there's two developers, and you can do those.
link |
As you grow, putting more of these processes in place,
link |
thinking about the documentation,
link |
thinking about what two developers
link |
care about, what kind of tools would they want to use,
link |
all of these come into play, I think.
link |
So one of the big things, I think,
link |
that feeds the TensorFlow fire is people building something
link |
on TensorFlow, and implement a particular architecture
link |
that does something cool and useful,
link |
and they put that on GitHub.
link |
And so it just feeds this growth.
link |
Do you have a sense that with 2.0 and 1.0,
link |
that there may be a little bit of a partitioning like there
link |
is with Python 2 and 3, that there'll be a code base
link |
in the older versions of TensorFlow
link |
that will not be as compatible easily,
link |
or are you pretty confident that this kind of conversion
link |
is pretty natural and easy to do?
link |
So we're definitely working hard to make that very easy to do.
link |
There's lots of tooling that we talked about at the developer
link |
summit this week, and we'll continue
link |
to invest in that tooling.
link |
It's when you think of these significant version changes,
link |
that's always a risk, and we are really pushing hard
link |
to make that transition very, very smooth.
link |
I think, so at some level, people
link |
want to move when they see the value in the new thing.
link |
They don't want to move just because it's a new thing.
link |
And some people do, but most people want a really good thing.
link |
And I think over the next few months,
link |
as people start to see the value,
link |
we'll definitely see that shift happening.
link |
So I'm pretty excited and confident that we
link |
will see people moving.
link |
As you said earlier, this field is also moving rapidly,
link |
so that'll help because we can do more things.
link |
And all the new things will clearly
link |
happen in 2.x, so people will have lots of good reasons to move.
link |
So what do you think TensorFlow 3.0 looks like?
link |
Is there things happening so crazily
link |
that even at the end of this year,
link |
seems impossible to plan for?
link |
Or is it possible to plan for the next five years?
link |
I think it's tricky.
link |
There are some things that we can expect in terms of, OK,
link |
change, yes, change is going to happen.
link |
Are there some things going to stick around
link |
and some things not going to stick around?
link |
I would say the basics of deep learning,
link |
the convolutional models or the basic kind of things,
link |
they'll probably be around in some form still in five years.
link |
Will Aurel and Gans stay very likely based on where they are?
link |
Will we have new things?
link |
Probably, but those are hard to predict.
link |
And some directionally, some things that we can see
link |
is in things that we're starting to do
link |
with some of our projects right now is just
link |
to point out combining eager execution and graphs,
link |
where we're starting to make it more like just your natural
link |
programming language.
link |
You're not trying to program something else.
link |
Similarly, with Swift for TensorFlow,
link |
we're taking that approach.
link |
Can you do something round up?
link |
So some of those ideas seem like, OK,
link |
that's the right direction in five years
link |
we expect to see more in that area.
link |
Other things we don't know is, will hardware accelerators
link |
Will we be able to train with four bits instead of 32 bits?
link |
And I think the TPU side of things is exploring.
link |
I mean, TPU is already on version three.
link |
It seems that the evolution of TPU and TensorFlow
link |
are coevolving in terms of both their learning
link |
from each other and from the community
link |
and from the applications where the biggest benefit is achieved.
link |
You've been trying with eager with Keras
link |
to make TensorFlow as accessible and easy to use as possible.
link |
What do you think for beginners is the biggest thing
link |
they struggle with?
link |
Have you encountered that?
link |
Or is basically what Keras is solving
link |
is that eager, like we talked about TensorFlow?
link |
For some of them, like you said, the beginners
link |
want to just be able to take some image model.
link |
They don't care if it's inception or rest net or something else
link |
and do some training or transfer learning
link |
on their kind of model.
link |
Being able to make that easy is important.
link |
So in some ways, if you do that by providing them
link |
simple models with, say, in Hub or so on,
link |
they don't care about what's inside that box,
link |
but they want to be able to use it.
link |
So we're pushing on, I think, different levels.
link |
If you look at just a component that you get, which
link |
has the layers already smushed in,
link |
the beginners probably just want that.
link |
Then the next step is, OK, look at building
link |
layers with Keras.
link |
If you go out to research, then they
link |
are probably writing custom layers themselves
link |
or doing their own loops.
link |
So there's a whole spectrum there.
link |
And then providing the preentrain models
link |
seems to really decrease the time from you trying to start.
link |
So you could basically, in a Colab notebook,
link |
achieve what you need.
link |
So I'm basically answering my own question,
link |
because I think what TensorFlow delivered on recently
link |
is trivial for beginners.
link |
So I was just wondering if there was other pain points
link |
you're trying to ease, but I'm not sure there would.
link |
No, those are probably the big ones.
link |
I mean, I see high schoolers doing a whole bunch of things
link |
now, which is pretty amazing.
link |
It's both amazing and terrifying.
link |
In a sense that when they grow up,
link |
some incredible ideas will be coming from them.
link |
So there's certainly a technical aspect to your work,
link |
but you also have a management aspect
link |
to your role with TensorFlow, leading the project,
link |
a large number of developers and people.
link |
So what do you look for in a good team?
link |
What do you think Google has been at the forefront
link |
of exploring what it takes to build a good team?
link |
And TensorFlow is one of the most cutting edge technologies
link |
So in this context, what do you think
link |
makes for a good team?
link |
It's definitely something I think a fair bit about.
link |
I think in terms of the team being
link |
able to deliver something well, one of the things that's
link |
important is a cohesion across the team.
link |
So being able to execute together and doing things,
link |
Like at this scale, an individual engineer
link |
can only do so much.
link |
There's a lot more that they can do together,
link |
even though we have some amazing superstars across Google
link |
But there's often the way I see it
link |
is the product of what the team generates
link |
is way larger than the whole individual put together.
link |
And so how do we have all of them work together,
link |
the culture of the team itself?
link |
Hiring good people is important.
link |
But part of that is it's not just that, OK,
link |
we hire a bunch of smart people and throw them together
link |
and let them do things.
link |
It's also people have to care about what they're building.
link |
People have to be motivated for the right kind of things.
link |
That's often an important factor.
link |
And finally, how do you put that together
link |
with a somewhat unified vision of where we want to go?
link |
So are we all looking in the same direction
link |
or just going all over?
link |
And sometimes it's a mix.
link |
Google's a very bottom up organization in some sense.
link |
Also research even more so.
link |
And that's how we started.
link |
But as we've become this larger product and ecosystem,
link |
I think it's also important to combine that well with a mix
link |
of, OK, here's the direction we want to go in.
link |
There is exploration we'll do around that.
link |
But let's keep staying in that direction, not just
link |
all over the place.
link |
And is there a way you monitor the health of the team?
link |
Sort of like, is there a way you know you did a good job?
link |
I mean, you're saying nice things, but it's sometimes
link |
difficult to determine how aligned.
link |
Because it's not binary, it's not like there's tensions
link |
and complexities and so on.
link |
And the other element of this is the mesh of superstars.
link |
There's so much, even at Google, such a large percentage
link |
of work is done by individual superstars too.
link |
So there's a, and sometimes those superstars
link |
could be against the dynamic of a team and those tensions.
link |
I mean, I'm sure TensorFlow might be a little bit easier
link |
because the mission of the project is so beautiful.
link |
You're at the cutting edge, so it's exciting.
link |
But have you had struggle with that?
link |
Has there been challenges?
link |
There are always people challenges
link |
in different kinds of ways.
link |
That said, I think we've been what's
link |
good about getting people who care and have
link |
the same kind of culture, and that's Google in general
link |
to a large extent.
link |
But also, like you said, given that the project has had
link |
so many exciting things to do, there's
link |
been room for lots of people to do different kinds of things
link |
and grow, which does make the problem a bit easier, I guess.
link |
And it allows people, depending on what they're doing,
link |
if there's room around them, then that's fine.
link |
But yes, we do care about whether a superstar or not
link |
that they need to work well with the team across Google.
link |
That's interesting to hear.
link |
So it's like superstar or not, the productivity broadly
link |
is about the team.
link |
I mean, they might add a lot of value,
link |
but if they're hurting the team, then that's a problem.
link |
So in hiring engineers, it's so interesting, right?
link |
The high rank process, what do you look for?
link |
How do you determine a good developer
link |
or a good member of a team from just a few minutes
link |
or hours together?
link |
Again, no magic answers, I'm sure.
link |
And Google has a hiring process that we've refined
link |
over the last 20 years, I guess, and that you've probably
link |
heard and seen a lot about.
link |
So we do work with the same hiring process in that.
link |
That's really helped.
link |
For me in particular, I would say,
link |
in addition to the core technical skills,
link |
what does matter is their motivation
link |
in what they want to do.
link |
Because if that doesn't align well with where we want to go,
link |
that's not going to lead to long term success
link |
for either them or the team.
link |
And I think that becomes more important the more senior
link |
the person is, but it's important at every level.
link |
Like even the junior most engineer,
link |
if they're not motivated to do well at what they're trying to do,
link |
however smart they are, it's going
link |
to be hard for them to succeed.
link |
Does the Google hiring process touch on that passion?
link |
So like trying to determine.
link |
Because I think as far as I understand,
link |
maybe you can speak to it that the Google hiring process sort
link |
of helps the initial like determines the skill set there,
link |
is your puzzle solving ability, problem solving ability good.
link |
But I'm not sure, but it seems that the determining
link |
whether the person is like fire inside them
link |
that burns to do anything really doesn't really matter.
link |
It's just some cool stuff.
link |
I'm going to do it that I don't know.
link |
Is that something that ultimately ends up
link |
when they have a conversation with you
link |
or once it gets closer to the team?
link |
So one of the things we do have as part of the process
link |
is just a culture fit, like part of the interview process
link |
itself, in addition to just the technical skills.
link |
And each engineer or whoever the interviewer is,
link |
is supposed to rate the person on the culture and the culture
link |
fit with Google and so on.
link |
So that is definitely part of the process.
link |
Now, there are various kinds of projects
link |
and different kinds of things.
link |
So there might be variants in the kind of culture
link |
you want there and so on.
link |
And yes, that does vary.
link |
So for example, TensorFlow has always
link |
been a fast moving project.
link |
And we want people who are comfortable with that.
link |
But at the same time now, for example,
link |
we are at a place where we are also very full fledged product.
link |
And we want to make sure things that work really, really
link |
You can't cut corners all the time.
link |
So balancing that out and finding the people
link |
who are the right fit for those is important.
link |
And I think those kind of things do vary a bit
link |
across projects and teams and product areas across Google.
link |
And so you'll see some differences there
link |
in the final checklist.
link |
But a lot of the core culture, it
link |
comes along with just the engineering, excellence,
link |
What is the hardest part of your job?
link |
I'll take your pick, I guess.
link |
It's fun, I would say.
link |
I mean, lots of things at different times.
link |
I think that does vary.
link |
So let me clarify that difficult things are fun
link |
when you solve them, right?
link |
It's fun in that sense.
link |
I think the key to a successful thing across the board,
link |
and in this case, it's a large ecosystem now,
link |
but even a small product, is striking that fine balance
link |
across different aspects of it.
link |
Sometimes it's how fast you go versus how perfect it is.
link |
Sometimes it's how do you involve this huge community?
link |
Who do you involve?
link |
Or do you decide, OK, now is not a good time to involve them
link |
because it's not the right fit?
link |
Sometimes it's saying no to certain kinds of things.
link |
Those are often the hard decisions.
link |
Some of them you make quickly because you don't have the time.
link |
Some of them you get time to think about them,
link |
but they're always hard.
link |
So both choices are pretty good, those decisions.
link |
What about deadlines?
link |
Is this defined TensorFlow to be driven by deadlines
link |
to a degree that a product might?
link |
Or is there still a balance to where it's less deadline?
link |
You had the Dev Summit, they came together incredibly.
link |
Looked like there's a lot of moving pieces and so on.
link |
So did that deadline make people rise to the occasion,
link |
releasing TensorFlow 2.0 Alpha?
link |
I'm sure that was done last minute as well.
link |
I mean, up to the last point.
link |
Again, it's one of those things that you
link |
need to strike the good balance.
link |
There's some value that deadlines bring
link |
that does bring a sense of urgency
link |
to get the right things together.
link |
Instead of getting the perfect thing out,
link |
you need something that's good and works well.
link |
And the team definitely did a great job in putting that
link |
together, so it was very amazed and excited by everything,
link |
how that came together.
link |
That said, across the year, we try not
link |
to put out official deadlines.
link |
We focus on key things that are important,
link |
figure out how much of it's important,
link |
and we are developing in the open, internally and externally,
link |
everything's available to everybody.
link |
So you can pick and look at where things are.
link |
We do releases at a regular cadence,
link |
so fine if something doesn't necessarily end up with this
link |
month, it'll end up in the next release in a month or two.
link |
And that's OK, but we want to keep moving
link |
as fast as we can in these different areas.
link |
Because we can iterate and improve on things, sometimes
link |
it's OK to put things out that aren't fully ready.
link |
If you make sure it's clear that, OK, this is experimental,
link |
but it's out there if you want to try and give feedback.
link |
That's very, very useful.
link |
I think that quick cycle and quick iteration is important.
link |
That's what we often focus on rather than here's
link |
a deadline where you get everything else.
link |
It's 2.0, is there pressure to make that stable?
link |
Or like, for example, WordPress 5.0 just came out,
link |
and there was no pressure to, it was a lot of build updates
link |
that delivered way too late.
link |
And they said, OK, well, we're going
link |
to release a lot of updates really quickly to improve it.
link |
Do you see TensorFlow 2.0 in that same kind of way,
link |
or is there this pressure to once it hits 2.0,
link |
once you get to the release candidate,
link |
and then you get to the final, that's
link |
going to be the stable thing?
link |
So it's going to be stable in just like 1.0X
link |
was where every API that's there is going to remain in work.
link |
It doesn't mean we can't change things under the covers.
link |
It doesn't mean we can't add things.
link |
So there's still a lot more for us to do,
link |
and we continue to have more releases.
link |
So in that sense, there's still, I
link |
don't think we'd be done in like two months
link |
when we release this.
link |
I don't know if you can say, but is there, you know,
link |
there's not external deadlines for TensorFlow 2.0,
link |
but is there internal deadlines, artificial or otherwise,
link |
that you're trying to set for yourself,
link |
or is it whenever it's ready?
link |
So we want it to be a great product, right?
link |
And that's a big, important piece for us.
link |
TensorFlow is already out there.
link |
We have 41 million downloads for 1.x,
link |
so it's not like we have to have this.
link |
So it's not like a lot of the features
link |
that we've really polishing and putting them together
link |
are there, we don't have to rush that just because.
link |
So in that sense, we want to get it right
link |
and really focus on that.
link |
That said, we have said that we are
link |
looking to get this out in the next few months,
link |
in the next quarter, and as far as possible,
link |
we'll definitely try to make that happen.
link |
Yeah, my favorite line was, spring is a relative concept.
link |
Spoken like a true developer.
link |
So something I'm really interested in,
link |
and your previous line of work is, before TensorFlow,
link |
you let a team and Google on search ads.
link |
I think this is a very interesting topic on every level,
link |
on a technical level, because if their best ads connect people
link |
to the things they want and need,
link |
and that they're worse, they're just these things
link |
that annoy the heck out of you to the point of ruining
link |
the entire user experience of whatever you're actually doing.
link |
So they have a bad rep, I guess.
link |
And on the other end, so that this connecting users
link |
to the thing they need to want is a beautiful opportunity
link |
for machine learning to shine, like huge amounts of data
link |
that's personalized, and you've got
link |
to map to the thing they actually won't get annoyed.
link |
So what have you learned from this Google that's
link |
leading the world in this aspect?
link |
What have you learned from that experience?
link |
And what do you think is the future of ads?
link |
Take you back to the end of that.
link |
Yes, it's been a while, but I totally agree with what you said.
link |
I think the search ads, the way it was always looked at,
link |
and I believe it still is, is it's
link |
an extension of what search is trying to do.
link |
The goal is to make the information
link |
and make the world's information accessible.
link |
With ads, it's not just information,
link |
but it may be products or other things
link |
that people care about.
link |
And so it's really important for them
link |
to align with what the users need.
link |
And in search ads, there's a minimum quality level
link |
before that ad would be shown.
link |
If we don't have an ad that hits that quality bar,
link |
it will not be shown, even if we have it.
link |
And OK, maybe we lose some money there.
link |
That is really, really important,
link |
and I think that that is something I really
link |
liked about being there.
link |
Advertising is a key part.
link |
I mean, as a model, it's been around for ages, right?
link |
It's not a new model.
link |
It's been adapted to the web and became a core part of search
link |
and in many other search engines across the world.
link |
I do hope, like I said, there are aspects of ads
link |
that are annoying.
link |
And I go to a website, and if it just
link |
keeps popping an ad in my face, not to let me read,
link |
that's going to be annoying clearly.
link |
So I hope we can strike that balance between showing a good
link |
ad where it's valuable to the user
link |
and provides the monetization to the service.
link |
And this might be search.
link |
This might be a website.
link |
All of these, they do need the monetization for them
link |
to provide that service.
link |
But if it's done in a good balance between showing
link |
just some random stuff that's distracting
link |
versus showing something that's actually valuable.
link |
So do you see it moving forward as to continue
link |
being a model that funds businesses like Google?
link |
That's a significant revenue stream.
link |
Because that's one of the most exciting things,
link |
but also limiting things on the internet
link |
is nobody wants to pay for anything.
link |
And advertisements, again, coupled at their best
link |
are actually really useful and not annoying.
link |
Do you see that continuing and growing and improving?
link |
Or is there GC sort of more Netflix type models
link |
where you have to start to pay for content?
link |
I think it's a mix.
link |
I think it's going to take a long while for everything
link |
to be paid on the internet, if at all.
link |
I mean, I think there's always going
link |
to be things that are sort of monetized with things like ads.
link |
But over the last few years, I would say
link |
we've definitely seen that transition
link |
towards more paid services across the web
link |
and people are willing to pay for them
link |
because they do see the value.
link |
I mean, Netflix is a great example.
link |
I mean, we have YouTube doing things.
link |
People pay for the apps they buy, more people
link |
they find are willing to pay for newspaper content,
link |
for the good news websites across the web.
link |
That wasn't the case even a few years ago, I would say.
link |
And I just see that change in myself as well
link |
and just lots of people around me.
link |
So definitely hopeful that we'll transition to that mix model
link |
where maybe you get to try something out for free,
link |
But then there is a more clear revenue model
link |
that sort of helps go beyond that.
link |
So speaking of revenue, how is it
link |
that a person can use the TPU in a Google Colab for free?
link |
So what's the, I guess, the question is,
link |
what's the future of TensorFlow in terms of empowering,
link |
say, a class of 300 students?
link |
And I'm asked by MIT, what is going
link |
to be the future of them being able to do their homework
link |
Where are they going to train these networks, right?
link |
What's that future look like with TPUs, with cloud services,
link |
I think a number of things there.
link |
I mean, any TensorFlow open source,
link |
you can run it wherever.
link |
You can run it on your desktop, and your desktops
link |
always keep getting more powerful, so maybe you can do more.
link |
My phone is like, I don't know how many times more powerful
link |
than my first desktop.
link |
You'll probably train it on your phone, though.
link |
Yeah, that's true.
link |
Right, so in that sense, the power
link |
you have in your hand is a lot more.
link |
Clouds are actually very interesting from, say,
link |
students or courses perspective, because they
link |
make it very easy to get started.
link |
I mean, Colab, the great thing about it
link |
is go to a website, and it just works.
link |
No installation needed, nothing to, you know,
link |
you're just there, and things are working.
link |
That's really the power of cloud, as well.
link |
And so I do expect that to grow.
link |
Again, Colab is a free service.
link |
It's great to get started, to play with things,
link |
to explore things.
link |
That said, with free, you can only get so much, maybe.
link |
So just like we were talking about free versus paid,
link |
and there are services you can pay for and get a lot more.
link |
So if I'm a complete beginner interested in machine
link |
learning and TensorFlow, what should I do?
link |
Probably start with going to a website and playing there.
link |
Just go to TensorFlow.org and start clicking on things.
link |
Yep, check out tutorials and guides.
link |
There's stuff you can just click there and go to Colab
link |
No installation needed.
link |
You can get started right there.
link |
Roger, thank you so much for talking today.
link |
Have fun this week.