Sunday, December 21, 2008

The one Google thing I miss the most

This week turns out to be exactly six months since I left Google to return to Microsoft. The year at Google was fun, and I learned a lot of new stuff. I am glad that I took this detour from my Microsoft career - it gave me a lot of new perspectives.

Six months later, what aspects of "The Google Way (TM)" do I miss the most?

Surprisingly, the free food is not it - not even close: luckily, most software engineers are paid enough so that the food costs are not really visible in our budgets.

Google's focus on engineering, where a developer owns everything about the product, including testing and the feature set, is nice at first, until you realize that it comes at a heavy price.

The products that can be built within this model tend to be smallish and not very well integrated. Every project that I've done at Microsoft was bigger in scope, more interesting as an engineering challenge, and all but two made far more money than anything I created at Google. And I am glad to be working on something really big again.

I miss the peer review system. But it is easy to replicate, and Microsoft already gives me all the necessary tools, and we are going to be trying it in my team, although I will implement it slightly differently.

At Google one needs to get positive feedback from people who are one's senior in order to be promoted. Feedback from peers and junior people does not matter very much. The problem is that at some point there are just not enough senior people around you - there are only a handful of technical fellows at Google, and less than 10 distinguished engineers, if I remember correctly.

As a result I have seen mediocre engineers grow very quickly just because they happened to work on a project that had a DE, and much better engineers stuck at lower levels because they were not so lucky.

So in my team we're going to value peer feedback much more uniformly, and assign a bigger weight to the manager feedback than the 10-15% that is customary at Google.

I don't miss my Google managers at all. They were nice people, but they didn't have much impact on my life. After a year at Google, I still do not quite understand what the role of a manager there is. Whatever they do, they certainly do not have the tools to succeed, and their impact on teams seems to start and end with allocating resources.

I worked with a bunch of very smart, talented, and passionate developers at Google. I even had the privilege of meeting Craig Silverstein through my work on the readability team and exchanging a couple of change lists with him. The amount of energy and brain power collected there is staggering.

But I also worked with a few people who were the weakest developers I'd seen in my entire career. They would show up at work around noon, eat the free lunch, browse the web, attend a few meetings, and generally be gone by 6-ish. In the 4 months I worked with them, they had each written a couple of hundred lines of code, and copied and pasted a few hundred lines more. I don't miss that project at all.

I don't miss Linux very much either - especially the C/C++ development tools. I now understand why so many smart people that I met at Google said that they are never going back to C++ after Java. I do not miss debugging with printfs.

The fact that Google had either reimplemented or wrapped every OS API - from thread and process creation to HTTP protocol handlers and RPC - did not help me appreciate the beauty of this extremely lean operating system :-). Yes, the OS was small, fast, (insert your own epithet describing efficiency here)...

But the applications that compensated for this leanness ended up being massive lumbering beasts, requiring hundreds of megabytes of RAM and a minute or more to start. And it was not because these apps were written by idiots - no, the people who created them were actually quite good. It was because they had to do a lot of things that the "bloated" Windows OS does for you.

I miss Python somewhat. It is a nice, effective script language that I learned to love. Python is a corenerstone in Google's infrastructure - everything, from the check-in tool to makefiles, is written in Python. The best thing about Python is its consistency - you can clearly see that the language had one designer, who actually DESIGNED it, rather than just stuffing random unrelated features in a la C++ (or Perl).

I think Linux people should just adopt Python as "the one script engine," and dispense with the proliferation of *sh interpreters with their horrible, horrible scripting support that leaves no doubt that it was simply bandaged on top of something that was never designed to do scripting. And after 25 years, the bandage is wearing off and the puss is seeping all around it.

But on the other hand, there is nothing - at least on Windows - that is doable with Python and cannot be done with C# in an equivalent amount of time, or faster. So while I like the language a lot, I found myself mostly using C# anyway because of it's "built-in" status on Windows and really nice Visual Studio support.

There is one thing at Google that I miss a lot, without any qualifications, with no "if"s or "but"s. It's Mondrian, the Google code review system.

It was written by Guido van Rossum, the creator of Python. It was his starter project, and it demonstrates another admirable thing that Google does, which is investing very heavily in development productivity. I would say that by creating Mondrian, Guido single-handedly improved the quality of Google code by at least 20%.

Guido describes it here: in great detail.

To summarize this presentation in just a few sentences: you create a change list in perforce. It automatically shows on the Mondrian web site. You go to this web and request code reviews by typing people's aliases. The mail gets sent with the link to the change list representation on this web site.

The web site shows - for every file in the change list - what it was before and what it is now, side by side. To comment, you click anywhere and start typing. The comment gets attached to the line of source on which you clicked. When you're done, submit the result, and the reviewee as well as the other reviewers are notified, and can see your comments on the same web site, right inside the files that are reviewed.

The reason Mondrian works so well is because the cost of making a comment is nearly zero - which is not true for over-the-shoulder code reviews, or trading files - where someone needs to take a note, fix the code, and then the reviewer has no easy way of verifying that the suggestion has actually been followed on. The point, click, and type UI paradigm that has been successfully driving the computer industry for so many years, has proven itself indispensable once again.

Unfortunately, Mondrian is an internal Google tool, not accessible from the outside. And even if it were, it is built on internal Google infrastructure, so it will be a cold day in hell before software organizations begin sending all of their code through Google servers. Obviously, this is not likely to happen at Microsoft :-).

So now that we've reached the end of M0 and a vast majority of my team is on their well-deserved vacations, I have a bunch of free time that I can spend on toying around. I have started playing with building my own Mondrian-like system that you can actually deploy on stock software such as IIS and SQL Server - stuff that is easily available to developers worldwide.

I have several objectives.

First, I want to learn ASP.NET and SQL. For the most part of my career, I was a very low-level developer - I know a lot about operating systems, implementation of compilers, low-level network protocols and file systems. I don't know very much about high-level abstractions - I have never written anything serious in C#, my experience building web services is confined to Google's infrastructure (and as such is not very useful outside Google), and I had never used a database until about a month ago.

Second, I want my team to have a nice code review system. We've had a few attempts at reusing tools that had already existed at Microsoft, but they all proved to be either very inconvenient, or very unstable. A good functioning code review system is a boon to developer productivity. This much I learned at Google.

Third, I think the world will benefit from a nice local, retargetable code review system that can be built on top of widely available software components. So when I am almost finished, I will be releasing it on Codeplex - Microsoft's shared code repository (

Meanwhile, you can read about my progress here. :-)

Update: It's up and running. We've used it for a month and a half, for more than 200 code reviews, 2000 files, and 3000 comments. Get it here:


Anonymous said...

Open source Mondrian-like tool from Guido:

Yi Li said...

I'm working at Google. Your view about the relationship between Sr.'s peer review and promotions is pretty wise.
I hadn't realized this until my first promotion failure when even some guys whom I look down upon got promotions just because they happened to work with a "Staff Eng".
This situation can not be changed. So I could only change myself. I would never work on projects without Sr. guys.
Maybe some workmates would tell me that those projects are also important to our company. I would reply: "F**k those projects, my PROMOTION is the most important!"

NS said...

Has its quirks, but by and large does everything on your list

Илья Казначеев said...

"And it was not because these apps were written by idiots - no, the people who created them were actually quite good. It was because they had to do a lot of things that the "bloated" Windows OS does for you."

Things like what?
I mean, do you have anything to back this up or you're just feeling it that way?

Sergey Solyanik said...

Things like RPC, HTTP listener, thread pools, process/thread wrapper classes, event/semaphore wrapper classes, distributed file system, flags, ..., ...

In my entire life at Google I called 1 (one) Linux API. Everything else was Google wrappers replacing and extending Linux functionality. They had wrappers for Java as well, but they were much thinner.

Илья Казначеев said...

Well, it's not the linux fault that they've decided to reinvent the wheel.
And I can't blame them for that. NIH is nasty when you're small and playing catch-up, and later on it's nice to fully controll all your environment.

However, I think, if they used windows, they would write equally as thick layer wrapping windows apis.
Besides, I don't understand how wrappers can make application to boot for a minute. One minute is a lot! I bet it did a lot of network calls?

Java wrappers are thinner because java native apis are more abstract, they expose much less ugly thing to wrap.

Sudarshan said...

If you are looking for more inspiration for web based code reviews you should take a look at reviewboard

Very easy to install and works well with most SCMs

Anonymous said...

Please note the irony of railing against Google's wrappers and extolling the virtues of Windows doing everything for you, yet having to do your own wrapper to work around one issue with C# .Net and also having to add runtime code to work around another.

By the way, do you know of a KB article covering the defect you're working around?

For a medium to large organization, centralizing such necessities in libraries rather than having each team or engineer reimplment them with varying degrees of success is quite a valid approach.

From the article:

(By the way, before I forget, here's a way to capture a process output in C# that actually works (just reading standard handles blocks forever if there is a lot of output on both stdout and stderr channels):

private delegate string StringDelegate();public static string ReadProcessOutput(Process proc, bool eatFirstLine, ref string errorMessage)