neugierig.org: Tech Notes

It's been around 10 years since I worked on Chrome. Here are some stories that have surely been embellished by my own memory over that decade of distance. Treat them all as only maybe 60% true. I figured I could at least write them down before I totally forget them.

What I did. I worked on Chrome from the early days, from 2007 to 2012, which just about covers the second half of my twenties. I had other stuff going on in my life, of course — including, for example, my wedding — but in terms of anything I did with my life that the larger world cares about, Chrome is what I fed my twenties to. Today it has more than 2.5 billion users. It's still a weird feeling to go to a cafe and see the person next to me have "my" software up on their screen.

I joined the project with the goal to make Linux Chrome a thing, which meant first working on Windows-only Chrome for some years before eventually leading the Linux team. Inevitably when a project is new you end up touching stuff all over it — I remember writing the code to make the "star" button light up when you clicked it, for example, which just entailed a new boolean of state and an additional image resource — but one of my larger lasting legacies is the build system we ended up with.

I kept a blog about Chrome for most of those years with a lot of technical details. See my first public post there. My farewell post about why I left talks more about my personal history.

Why was I there? I came in to Google and Chrome thinking of myself as a member of the free software community. In college we used crazy hacks involving running a Windows computer alongside your Linux one just to get a working browser because the web required Windows-only Internet Explorer. As Brad wrote there: "I've waited for Mozilla [the pre-Firefox open source release] to not suck for over a year". I have a memory of being in my dorm room failing to get Mozilla to build on Linux and thinking to myself that one day I could fix this. Chrome gave me exactly that opportunity.

Ironically, I think I ended up in the opposite place: I learned through painful experience the structural reasons why user-facing software on Linux, both Mozilla’s and mine, generally isn’t great. And similarly I learned how free software can come from a place of giving and gratitude and end up as yet another weapon for corporations. In retrospect it's an old trope, where I tried to "change the system from within" and ended up becoming consumed by it.

Why was Google there? With today's success it feels obvious in retrospect, but at the time it seemed really crazy for Google to build a browser. (At the time it was even a recurring joke within the company about how it would never happen.)

History will surely uncover a Machiavellian plot about usurping control over the web, but at the time the story was pretty simple, at least as it was explained to naive pawns like me. Google wanted the web to succeed and even had a team of people contributing to Firefox. Google wanted more influence over the product, Mozilla was (reasonably) uncomfortable with that, so we went our own way. It’s easy to read some sort of great conflict in that but I still don’t see it. As far as I understand it — it happened after my time — the WebKit fork was the same thing: they wanted WebKit to do things Apple didn’t and eventually it was too much.

There was a second subplot about how Microsoft was increasingly turning to nasty practices to get back at Google, and fears about using IE to pull some anti-Google shenanigans. At the time I think a Windows update was resetting your search engine, but Microsoft does this kind of stuff all the time. You can read Valve’s attempts at a Linux-based OS as a similar sort of defense against the Windows app store. I know Microsoft today is supposed to be cool again or whatever but I still remember the 90s, and I'm regularly reminded by how vicious and petty companies can be.

Third, even if Chrome was a total failure the hope was that it would light a fire under the other browsers, and I think it did; nobody really cared about JavaScript benchmarks until v8 came out. (You can interpret Google Fiber as trying the same thing, though I don’t have much insight into that.) It’s easy to forget how bad the web was back then, but Chrome came out during the three year pause between IE7 and IE8.

Why did it succeed? Obviously having Google pour resources (engineering/advertising) into it was a big component. But it’s also the case that Google pours resources into plenty of things that go on to not succeed.

That is in fact one of the things I take solace in when I feel grim about technology: even with all the money in the world, company X still cannot make a Y that customers love. You can substitute in your own X/Y for whichever megacorp you’re feeling bad about. Amusingly, when I was first drafting this paragraph, the news came out that Google gave up on its Facebook clone, as if to perfectly demonstrate this point. Or think of Microsoft and Amazon’s attempts at phones.

As best as you can be objective about the thing, I think I can say that the original product we launched was genuinely good software. Some of the things we did, like OS-level sandbox isolation between pages, took literally years for any other browser to implement.

And a big part of that is because we had some unexpectedly great engineers show up. It's neat to ponder over how for however many people you know about from their internet presence, there is a sort of "dark matter" of other super smart people you never hear about. For one example I think of the very talented Antoine, author of gnarly comments like these and seniormost person on ChromeOS, who appears to be otherwise mostly invisible on the internet.

Sometimes I suspect that my own biggest contribution to the project overall was just inviting agl to work on it. (In retrospect, did I? Maybe it was all his own plans within plans...)

But I’ve also seen plenty of good engineers make unsuccessful software. If I try to pinpoint the one thing we had that other projects I’ve worked with have lacked I think I’d call it alignment: the product had a clear purpose that everyone knew and were all heading in that direction. I have a memory of second-guessing some random design detail and Glen, our designer, describing all the factors he'd balanced to make the particular design decisions he had. From that point forward I had such confidence that he knew exactly what he was doing.

Maybe this is just saying that we had good leadership. Sorry to not have a more specific secret recipe for success. There’s of course always a massive luck component.

The Glenhead. Chrome originally was Windows-only and had very Windows-looking buttons in its custom title bar. I had been nagging Glen to give us some Linuxy buttons, and as a prank I replaced the close button with his face.

In retrospect... I still think this is funny. Though also I learned the reason he hadn't given me the buttons I wanted was because he was stressed for time about something else at the time, so it was a little inconsiderate of me. But I think he today also thinks it's funny so it's all good.

Polish. When we were about to ship the very first Chrome release we picked a release date and did the standard "burn down the last bugs you can and throw everything else overboard" as the date approached. For example I had worked on an RSS-related feature that we jettisoned at launch time and never brought back. And then some unrelated crisis occurred — how sad is it that I cannot remember which? some news story that would've overshadowed ours, in any case — which caused us to delay the release for a month.

This gave us an additional month where we couldn't really add features, but instead just had leeway to polish the thing, fixing all those last-minute detail bugs that you normally wouldn't have time for. In retrospect that is a rare relief for a product and I think it made the initial release more impressive than it otherwise would've been. It's strange how important polish can be for some products, when simultaneously it's so hard to motivate prioritizing polish work over feature work in general.

The comic. I am briefly in the Chrome comic on page 22/23. It was pretty cool to meet Scott McCloud. His book Understanding Comics is really great, worth your time. I'm pictured giving an example for Incognito mode around surprise gifts but I'm pretty sure the example I would've given is porn, not that that would surprise anyone.

The intent was to ship the comic at the same time we shipped the browser, as a sort of supporting document for the release. Due to a (shipping?) mistake, someone got the comic before we released the actual product, so there was a bit of press around the contents of the comic separately from reviews of the browser. In retrospect this was kinda great because it meant excited reviewers first engaged with the ideas of the product without being distracted by the visual details of it.

Version control. We initially developed Chrome using the Perforce version control system used for code at Google, then at some point switched to an internally hosted SVN in anticipation of publishing it as open source on Google's public SVN hosting service at launch time. When it came time to actually launch we discovered that the service wasn't designed to scale to a project as large as Chrome, which led to a fire drill around how to actually publish the code.

We ended up renting servers outside of Google's network and doing our own sysadminning of them. I recall personally getting tasked with figuring out how to block IPs from Iran — some dumb legal requirement around crypto exports — and hacking up a xinetd script. This story is probably the sort of thing that is familiar to anyone working at startups, but it was pretty abnormal in the context of Google where server management was extremely custom, well understood, and managed centrally. I think we couldn't use even Google hardware because it was such a foreign idea to run non-Google software in our own data centers, or something like that! It was definitely one of the most seat of the pants things I can remember working on.

Meanwhile, I was a version control nerd (I have written software using all of the weird ones — monotone, darcs, tla!) and was pretty certain that Git was going to be a big thing. So I wrote the Git-related tooling for working on Chrome to integrate it with our SVN-based code review process etc. In retrospect I think of this as a failure of leadership on my part. Many years later when Chrome eventually inevitably switched to using Git only, I think it was a multi-year project to dig out from SVN. Had I spent a little bit of time advocating for something that was pretty obvious to me I would've saved those people a lot of time.

Since I'm on the subject of being a version control nerd, here's a bonus fun esoteric Git feature: within Google I used git grafts to splice together the P4 and multiple SVN histories together into one big continuous history, which sometimes ended up being useful for code archaeology via git blame. I guess today that combined history might be something like 16 years long!

Working with Apple. About half of Chrome's code was our fork of WebKit, which we reintegrated into Apple’s upstream after Chrome released (only to eventually permanently fork again many years later, after my time). This meant that, shortly after our release, changes we made needed to be approved by Apple engineers.

In retrospect I enjoyed this a lot, for two reasons. One is that getting code review from experienced engineers is one of the best ways to grow as a programmer, and the Apple engineers came from a totally foreign engineering culture — no unit tests! no comments! — that also was clearly producing a high-quality product. The second is that the Apple approval process was a gate on us stuffing random Chrome-specific things into the web platform; you had to make the case that it made sense for Apple too.

However, I didn’t do much work in this area and I know it was the source of a lot of stress for others, in part because our fundamental goals were so divergent. While the individual engineers might disagree with this characterization, I think it's not a stretch to say that structurally Apple cares about making a browser that renders web pages just adequately enough to be acceptable and that real applications belong outside the browser. There were plenty of technical disagreements too. Maybe I just got lucky or I've forgotten, but I mostly have warm memories of the code reviews I got.

I mention all of this also to reflect on one especially weird part of all of this. At some point I was told to not offer jobs to anyone at Apple. The rumor was that we had successfully recruited someone important from the WebKit team and Steve Jobs personally flipped his lid about it. Years later I learned this was part of an industry wide illegal agreement. I eventually got a class action settlement check for something like $12k that I guess was meant to cover the wages I'd lost.

Missing stair. Years later at a party I met someone from Apple who worked on WebKit and I said, "Oh neat! I worked on Chrome so I touched that code too." And they asked: "What was the deal with [person X] on your team, why are they such a jerk?" (Sadly, if you worked on Chrome you probably know exactly who I'm talking about.) The experience gave me an appreciation for how a group, especially a faceless internet group, can be perceived by its worst behavior.

My "interview". At one point I returned from a vacation to find an email that said: "A Linux magazine wanted an interview with you but you were away so we wrote the responses for you, you can edit them if you like." In retrospect this was really not ok and I'm not sure why I didn't make a big fuss about it. You can still read "my" responses in the article, complete with eye-rollers like like "We built Google Chrome because we believed we could add real value for users".

Reward. Google had a program where they'd occasionally give large stock grants, outside of the normal compensation, as rewards for projects that were exceptional in some way. We got one — maybe the last? — for Chrome. (I wrote "last" but it's plausible to me they kept giving these kinds of awards just without announcing them.)

I mention this because when we learned we got the award, and each person was told how much they would get, one of the most senior people on the team announced soon after that he was leaving the company. I remember being surprised by it and asking him about it. His response has stayed with me, something along the lines of: "Once I knew the highest upside I would get for staying, I could better evaluate it against my other options."

I think there's a lesson there about how rewards can have the opposite effect you intend. And similarly, while I can see why a company might want to give out of line bonuses for behavior they'd like to reward, it ends up feeling gross for a company to dangle the potential of a future reward to retain people.

Web apps vs native. We take for granted today that you can build real software on the web. But here’s a story to maybe illustrate the era: at the time we launched Chrome, we had a search feature where the browser would display a tab that showed links and text snippets, like a web search results page. (More on this feature, which is gone now.)

I remember writing native UI code to do our own custom text layout and clickable links for that, despite the intended appearance and behavior of the page looking just like a web-based search results page. This is to say, at the time using HTML to implement anything you cared about seemed horrible, even to us browser developers, even for UI that was supposed to behave like a web page!

From my experience on that project I later did some of the work in Chrome to expose HTML-based browser UI and I remember advocating for it within the team. Since then a lot of the browser's UI is implemented in HTML. (Do you remember that Chrome's settings UI used to be all-native?) Of course, now in today's Electron world it's becoming increasingly rare for a desktop application to not be implemented in HTML...

Chrome vs the web. When we were small, sites wouldn’t work around our bugs. We had to work for them. Here's a WebKit hack that specifically worked around a bug in MediaWiki (aka Wikipedia). It was added in 2009 (!) and hung around for a decade.

At one point the question came up: how many redirects should a browser follow? Say you try to load A, but it redirects you to B, which redirects to C, and so on; at some point you should give up and say something is broken. Someone picked some reasonable-seeming threshold, like 10 or so, after which Chrome would give up. Then Darin, who had previously worked on Firefox, said "no, it has to be 30, or it breaks the New York Times — I know this from experience". (Today Darin is working on another browser, hopefully getting a third use out of that hard-won lesson.)

Then Chrome became big and sites started testing to make sure they worked around Chrome’s bugs. Then Chrome became enormous and started dictating what sites are allowed to do. I think this can come from a good place (for example, why even static websites actually need https) but it makes me anxious because there’s no structural reason these decisions ought to be good.

I have read that one reason the US government is so ineffective is that the founders intentionally designed it to be hard to make changes, under the reasoning that it’s more important to defend against tyrants making sudden changes than it is to change the law easily. The web moves more quickly, but we're mostly relying on the benevolence (or equivalently, we're victims of the malevolence) of a few dictatorial actors. The failure of the standards process that led to the HTML5 schism demonstrates why having an arbitrary body define standards also doesn’t work — it’s a similar problem of aligning incentives. I think there is no easy answer.

ChromeOS. I remember getting asked if I wanted to work on this "Chrome OS" idea — my baby Linux Chrome was an important piece of it — and being really skeptical about it.

But ChromeOS surprised me in how great it turned out, particularly on the security and updates front. They somehow managed to again get the right people in the right places to make a genuinely impressive thing.

Chrome today. Once we got Linux Chrome mostly working I think I didn’t realize that my work was done. At the time I’d joke that "Chrome 12 [an old version] was the best Chrome", but since I’ve come to see underneath that for me what I wanted was a fast tight browser, and once I had that, pretty much by definition most subsequent work was just gonna make it worse. Sure, there’s work like fixing rendering crashes or improving scroll performance, but most work involves adding features that make it bigger or slower.

Longer term, what I have come to appreciate about this is that I put a lot of myself into the product and I am proud of what I made at the time, but that product is a living thing that I do not control and it has gone on in directions I wouldn’t have taken it. I'm especially disappointed (or maybe "distraught" is a better word) by Google's progressively more casual relationship with privacy. Old Google was a lot more respectful than what you see today.

It's also funny to reflect on how much time we spent agonizing over making the most minimal thing — Chrome initially had no "Home" button and I remember there being long arguments about whether adding it even as an option was contrary to the goal of the thing — vs today's browser which is covered with buttons and settings, just like the browsers it was intending to revolt against.

The fate of the web. I am kinda gloomy about the web. After Chrome I worked for three years on a product whose founding goal was trying to figure out some way to "save newspapers".

Everybody hates ads but nobody has a plausible plan for how journalism gets funded; that project tried a thing that I think also didn’t work out. Inevitably in discussions like these someone will say something unhelpful like "if the newspaper had reasonable payment systems I’d pay for them" or whatever but the gulf between hand-waving about this and actually making the numbers work out is massive.

Today's web feels like it's in a similar spiral to the bottom around ads. It's possible to make some pretty neat stuff using web technologies — witness Figma, for example — but it's easier to make user-hostile garbage and so that is what we usually get.

People seem to think it's the browser's job to block ads, but my perspective is that if a business owner wants to make their business repulsive, the only sensible response is to stop using the business. Somehow once technology is involved to abstract what's happening, people start talking about how it's their right to unilaterally renegotiate the transaction. Or for another analogy that will likely make you upset: "I hate how this store charges $10 for a banana, so I am just going to pay $2 and take the banana anyway".

Meanwhile, the only real non-web platforms are run by companies (Apple/Google/Microsoft) who aren’t afraid to stomp on competitors, often in ways that are illegal or that ought to be. Just in this post I have instances of all of those companies behaving badly. So maybe I'm just gloomy about where technology is headed in general!