Ninja, a new build system

February 06, 2011

When we first started porting Chrome away from just Windows, we intended to use Scons to build Chrome on all our platforms. But early on in development I discovered that Scons, despite its admirable goals of correctness and ease of use, was quite slow — it could take 40 seconds from starting Scons before it decided to build some source. I don't necessarily fault Scons; Chrome builds as one single enormous executable with something like 30,000 inputs (including the entirety of WebKit) to the build system.

I eventually ported our Linux build to plain Makefiles. (This was only possible because in the meantime we migrated to a meta-build system which generates native build files on Windows and Mac.) In doing so I become more and more obsessed with build performance; I once clocked our Windows build taking eight minutes to finish linking after a one-file change and I found it devestating for both my productivity and my morale.

Build performance is a combination of many factors: the build system needs to compute what needs to be done, the compiler takes time to compile files, the linker takes time to link. Each of these can be improved; in reverse order, the gold linker written by fellow Googlers is lightning fast; compilation time can be improved both by faster compilers (see my recent work on Clang) and by better parallelization (I spent a lot of time fiddling with and coaxing coworkers to run distcc); and the build system itself can be improved.

Combining all of the above tools with some carefully-generated (non-recursive) Makefiles got our incremental builds down to a handful (10-20) of seconds. I am especially proud of our work on the Linux build because I realized early on our little port of Chrome was only likely to survive if I attracted other developers to use it. For example, much (perhaps all?) of the Chrome extensions team uses Linux as their primary development platform because, despite the lame debugger, the other pieces are so much nicer than the alternatives. (This is also why I am responsible for our support of Git, another tool that is significantly more productive than the alternatives and significantly faster on Linux than on other platforms.)

But I still wasn't happy about the ten seconds of waiting between running "make" and the first compilation step starting. It seemed to me that with a warm disk cache it shouldn't need to think that hard.

Our Makefiles use a bunch of clever hacks — a claim I can make because I stole most of the ideas from the Linux kernel build system — to make Make do things that Make doesn't do by itself. For example, when your compilation flags change for a given output, you'd like to rebuild the output file, which you can achieve by making every Makefile rule depend on a nonexistent input (so the rule always fires) and then manually judging whether you actually need to rebuild. (You also need to track what command lines were previously used to build a given output.) This kind of subversion of the way Make is intended to work likely is part of the reason our Make build wasn't as fast as I wanted.

In porting our build to Make I learned a lot about it, including some surprises. For example, try this:

touch foo.c
echo 'foo: foo.c' > Makefile
strace -estat make

You'll see that GNU make, as part of its build rules, looks for RCS and SCCS metadata! (If you're not familiar with those, they are used by an obsolete version control system. This behavior can be controlled by the -r flag to Make.)

And so with all of that in mind, and with a spare weekend to hack, I thought I'd try making a very simple build system; conceptually very similar to Make, but without hardly any features. (Why not just improve Make or some other build system? Because I was doing this for fun, and I wouldn't have done those other things for fun.) Once I had that I found it easy to hack in some missing features of Make; for example, when running build commands in parallel, I can buffer the output of all build commands so that error output can be tagged with the full command line of the failing command without interleaving with other output. And with just some preliminary profiling I got the Chrome build startup to under one second.

Combined with our other tools (in particular gold) and a fast computer, I have clocked an incremental build of Chrome after editing one file at six seconds. I got permission from work to open-source my new build system. I called it Ninja because it strikes quickly, and you can get it from github or read the manual for more on the underlying philosophy.