Tech Notes

Adventures in heap profiling

2012-03-06T17:06:00Z

There's an obvious metaphor for heap profiling that I've yet to find a good opportunity to use, so here goes. Imagine you're investigating a company that is spending too much money. You have all of the credit card bills, but they only implicate the person who signed for a given purchase. Ideally you'd track down who authorized too many purchases. But then if you trace the management hierarchy all the way to its root, the CEO is ultimately responsible for all spending, and that's not a useful fact either.

In code, your credit card bills are the calls to malloc() and your CEO is the main() function that invokes all the calls. Heap profiling, then, can be broken into two pieces: one, collecting the chains of authority, and second, analyzing those chains to allocate (pun intended!) blame.

Let's start with the first one. Collecting a heap profile is as simple as recording the call stack at each memory allocation point, which is to say it's not exactly simple.

First you must hook allocation. Glibc provides hooks for malloc, and if you're using a custom memory allocator like tcmalloc you're already doing this as well, but there's also various pieces of your program like C++ or glib which may use its own allocation pools (for glib, for example, you can adjust G_SLICE to aid debugging).

Once your code is getting called at the right places, you must find a way to get a stack trace at runtime. One approach is to look at the stack for return addresses. It appears gcc for x86-64 Linux provides API for such at thing, but check out the gnarly code Google uses for x86; libunwind is a more recent implementation that likely works better. Another approach is to record extra information while the code is running; tcmalloc can use gcc's -finstrument-functions flag to hook every function call with a bit of code that simply maintains the call stack as an array on the side (a "shadow stack").

Now you've collected a bunch of allocation stack traces, and it's time to shuffle them together. Typically this information is presented as a "top N" list or a tree widget, but tcmalloc provides a cool tool called pprof that generates those as well as input to graphviz to generate pictorial graphs. You can see a snippet of such a graph for GFS, the Google File System, on on the tcmalloc heap profiler page.

However, to map the addresses seen in the recorded call stacks into human-readable function names, pprof relies upon helper tools from GNU binutils such as addr2line, which turns out to be agonizingly slow -- for Chrome I clocked it as taking over 4.5 minutes to load a profile! For fun I recently dug into how this all works, and so I have written two programs to share with you.

The first I wrote is maddr, a halfway reimplementation of addr2line. To map an address to a line number in code you rely on debug information, which is in an awkward format described in the DWARF spec. The line number info is a bytecode format that generates a large table mapping every offset within the binary to a source file and line number; maddr decodes that table into memory and does binary searches over it to quickly answer queries. In principle this could be extended slightly to support the same formats addr2line uses, and then plugged directly into pprof.

But I was curious about how the other pieces of all of this work, like how pprof shuffles the stacks together, and so my second program is the boringly-named hp, which loads the heap profiling output as emitted by tcmalloc and generates graphs. hp is written in Go which makes concurrency so easy that it trivially uses multiple threads to load a heap profile and the symbols from a binary in parallel. It then has a "server" mode which brings up a web server, letting you adjust parameters (well, one parameter, but you get the idea) dynamically by clicking around.

You can play with a demo of hp's output online. (Caveats: it's a static snapshot so "rerender" doesn't work in the demo; it uses WebKit flexbox so it only works in WebKit. Use the middle mouse button to pan.) You could easily extend this so that you could click on the graph nodes to focus in on a single call stack, but I think I've exhausted my curiosity at this point (and I no longer work on Chrome) so that's all you get!

Sabbatical

2012-01-16T19:44:00Z

In late September I took a three-month sabbatical from Google. (This post was written both during and after the sabbatical; sorry for the mixed tense in some of the wording.)

I told my managers: in the old days my officemates would make fun of me for being excessively optimistic and happy but more recently I had grown a reputation at work for being grumpy. I wanted to figure out whether it was my work or something else in my life that had caused this change. And thinking back, even through college I always took classes through the summer; I think my last real break from work of some sort was maybe my first year of high school, before I started working in the summers.

I've read about experiments where people stay in a place without the sun or a clock to guide their sleeping behavior, and how they fall into a 25-hour schedule of sleeping and waking. I was curious how my life would similarly arrange itself without constraints.

So I made nearly no plans for my new free time. My wife works and I didn't want to leave her behind on some trip around the world. Instead, I had the vague idea to relax, work on projects, and catch up on video games. (I went cold turkey on games early on in college in an attempt to focus; in retrospect, putting Linux on my primary computer to help enforce that was likely a valuable career decision.)

Here's what I found.

Work vs play. Even when I was working, I spent a lot of my free time hacking on software. It's something I enjoy; it's something I'd do even if it weren't something I could be employed for, so it's great people are willing to pay me to do it too. Without a job structuring my hacking time, I found that I fell into a pattern of hacking from morning to mid-day, then I goofed off for the brain downtime that frequently accompanies lunchtime through 3pm or so, and often returned to hacking in the evening.

When I was working I think I got my most productive time in after lunch. I think the reason mornings worked for me now is that there is nothing extraneous between me and my goals. At work there was always side tasks like email and keeping up with daily churn to occupy my mornings. With my new perspective, I wonder how much of that was self-inflicted; I'm considering strategies like "don't open mail until after noon" when I return to work.

Video games. The parallels to addiction are obvious to me, especially the modern horrible stuff like Zynga products. I think the way they hook is interesting to reflect on for a moment.

The joy of a good video game is the feeling of achievement you get. That is, I think, why they often hook the same personality type as a programmer's: making a WoW character and writing code have the same reward feedback loop.

But since a game is synthetic, the achievement is also ultimately synthetic ("I pushed the buttons at the right times so that the reward light turned on"). The cynical craft of modern video games is tricking the gamer into feeling accomplishment without actually making them work for it. I read a review of one recent game (perhaps God of War?) where the reviewer discovered you could beat the entire game just by pressing the single attack button repeatedly -- your character doing amazing leaps and smashes throughout -- and it's interesting to contrast that with the reviewers trying to find the words to describe the sense of "real" accomplishment found in a truly hard game like Dark Souls. (I haven't played either.)

With all that in mind, I think there is room in my life for video games just as there is room for alcohol. I'm not sure which, though; Skyrim is really pretty but I was let down by how every problem was solved by entering a dungeon and killing everything. Starcraft II is pretty amazing, though.

Getting things done. A few days into my leave I got out a piece of paper and made a list of all the sorts of menial life tasks I've been intending to do but putting off, stuff like "figure out to do with that cardboard box full of CDs". I worked through some of that.

Most stuff I own that lives in a closest is trash, almost by definition: I haven't used it. Much of it is in fact difficult-to-dispose-of trash, like old hard drives where I was afraid of leaking data. Or those CDs: given that I've re-ripped them, what is the ethical thing to do with them?

Living your dreams. When I was planning my leave I had wild dreams about getting fit or learning Arabic or whatever. Once I no longer had my job to blame for it I was confronted by what I already subconsciously knew: my own motivation is at fault. I was using "no free time" as an excuse.

Daisy asked me recently: "Should I take a sabbatical too? Will it give me a chance to finally do all those things I've wanted to?" And my response was, "No, if you really wanted to do those things you would've found time for them already."

Money is a means to an end, not a goal. I have been very lucky in life, to have been born white in this country to a middle class family, so that I can be in a position to afford to not work. But I'm not some tech startup lottery millionaire; I'm paid like an engineer and I live well within that.

Do I enjoy the nicer things in life? Yeah, sorta, but not enough to strive for them. (Or, to use the "Your Money or Your Life" computation, it rarely seems worth it to spend another hour in a cubicle to have a more tender piece of meat for dinner.) It always felt so natural to work all day because that's just what people do, but seeing my wife leave in the early morning and return as the sun is setting really drives home for me how unnatural it is.

What now? I'm back at Google, still a little unsure of myself. I get no shortage of job offers but it's not clear to me that anyone provides a opportunity with as much freedom and the opportunity to work on free software as Google does. If you are also a Googler and have the opportunity to take a sabbatical, I highly recommend it.

Linux on a G4 Mac Mini

2011-12-21T18:05:00Z

I wanted to put Linux on this old Mac Mini so I can use it a file server. Some install instructions suggested burning a CD, but I don't have a CD burner handy. Others suggested booting off a USB stick, but the newer Debian/Ubuntu installer docs removed that section, and it's not clear how to partition the USB device. What to do?

It turns out it's relatively easy to completely install via the network due to Open Firmware's tftp support. You'll need a pretty standard networking setup (DHCP) along with a Linux box and a network wire for the Mac.

Start by searching for [debian netboot], not to be confused with netinst: the former is how to net boot the installer, the latter is minimal CD images that download the remainder of the install from the internet. What you want is the netboot version of the netinst installer, and you'll find it in the "network boot" section of the above page.

For a G4 Mac Mini, follow the link to the powerpc images, then navigate to powerpc/netboot and download the following files into a directory:

yaboot.conf, configuration for yaboot
yaboot, the boot loader
boot.msg, the bootup message
vmlinux, the kernel
initrd.gz, the rest

(How to remember this file list? You need yaboot.conf and all files it references; it's a short text file.)

Next, install tftp on your hosting machine. Run it like:

sudo in.tftpd -L -s path/to/files

It runs as root due to the port it uses, -L puts it in foreground mode, and -s makes paths relative to the directory you give it.

To verify that tftp works, try a command like:

tftp localhost -c get yaboot.conf

Fix your setup if that command results in an error (for example, I learned the hard way that the Mac doesn't like pulling files from a subdirectory).

Then boot the Mac into Open Firmware by holding Windows-Alt-O-F (that's Command-Option-O-F on a Mac keyboard) as it boots. At the prompt, tell it to fetch yaboot:

boot enet:192.168.1.114,yaboot

(adjust the IP as necessary).

From that point, just follow the menus. Simple.

(But actually it was not so simple to figure out the above. If you run into yaboot errors like can't read elf e_ident/e_type/e_machine info or the installer being unable to detect your disk, I found both of those were solved by using the latest version of Debian, and not Debian stable or the most recent PPC Ubuntu.)

PS: wow was OS X slow on this thing. I can't believe it was considered tolerable at the time of release.

Nonblocking disk IO

2011-12-20T20:29:00Z

It's natural, when writing an event-driven application, to want to perform disk operations like file reads and writes in the same non-blocking manner you use for sockets.

This turns out to be hard. In theory there's POSIX AIO, but it's reported to not really work on Linux (my info might be out of date). Async libraries like node.js use an internal thread pool to simulate its desired event-behavior for files.

Sometimes in discussions about this people point longingly at Windows, which does have an API for performing overlapped disk operations. See Ryan Dahl's nice overview for a ton of links. In fancy apps like Chrome, subsystems like the disk cache use overlapped IO on Windows and threads on non-Windows, with the hope that Windows async IO involves less overhead (less bookkeeping, fewer copies, etc.).

But it turns out that Windows async IO is just broken. In any of a number of situations, including if your disk is encrypted, Windows will silently make your async file operations synchronous. See this MSDN doc for a list of other potential reasons. Special highlights include how extending a file's length is synchronous unless you use a special API, while that API on NTFS file systems requires a special privilege that is only available to administrators by default.

I don't write this to just say "man, Windows sure sucks" -- the Linux situation is worse, and this may well all have been fixed in Windows 7. But rather I observe that there is just a ton of API surface in an operating system, and any of it may block (see e.g. Brad discussing sendfile). (In Chrome's case, we found via instrumentation that real users were encountering browser hangs because its supposedly async disk interface wasn't async for seeking within a file.) In practice for anything you want to be truly asynchronous you probably need to use a thread. You can still use the AIO APIs, if you have some that work, from that thread.

There are two ways to interpret that conclusion. One is that trying to make a synchronous world async in a piecemeal fashion doesn't work, and instead it's better to make it easy to coordinate synchronous tasks -- the Go model (see rsc's convincing slides). The other view is that the problem is the synchronous system, and any work we can do to move away from that the better -- the node model, where you replace the world.

Return by value

2011-12-15T17:58:00Z

So you've been writing some C after using Python or Go or Haskell or pretty much anything other than C and you're jealous of being able to return more than one thing from a function. How do you do it in C?

The standard way is to return one thing (perhaps the "primary" thing you're returning) as the return value, and then have the caller provide pointers for the rest as "output parameters". You could return everything by value in a struct, but it feels like all those copies might be bad, maybe?

Let's check. To reduce inlining confusion, let's split it across multiple object files. So here's the interface:

typedef struct {
  int a;
  int b;
} Pair;

Pair return_pair();
void fill_pair(Pair* p);
void fill_ints(int* a, int* b);

And the trivial implementation:

#include "lib.h"

Pair return_pair() {
  Pair p = { 3, 5 };
  return p;
}
void fill_pair(Pair* p) {
  p->a = 3;
  p->b = 5;
}
void fill_ints(int* a, int* b) {
  *a = 3;
  *b = 5;
}

And finally here's main to run it, including calls to the functions so we can see what work the caller must do.

#include "lib.h"

int main(int argc, char** argv) {
  Pair p1 = return_pair();
  Pair p2;
  fill_pair(&p2);
  int a, b;
  fill_ints(&a, &b);
  return p1.a + p2.a + a;
}

And now to a disassembler.

Starting with the last function, fill_ints(). Passing in two pointers means that two registers get addresses put into them:

   0x00000000004004e7 <+23>:    lea    0x8(%rsp),%rsi
   0x00000000004004ec <+28>:    lea    0xc(%rsp),%rdi
   0x00000000004004f1 <+33>:    callq  0x400530 <fill_ints>

and the implementation of fill_ints() fills in the pointees. Pretty much what you'd expect.

Dump of assembler code for function fill_ints:
   0x0000000000400530 <+0>: movl   $0x3,(%rdi)
   0x0000000000400536 <+6>: movl   $0x5,(%rsi)
   0x000000000040053c <+12>:    retq

The fill_pair implementation is similar, but with just one pointer and two offsets.

return_pair is quite different:

Dump of assembler code for function return_pair:
   0x0000000000400510 <+0>: movabs $0x500000003,%rax
   0x000000000040051a <+10>:    retq

Because two ints fit in a 64-bit register, the whole function can be implemented with one immediate load and no memory accesses!

But surely, you say, that's just because your Pair type is simple. How about pointers? If the second field were a pointer, it wouldn't fit into a single register.

Here's what a pair of an int and a pointer compiles down to:

Dump of assembler code for function return_pair2:
   0x0000000000400510 <+0>: mov    $0x40062c,%edx
   0x0000000000400515 <+5>: mov    $0x3,%eax
   0x000000000040051a <+10>:    retq

Again no memory references, just registers.

Ok, how about something that can't fit in multiple registers? Like say a buffer.

typedef struct {
  int a;
  int big[1024];
} Pair;

Pair return_pair3();

and the associated code:

Pair return_pair3() {
  Pair p;
  p.a = 3;
  p.big[0] = 5;
  return p;
}

Here's the dump:

Dump of assembler code for function return_pair3:
   0x0000000000400510 <+0>: sub    $0xfa0,%rsp
   0x0000000000400517 <+7>: mov    %rdi,%rax
   0x000000000040051a <+10>:    movl   $0x3,(%rdi)
   0x0000000000400520 <+16>:    movl   $0x5,0x4(%rdi)
   0x0000000000400527 <+23>:    add    $0xfa0,%rsp
   0x000000000040052e <+30>:    retq

To "return" a large structure, the caller provides stack space for it and the function fills in the caller's copy -- sorta like the return value optimization. This code is the same as the code that explicitly passes a pointer. (I don't get why this function adjusts %rsp, it seems like it doesn't even use it...)

In each of these cases, returning by value seems to equal to or better in terms of generated code to the approaches using pointers. So why not do it?

Here are some reasons. (Note that I'm avoiding C++ here, which has its own additional complicated rules as described in the above wikipedia article.)

Most importantly, you need to create a new tuple type whenever you want to pass more than one value around. It is inconvenient, especially when the caller already has a variable handy for the value it wants to get back from the function and could just pass its address.
Passing structures via registers appears to be a newer ABI; gcc has -fpcc-struct-return and -freg-struct-return to select between them. But my system appears to be built with return-via-registers on (it appears that it as introduced into gcc around the year 2000) and even when I manually select returning via memory it just means return_pair and return_pair2 decompose into the behavior of return_pair3.
If your structure contains any character buffer the function gains a bunch of checking code due to -fstack-protector, removing the benefit.
For larger structures, you may have to worry about stack space. But such things don't belong on the stack in the first place; you are working with pointers to them to start with so functions that fill in those pointers are more convenient anyway.
(This point and the following were contributed by Jeffrey Yasskin after the post was first published.) If you return several different variables, depending on conditions inside the function, NRVO doesn't kick in. This is often a missed optimization in the compiler, but we still have to deal with it.
If the return value owns some allocated space, you can often save allocation time by passing in a variable that already has the space allocated.
Insert your reason here. What else am I missing?