neugierig.org: Tech Notes

As promised as a followup to my beginner's notes on WebAssembly, here are some notes on WebAssembly specifically around C/C++.

Emscripten. Clang has built-in support for generating Wasm code, but there's a lot more to building code than just translating C++ into the appropriate machine code. Emscripten is the C++ to browser Wasm toolchain that manages this, giving you an emcc compiler that goes from C sources to .wasm outputs. For some examples, emscripten provides the linker and the implementation of malloc, as well as support .js files needed for the wasm to load, and it even goes as far as generating HTML files too.

The other big thing Emscripten manages is making existing C++ code work, doing things like shimming C++ calls like puts() into calling JS console.log(). For another example, when you write C++ GL code it shims that the C++ GL API onto browser WebGL calls, and similarly for lots of other C++ APIs; see their docs section on Porting. But for how Figma uses C++ we haven't used much of this, since we're generally writing the code from scratch to target browsers.

In all, Emscripten works fine but it's also pretty clunky. I don't mean to just criticize it — as a hobbyist I know that making things is so very hard — but rather to remark that there is a surprising amount of surface area in it, such as multiple APIs for interacting with JavaScript and compiler flags with names like EMULATE_FUNCTION_POINTER_CASTS. There's a particular irony in how Emscripten is kind of a JS tool but its JS libraries are structured as a bunch of globals.

Null pointers are surprising. In typical C environments dereferencing a null pointer crashes. In Wasm a null pointer just refers to memory[0] and it is a legal address to read and write from. C code that reads/writes a null pointer is something we generally try to avoid already, but in a Wasm environment null pointers feel a bit like On Error Resume Next.

To this non-expert, I have idly wondered if you could restore the C semantics around null pointer handling when compiling C to Wasm if you could instead make null pointers refer to some memory address so high as to trap. For example, I recall there are some language-lawyering rules around whether C's NULL is actually 0. Alternatively, I wonder if you could codegen every memory reference to instead refer to theAddress-0x1000, effectively shifting all memory down at runtime, which would then cause null pointers to underflow to similarly illegal high addresses. (I think it might not even cost too much — most of the Wasm memory instructions take a constant offset parameter already...)

Safety. On one hand, null pointers not crashing is pretty bad for running C. But on the other, it's interesting to reflect upon how much support C gets from modern hardware and operating systems. It's not just guard pages to make null pointers crash, but also overcommit and W^X pages and ASLR and CFI etc. etc., all in some sense to mitigate flaws that are solved by compilers/runtimes elsewhere. (For example, null handling in basically every non-C language doesn't rely on help from the CPU!)

The Wasm environment doesn't really have that same support, which means language-level safety can end up more important under Wasm. On the other hand, Wasm's Harvard architecture also just excludes by construction a bunch of C's problems such as ROP; the more I've worked with it, the more it feels like it would have been the "right" choice. And the consequences of bugs are potentially lower too, because a buffer overflow doesn't directly lead to a Wasm sandbox escape. Here's a nice analysis of a recent paper on the balance between these factors from one of my favorite blogs.

It seems likely to me that we'll see "escape Wasm sandbox into browser eval" types of XSS bugs in the future, because that is how security works everywhere: for whatever Wasm-specific security properties exist, that just means the bugs will be at the boundaries between Wasm and the host system. Though Wasm-caused XSS is a much different kind of compromise than attacking the browser itself, Wasm-based XSS equivalents targeting nodejs easily escalate into arbitrary code execution (see the above paper).

Memory layout. Wasm memory is one big flat buffer. To map C to this, emscripten puts the stack at some fixed offset and has it grow downwards, and has the heap start at the same point and grow upwards. If you look at a C++ file in weave you can see the initial stack pointer defined in the "global" section. (The paper linked from the blog post linked in the previous section goes into this more.)

Virtuals. In the other post I mentioned traps, which are cases where the Wasm machine stops your program. At Figma we encounter traps in an interesting way related to virtual calls.

In C++, given some struct Iface { void foo(); }, if you call ptr->foo() when ptr is null, nothing goes wrong immediately; it just invokes foo with this == nullptr. And Dereferences like this->someMember just read low memory addresses which are all legal in Wasm.

But if foo is a virtual method, things get more complex; look at the generated output here. Per the C++ semantics, a virtual call first looks up the vtable by dereferencing the pointer, and then gets the address of the target function from that table. (You can see this in the godbolt output by the pair of i32.load instructions.) If there are any nulls involved here it's still no problem for Wasm because you just get a garbage index back. Finally, there's a Wasm opcode call_indirect which calls a function selected by a runtime-computed index.

If that index refers to a function outside of your program's declared function array it traps. But what if after dereferencing some nulls you happen to accidentally refer to some existing function? In the above output, note how the type of the expected target function is included in the call_indirect instruction.

The reason function types show up in the Wasm semantics is something about the Wasm soundness properties, where as I understand it, it needs to know what value types are on the stack at any point. But what it means in this case is that you have a garbage function index, and if that garbage index refers to a function with a type different than the one you expect, Wasm will trap.

To summarize this section: null pointers don't crash in general, but null pointers to virtuals can, if they happen to accidentally refer to a function with a different Wasm-level type. What's gruesome about this is you can have some code involving null pointers happily scribble all over memory and keep running and the only way you'll ever notice is if it eventually traps in a virtual call.

Indirect buffers. One interesting pattern we rely on at Figma is called IndirectBuffer. The basic idea is you can allocate an array in JavaScript and still manipulate it from C++ by exposing calls on it, even though the array's bytes itself never live in C++ memory. This ends up important because Wasm memory is limited to ~4gb, but JS memory isn't.

In particular, Figma documents deal with a lot of images, so we try to keep the image pixels in JS or GPU memory while still rendering the scene from C++. For one example of this, one project I worked on at Figma was related to the "save current document as a file" functionality, which needs to serialize all the current document's contents (including image pixels) into one large buffer. The loaded document itself is already eating most of your Wasm space, so we moved a lot of serialization output into JS-level buffers, despite all the serialization code still living in C++.

As a kid I got my programming start in the DOS days and I still remember segment registers and far pointers. The whole thing gives me fun memories of that time. But this problem is also possibly fairly specific to Figma, which will happily eat all the RAM you can give it.

It's still early. Work is ongoing to hook DWARF debug info into the browser devtools. It looks pretty promising, but the tools are still pretty early. It's out of scope for this post but Figma has a pretty amazing/bonkers setup where we can build and debug the app C++ portions using a native toolchain despite the app being mostly written in HTML, which we keep alive mostly because the native development experience is still a lot better.

That aside, if you have a C++ call stack that calls into JS and then calls new Error(), the resulting object has C++ frames on it, and with enough machinery you even get symbols. I mention debugging mostly to mention that it's improving rapidly (it's pretty neat to single-step in a browser through a C++ stack!) and also to link you to another really thorough blog post.

For another example, in the emscripten toolchain the stack of tools used to link is Wasm/emscripten-specific, which means they've not seen the years of refinement to both run quickly and produce tight code you get on other platforms (particularly Linux).

One more "it's early" example, though I guess now that I'm on the subject I realize this has nothing in particular about C++: we recently hit a code generation bug in Firefox's Wasm optimizer. I am still kind of amazed that my coworkers were able to distill a runtime Figma bug into such a small repro, and further impressed by how the Firefox engineers were able to write a patch for it 90 minutes after the report(?!).

In all, developing C++ on Wasm feels like what I imagine developing C++ for something like an embedded system might be like: it's definitely still C++ but the tools are a bit weaker and different in random ways from the tools you're familiar with.

The fate of C++. As of this summer it's been 25 years since I had my first programming job, writing C++ for a dotcom startup, and here I am today still writing C++ for a dotcom startup. From that perspective, learning C++ was a wise investment. But I think many people with similar C++ experience would agree with me that it's pretty rare for it to actually be the right tool to use for software these days.

In particular I think of C++ as an expert tool in a bad sense, where if you make a mistake it fails maliciously: with either silent memory corruption or silently bad performance due to accidental copying. Meanwhile, as the Zaplib post-mortem also found, it's not obvious a better performing language actually easily translates into a performance win, and if not for performance there's even less reason to be writing C++.

With that in mind, I see the future of C++ and Wasm to mostly be around incorporating legacy C++ code, not writing new code. That is mostly just a statement on how I feel about C++ in general, but Wasm in particular breathes new life into being able to wedge old C++ code into new contexts. For example, Mozilla shipped a pretty amazing hack that uses Wasm to sandbox C++ code used within Firefox itself: they compile their C++ to Wasm and then link it back in to the larger C++ app, with a runtime effect almost like running the C++ submodules within an emulator.