neugierig.org: Tech Notes

This post is part of a series on Theseus, my win32/x86 emulator.

This post talks about AI. This text is wholly human-written, and most of my code is too. I talk a bit about how I feel about slop in the post.

retrowin32, my previous emulator, never supported DOS. Recently I added preliminary support for it in Theseus, which can now convert a DOS executable into a native Rust binary. It's not complete but I got it to run a few programs. This led to a maybe surprising new direction that the latter half of this post goes into.

I have just the vague memory of DOS from my childhood, of tinkering with configuration files I didn't quite understand, fiddling with IRQ conflicts, puzzling over terms like "high memory area". I guess retrocomputing is often motivated by nostalgia, and it's definitely been nostalgic to come back with a computer science education to understand these things.

But what is there to post about it? I think of software projects as two halves: learning new things and slogging through uninteresting details. Without nostalgia to motivate your interest, I would say it was mostly just slog.

Theseus's x86 emulator already supported most of the needed instructions (they are used to work with 16-bit data in 32-bit mode anyway). Memory accesses and jumps have to be careful about segment registers. There's a bit of new code for some DOS subsystems for things like timers and hardware access.

The program I targeted was Dr. Sbaitso, or in particular the voice synthesizer from it. I have a vivid memory of the way it sounds when it first greets you. (Mihai of Infinite Mac fame told me that one frequent use case of his emulators is people who want the voice from Radiohead's Fitter Happier.)

The voice synth turned out to be surprisingly complex. The program READ.EXE accepts the text to speak, but it relies on the sound engine SBTALKER.EXE first being loaded as a TSR — another childhood term I never understood. It turns out a TSR is literally just a program that stays in memory after quitting, which means another program (if it can find the right address) can call into its leftover code. Meanwhile, SBTALKER.EXE loads a third executable "sound driver" BLASTER.DRV to handle its interaction with the Sound Blaster sound card.

Friends, I did the requisite slog to get this thing to make a sound and I kind of regret it. The three programs load and invoke one another via DOS system calls; the driver writes to sound card IO ports; there's a DMA chip used for transferring audio data around; also an interrupt controller. All of which I had to emulate. Even writing this paragraph it feels like I'm enumerating Pokemon and it is making me a little bored.

I had thought that getting sound working would be a step on the path to running an old DOS demo in my browser. And then I saw someone had translated some old DOS demos directly to the web "using AI" and I kind of got discouraged. Why go to all this effort when it's easy to get the same results?

I am not an AI hater, but I am not an AI lover. I guess I am an AI cautious user and observer. But probably much like you, it's pretty uninteresting to me when someone just prompts out an end result. Why is this objectionable, while using other tools like a compiler isn't? How does AI fit into my future work? Here's where I'm currently at on these questions.

I wrote above about how projects are some part learning and some part slog. For me, the interesting part is mostly the learning. Today's AI can effectively eliminate slog and at first glance that sounds great.

But in my experience the two sometimes aren't easily separable. Sometimes the slog is part of the path to understanding; you can read a book about math and think you've understood it, but sometimes you also have to grind some math to actually internalize it. (I used to run a club where we read computer science papers. Frequently once we met and tried to exchange notes we'd discover we hadn't understood the paper at all.) This is to say in a world without any slog you sometimes end up without any learning too.

That doesn't mean we hobbyists must throw away all our tools and return to nature. But it does mean you must use your judgement, to know that sometimes reading the summary is just not a substitute for reading the actual book. The software artifact is like a photo we take after climbing the mountain; the photo isn't the point, it's a memento.

So I think that's where I draw the line about AI in projects: use it as much as you like, but for me to care about it there must be some part of it that leads to growth or is interesting to read about.

I have been thinking about the above because of another piece I made as part of the DOS project. It is deep in AI — judge me as you see fit — but it's deep in learning about driving AI, not just a prompt.

Part of the work in extending an emulator, as this DOS work did, is building some understanding of what a new target program is trying to do. I've mostly used Ghidra for this. It disassembles a program, shows you how things cross reference, lets you label functions and memory, and even has a decompiler that attempts to reconstruct C code from the assembly.

Ghidra is powerful but clunky. The point of a tool like Ghidra is to dig through the program to understand it. I don't actually need C code as much as I want to know what the function is doing at a high level.

It turns out LLMs are pretty good at reading assembly. I was curious how you might integrate them into the understanding process. You can just have an agent drive Ghidra, but there's a lot of Ghidra I don't really use. You can probably point an agent at a raw binary and tell it to just figure things out, but the output ends up difficult to constrain. I've seen plenty of slop projects on GitHub that are covered in nonsense documentation, where it's never clear what is vestigial or what matters.

Instead, I made a tighter thing, a headless disassembler. It manages a database of program code, including derived data like basic block boundaries and the function cross references. (To resolve where indirect function calls point to, it even does a tiny bit of what I guess you'd call abstract interpretation, maybe a topic for another post.) Then I gave it an interface to store annotations on that code like function names, parameter descriptions, block labels, and so on. Then I made LLMs drive that, in two different ways to better understand the tradeoffs.

The first way you can hook an LLM into a program like this is by having the program make the LLM calls itself. For each unlabelled function, prompt the LLM to label the function. The problem of labeling functions is a little more complex than just making an API call.

The first detail is that to understand what a function does you (and also the LLM) often need to read the functions that it calls and that call it, recursively. But you also don't want to feed in the whole program as input because it's too much code. The agenty way to do this is to let the LLM choose what code to read, in the form of "tool calls". You start with a prompt that asks to label just the function you're interested in, but you also expose to it a function read_function(addr). The LLM "calls" this function by generating output that looks like a function call, and you pass the result back by putting it in the context on the next LLM call. You then run it in a loop until the LLM is satisfied.

The second detail is that you eventually want to output something more structured than plain text from this process. For example, I want to gather info about the function arguments with names, descriptions, and types. There seem to be two ways of doing this. One is that LLMs can produce "structured outputs", where you give the expected schema as JSON and it generates something matching that schema as its output. The other is to just provide a write_output tool. It's not quite clear to me when to prefer one over the other. I think structured output is for when you're making a single request-response call, while the tool calling interface requires an agent loop but allows single prompt to potentially write more than one result. (In my full program, the agent writes notes about functions and also about global memory locations that different functions interact with.)

To put this all together, I found openrouter pretty useful for abstracting over LLM companies — it feels pretty nice to be able to switch between models and providers with a simple string change. (Google's models reportedly score well but trying to pay for a service from Google sounds like a big hassle.) I also learned there is a nice Rust library that generates JSON schema from a struct that has doc comments, which means you get the relatively magical experience of writing a typed struct of what information you wish you had and the AI plumbing magically summons it.

In all, going this way means building my own agent loop. It is a decent amount of plumbing and fiddling, and also a little weird to tinker with because each call costs actual money. (It's just something to keep an eye on, not a real problem. At the scale I work at, the actual money is fractions of a penny. I've spent maybe $4 total on openrouter across all my AI experimentation so far.)

Also I learned, perhaps unsurprisingly, that people who work in the AI space are very AI-pilled. Filing a bug on a library I used led to bot comments writing questionable bot code fixes and it all feels pretty sloppy and unsatisfying.

There's a different way to integrate all this that is much simpler. Instead of worrying about openrouter APIs and tool calling, you can make a program with a collection of command-line-invokable subcommands to do things like dump a function or record some documentation that just works over stdio. Then you write a text file with some instructions about how to run that program and hand the whole thing over to an existing coding agent.

The main downside of this approach is that the agent can do whatever it likes, like grepping around your hard drive for information it seeks. In theory you can write strongly worded messages in all caps in the instructions or sandbox the process but neither of these feel like you're really controlling anything. The advantage to the free reign of an agent is that you can also ask the agent for things outside of the tasks you enumerated ahead of time, such as "write a document that summarizes the public interface of this module". In our modern times of possibly heavily subsidized LLM subscriptions, I think this approach also is ultimately cheaper.

To summarize, making a program that hooks up to LLM providers with API finessing around tool calls ends up effectively the same as making a boring CLI program that an LLM can itself shell out to. The two forms end up similar in many ways, where tool calls become subcommands, and I still used JSON schema in the CLI to describe the form of structured data coming in and out.

The result is that you hand a binary over to this machine. It grinds for a while and produces a database of what the program does. Here is some output, a dump of the web viewer for the sound driver BLASTER.DRV. (Slop warning: to be clear, the assembly and cross referencing from there is my own code, but the comments and function names etc. come from the LLM.) From that, I had the agent generate a document just describing the driver interface.

With that document in hand, I was able to write my own trivial implementation of just this driver interface and Theseus lets me substitute in those Rust functions in place of the code from the actual binary.

I'm still not sure if there's any point to writing software anymore or if next month's model will be able to write better blog posts than me anyway. But at least for my initial problem of making this DOS program talk it feels like maybe this sort of man-machine hybrid work is still interesting enough to post about. And with LLM calls as another tool in my toolbox I've gone on to use them in other projects, a post for another day!