neugierig.org: Tech Notes

This post is part of a series on retrowin32.

In retrowin32 I sometimes want to make my own win32 programs that I can run either natively or through the emulator to compare behaviors.

The first time I wrote about this I tried Zig, found it promising, but gave up and used C. The second time I tried Zig again and had a good time but also wrote about downsides. Recently I tried a third time, this time trying to build an app that actually brought up a window, and again got frustrated by Zig's (lack of) documentation and language churn.

The bulk of retrowin32 is written in Rust, so an obvious question is why I haven't used Rust for this purpose. The answer is that I hadn't figured out how to make it work... until recently!

Naively you might expect all the cross compilation pieces to be in place already and wonder why this was even hard. The answer is they kind of are, but the details are finicky, in part due to my requirements. This post walks through the details.

To start, Rust itself supports cross compilation via the --target flag. For win32 there are two targets: i686-pc-windows-gnu and i686-pc-windows-msvc. The differing last component here specifies the ABI but for our purposes that mostly means which toolchain you end up using for all the pieces after compilation (linker, libraries).

The -gnu target uses mingw, which is the world of gcc. On a Mac it's an easy install with brew. Using some of the tricks discussed here I was able to produce a binary, but due to this unconventional (for Windows) toolchain it ends being fairly different from the binaries retrowin32 is intending to emulate. I could surely make them work but it's a bit of a yak shave.

The -msvc target instead relies on the Visual Studio toolchain, producing binaries much closer to what we want. This is our goal, but at this point there are a number of hurdles and churn between Rust versions that meant my previous attempts at getting this working failed.

I don't have all the details straight but here are hopefully enough keywords for future searchers to find this post.

Linker: Running the MSVC toolchain includes running link.exe to link. This of course thwarts cross compiling from a Mac, but LLVM has its own linker that Rust can use. It's all confusing and churning (see e.g. this bug) but I found creating a .cargo/config with linker = "rust-lld" worked.

Libraries: Windows programs make system calls into system libraries like kernel32.dll. To link against these there is a corresponding .lib file that the linker uses, and without them you get errors like

rust-lld: error: could not open 'kernel32.lib': No such file or directory

Getting these files is kind of a mess. Someone wrote a nice installer that attempts to download these from Microsoft (note that if you want to run this on Mac you have to pass a weird flag), but I eventually twiddled enough that I didn't need them.

(Update: after writing this post, I dug into why the windows-sys crate can build code without these. The answer is that (1) it actually bundles its own .lib files via some magic crates, and there is more documentation about why; and also (2) there is some work on a "raw-dylib" feature in Rust to avoid needing .lib files at all. Unfortunately with a small amount of poking I wasn't able to convince Rust to use the .lib files from windows-targets to satisfy the above failing link.)

no_std: I believe the reason you need all of the above .lib files is because Rust's standard library uses them. For my emulator purposes I'd rather have less code in my executables in general — I'm often tracing through the assembly — so next we descend into disabling Rust's standard library with a

#![no_std]

at the top of the file.

eh_personality: With that you might see an error about language item required, but not found: eh_personality. There is Rust syntax for defining this (if you look it up) but it requires unstable Rust. Instead you can make panics abort rather than unwind via -C panic=abort.

panic_handler: You need to provide an implementation of what to do on panics. Here's some code with some comments:

fn print(buf: &[u8]) {
    unsafe {
        let stdout = GetStdHandle(STD_OUTPUT_HANDLE);
        WriteFile(
            stdout,
            buf.as_ptr(),
            buf.len() as u32,
            core::ptr::null_mut(),
            core::ptr::null_mut(),
        );
    }
}

// rust-analyzer gets confused about this function, so we hide it from rust-analyzer
// following https://github.com/phil-opp/blog_os/issues/1022
#[cfg(not(test))]
#[panic_handler]
unsafe fn handle_panic(_: &core::panic::PanicInfo) -> ! {
    print(b"panicked");
    windows_sys::Win32::System::Threading::ExitProcess(1);
}

The Windows API: In the above you see a call to ExitProcess via the windows-sys crate, which is autogenerated to cover the whole Windows API and is compatible with no-std.

main: As best I understand it, the normal Windows startup call stack is that the executable entry point goes to the C runtime mainCRTStartup/WinMainCRTStartup which then invokes main or WinMain. Rust makes it always mainCRTStartup. So the entry point is:

#![no_main]
...
#[no_mangle]
pub unsafe extern "C" fn mainCRTStartup() {

Compiler intrinsics: And, for my final trick! Once you start writing some code you might encounter errors like rust-lld: error: undefined symbol: _memcmp. These are some functions that the compiler emits references to with the expectation that some other layer provides them. There's some crate that is supposed to help with this but I couldn't get it to work. I am not really clear on why it works (possibly inlining?) but I found that building with --release was enough for my program to work.

(Update: after writing this post and writing more code, I eventually still ran into failures due to these missing symbols. I ended up defining my own implementations manually.)

Putting it all together: Finally, here's a fully worked example program (along with the .cargo/config) that brings up a window. The resulting exe is 3kb.