OCaml cross-compilation: an experiment

I've recently become interested in cross-compiling OCaml code for other platforms. It's always been a bit tricky with OCaml, and the existing ecosystem of cross-compilation is patchy and complex, and usually engineered to specific targets, such as mobile development¹, building software for windows², or embedded system use cases³. Each effort seems to be different, with everyone attempting to solve the same problems.

My own investigation has also been looking at how to simplify it for end-users, and how to make it more generic and reproducible to reduce the duplicated effort that currently seems to be necessary.

There are various reasons why you may need cross-compilation:

• compile software for mobile or embedded systems (where compiling on the device is too slow or impossible)
• distribute software for other operating systems without needing to run the OS on your machine or in a VM
• run software in a cloud environment that uses a different OS than your developer or CI/CD machine

Why not just use docker?

Docker is great for running you code in a lightweight and easily reproducible container, used for everything building production systems, ensuring consistent development environments (e.g. via dev containers), and of course, simplifying the use of cross-compilation.

My interest in cross-compilation comes from the serverless world, where it is used to produce binaries for running your application in a serverless runtime container.

Serverless applications rely on a base image, which provides the operating system and runtime environment for your chosen language. Interpreted and byte-code languages (Python, JavaScript, .NET, Java, etc) can just put their runtime in the base image and avoid compilation altogether, but compiled languages that target native code like OCaml or Rust require cross-compilation to build for your target serverless base image.⁴.

Docker handles this requirement quite nicely, but it comes with a significant overhead in terms of performance. This overhead is irrelevant in CI/CD environments, but the typical serverless developer relies on the use of ephemeral environments to speed up iterative development.⁵. In this workflow, you want to continually make changes and deploy them to a cloud environment to validate them, so every ounce of performance in compile-deploy cycle matters.⁶

What do you need to cross-compile OCaml?

There are three main components to a cross-compilation "system" for OCaml:

1. C cross-compiler and tools for the target OS and architucture

   The classic way of doing this is to build a cross-compiling gcc+binutils+glibc for the target environment which is notoriously difficult, but systems like mxe can make this simpler.

2. OCaml cross-compiler

   The OCaml compiler is historically difficult to set up as a cross-compiler, but this has become easier with recent changes in the mainline (also backported to 5.3).

3. Packages to cross-compile and install

   This typically an opam repository, but with the same packages cloned, renamed, and its build scripts modified for the cross-compiler (I describe this is in more detail below)

There are various ways of achieving each of the above in combination, as well as whole of system examples e.g. there are nix-based repositories for OCaml with a number of pre-compiled packages.

I've found none of these problems easy to tackle, but by treating them as separate problems, I've been able to address them in more generic and (hopefully) more reproducible ways.

zig: a C compiler

zig is an interesting language in its own right, aiming to be the ideal systems programming language occupying a niche somewhere between Rust (correctness) and C (simplicity and ease at low level programming).

What's also interesting is its laser-like focus on integrating well with C. zig uses LLVM as a back-end, and can easily compile C code both in zig build files as well as through a built-in C frontend the so-called zig cc toolchain.

Even better, zig cc can cross-compile C code to a variety of CPUs and OSs with its -target option, all without the hassle of setting up a cross-compile environment: just unzip zig and run it, and it will download the required headers and libraries as needed.⁷

Building an OCaml cross-compiler

In order to target Amazon Linux on AWS, I've put together an overlay repository for opam with OCaml cross-compiler packages for Amazon Linux 2023 on x86_64 (ocaml-x86_64_al2023) and aarch64 (ocaml-aarch64_al2023) architectures. They use zig under the hood as their C cross-compiler, only requiring that you put the zig binary in your PATH to install them. ⁸

These packages borrow heavily from the work in OCaml's nix overlays and opam-cross-windows to bootstrap OCaml 5.3 and build it as a cross-compiler.

These packages install themselves into a separate subdirectory under the opam switch root (e.g. $OPAM_SWITCH_PREFIX/x86_64_al2023-sysroot) so they don't override the system compiler. They also modify the findlib.conf file to register themselves as a "toolchain" with ocamlfind - this lets you build code with them using the -toolchain parameter to ocamlfind, or the -x parameter to dune.

Building packages

An OCaml cross-compiler on its own isn't enough: we also need to be able to build software on top of that.

For our own code, it's enough to pass -x <toolchain> (e.g. -x aarch64_al2023) when building our code with dune. However, if we rely on third-party packages from opam, we can't just use the packages directly from opam because they will be compiled for our host machine, not our cross-compile target.

Cross-compile overlay repositories like opam-cross-windows or opam-cross-android are constructed by copying the entries of regular opam packages, add the toolchain name as a suffix, and rewriting the build script to pass the right flags to the build tool (in most cases, dune or topkg) in order to cross-compile them.⁹

Although this is relatively straightforward, it's somewhat tedious to have to make a copy of every package, and its transitive dependencies, in order to build them for your target, especially when most of the time the changes follow a regular pattern.

For example, lets say we want to cross-compile the yojson package (a JSON parsing library) for our aarch64_al2023 target. We would need to:

1. create a new package called yojson-aarch64_al2023 in my overlay repository.
2. update its build script to call dune as dune -p yojson -x aarch64_al2023 @install (if it's a dune based package; for topkg its something else)
3. locate each of its runtime dependencies (in this case, its just the seq package) and repeat steps 1 & 2.
4. For each package in step 4, repeat step 3 to find their dependencies (i.e. process transitive dependencies until we get everything) and update them as per 1 & 2.

In order to simplify this, I've put together a small tool called packman that can take a package name, resolve its transitive dependencies, and rewrite all of them to be used with a particular cross-compiler toolchain.

This tool is not enough for cases where packages cannot be mapped automatically - some dependencies like seq are special and need custom handling - others like conf-* based packages won't work as they only validate the host machine, and need to be replaced with something that compiles or stubs our a C library target (e.g. conf-libssl needs to present a cross-compiled version of openssl).

For those cases, I've introduced the concept of a template repository, which contains hand-crafted cross template package definitions, with placeholders for the toolchain name that can be substituted when it is processed.

packman works by first identifying all the runtime dependencies. {build}, {dev}/{with-dev-setup}, {with-test} and {with-doc} dependencies are not processed, as they only are needed on the host machine. (Known build-time packages like dune, ocamlfind, topkg, etc. are also ignored as they are often not marked with the {build} filter).

It will then process each dependency and their transitive dependencies, first checking if a dependency has an entry in the template repository, before attempting to rewrite the package definition from the source opam repository (usually the main opam repository).

The invocation looks something like this:

$ packman map-packages \
  <source_repository> \
  <overlay_repository> \
  <template_repository_path> \
  <destination_repository_path> \
  <cross_name>

where:

• <source_repository> is the name of the source opam respository (usually the main repository), used to resolve standard package definitions for rewriting
• <overlay_repository> is your overlay repository containing ocaml-<cross_name> and other base.*-<cross_name> packages for your cross-compiler that have been hand-crafted (they are used to ensure package resolution works)
• <template_repository_path> is the template repository (used as described above for packages that cannot be automatically rewritten)
• <destination_repository_path> is the directory where to write the new cross-compiler packages
• <cross_name> is the name of the cross-compiler toolchain (e.g. x86_64_al2023), used to suffix the new packages in the destination repository.

This tool is still very much in the development phases, but with I've been able to cross-compile some packages of interest for AL2023 for use with AWS Lambda, including eio for concurrency and piaf for a HTTP library (with my own smaws AWS SDK not far away).

Its development has meant I can focus more on the exceptional cases of packages that don't work automatically (e.g. uring which provides a custom copy of liburing and needs to be patched to work with zig) or don't use a known build system.

Looking forward

The next task is to further refine packman and test it with a greater variety of packages. I want to be able setup a CI/CD which can validate the generation and cross-compilation of packages before packaging it for general availability.

Another avenue to explore is whether I can avoid the whole mess of rewriting opam packages with packman altogether. If we are using dune as our primary build tool with new experimental package management support, there may be an avenue to avoid requiring opam.

If I understand how it works correctly, it ensures that all packages are converted to use dune (offering an overlay repository for those that don't) so they can be all compiled in the same workspace. I should be able to leverage this by simply passing in the right -x <toolchain> parameter. The only problem remaining would be "template" packages that are only used in the case of cross-compiling (and not for the standard default toolchain), something I don't think it currently can support.