Distributing binaries with Common Lisp and foreign libraries

The ability to create a new binary executable by simply “dumping” to disk the code in a running Common Lisp executable is one of the features that makes Common Lisp well suited for rapid development.

That ease of development breaks down when using foreign libraries (usually written in C/C++): in the most popular Common Lisp implementations, the Foreign Function Interface loads foreign libraries using dlopen() and stores their paths in the Lisp image, so that the resulting binary can start from their previous state. Unfortunately, that makes the resulting binaries hardly portable because on the target environments (e.g. users’ laptops), even though there might be an ABI-compatible version of those foreign libraries, their paths (and even the names) might be different.

A common solution to this problem is to distribute an application as a bundle (an archive) of the main binary and its foreign library dependencies, wrapped by a shell script that sets LD_LIBRARY_PATH and other environment variables. This works, but it requires very good knowledge of the runtime linker, and also removes the convenience of distributing an application as a single file.

This is very similar to the developer experience in other languages that have C FFIs like Python, Ruby or Java. Their developers have either gotten used to deploying applications-as-tarballs or just require end users to resort to some package manager to use an application: in other words they don’t even bother to compile, package and distribute an application but require all end users to download the developer toolchain and compile the application themselves. Perhaps we can do better than that.

A solution

I’d like to outline a solution and a prototype that works with my implemention of choice, SBCL.

Statically link foreign libraries into the runtime

One way to cur the Gordian know is to entirely side-step the problem of how to find and open a foreign library on all possible target environments, and simply link said libraries statically into the Common Lisp runtime.

This has the advantage of being able to ship an application without requiring end users to install or update foreign libraries (which in some cases might not even be technically or legally possible), but there’s one major disadvantage: the application developer is now in charge of keeping up-to-date with the upstream releases of bundled foreign libraries. That might not seem like a big problem, but foreign libraries tend to have security vulnerabilities that require prompts updates. Fortunately, that can be automated (more on it later).

Make CFFI work seamlessly with static and dynamic libraries

Once one has a runtime that contains a certain foreign library, how does one work with it ? Most Common Lisp FFI wrappers are unaware of static linking and load their foreign library unconditionally.

Luckily, CFFI already implements a handy solution: the define-foreign-library macro has a :canary keyword argument that is meant to be a foreign symbol unique to that library (usually a function). A subsequent call to load-foreign-library or use-foreign-library will first check if that symbol is already present in the runtime, and if so it will simply mark the foreign library as loaded, skipping the dlopen().

1(define-foreign-library
2    (libfoo :canary "foo_init")
3  (t (:default "libfoo")))
4(load-foreign-library 'libfoo)

One can see a real-life example in iolib/src/syscalls/ffi-functions-unix.lisp.

The prototype

I’ve implemented a prototype, available at github.com/sionescu/sbcl-goodies. That repository will automatically release SBCL binaries for Linux/x86_64 that statically link libfixposix and OpenSSL. It already has one release: 2.3.1+r00. I intend to maintain this repository and if there is interest, potentially extend it.

Modifications

• src/runtime/sbcl is statically linked to libcrypto, libssl and libfixposix
• in the SBCL core, a new keyword was added to *features*: :CL+SSL-FOREIGN-LIBS-ALREADY-LOADED. That feature is used by cl-plus-ssl/src/ffi.lisp
• CL:LISP-IMPLEMENTATION-VERSION returns a string containing the revision, e.g. "2.3.1+r00"
• the subdirectory third_party/include contains the headers of libfixposix

Implementation details

Build environment

For backwards compatibility, the compilation is done on Ubuntu 20.04 LTS. Compiling a binary on a newer distribution, then running it on an older one often doesn’t work because GLIBC uses symbol versioning and the symptom is an error like this:

# sbcl
 /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found

OpenSSL

The static libraries are from the official OpenSSL dev package from Ubuntu (with security updates).

LibFixPOSIX

The latest Github release.

Releases

The release process publishes both a source and a binary distribution tarball of SBCL. The naming scheme adds a two-digit revision that is increased every time new releases of the “goodies” occur after SBCL upstream makes a new release. When SBCL is released, the revision is reset to “00”.

Example:

• sbcl-2.3.1+r00-x86-64-linux-binary.tar.bz2
• sbcl-2.3.1+r00-source.tar.bz2

Build scripts

The subdirectory scripts contains a series of shell scripts that are invoked by the Github workflows and serve to prepare the build environment and perform the actual compilation.

The input version of SBCL, libfixposix and OpenSSL are kept in build.env to make it easy to change them programmatically.

Build & release workflow

The Github workflow build.yaml will build SBCL (on a pull request), or build SBCL then upload the source and binary distributions to Github release when a pull request is merged.

The dependency check workflow

The workflow update-deps.yaml will periodically check for new updates of SBCL or the linked libraries. If an update is found, it edits build.env and creates a pull request that will check that the set of dependencies can be compiled and linked without errors.