Distributing binaries with Common Lisp and foreign libraries
The ability to create a new binary executable by simply “dumping” to disk the code in a running Common Lisp executable is one of the features that makes Common Lisp well suited for rapid development.
That ease of development breaks down when using foreign libraries (usually written in C/C++): in the most popular Common Lisp implementations, the Foreign Function Interface loads foreign libraries using dlopen() and stores their paths in the Lisp image, so that the resulting binary can start from their previous state. Unfortunately, that makes the resulting binaries hardly portable because on the target environments (e.g. users’ laptops), even though there might be an ABI-compatible version of those foreign libraries, their paths (and even the names) might be different.
A common solution to this problem is to distribute an application as a
bundle (an archive) of the main binary and its foreign library
dependencies, wrapped by a shell script that sets
and other environment variables.
This works, but it requires very good
knowledge of the runtime linker, and also removes the convenience of
distributing an application as a single file.
This is very similar to the developer experience in other languages that have C FFIs like Python, Ruby or Java. Their developers have either gotten used to deploying applications-as-tarballs or just require end users to resort to some package manager to use an application: in other words they don’t even bother to compile, package and distribute an application but require all end users to download the developer toolchain and compile the application themselves. Perhaps we can do better than that.
I’d like to outline a solution and a prototype that works with my implemention of choice, SBCL.
Statically link foreign libraries into the runtime
One way to cur the Gordian know is to entirely side-step the problem of how to find and open a foreign library on all possible target environments, and simply link said libraries statically into the Common Lisp runtime.
This has the advantage of being able to ship an application without requiring end users to install or update foreign libraries (which in some cases might not even be technically or legally possible), but there’s one major disadvantage: the application developer is now in charge of keeping up-to-date with the upstream releases of bundled foreign libraries. That might not seem like a big problem, but foreign libraries tend to have security vulnerabilities that require prompts updates. Fortunately, that can be automated (more on it later).
Make CFFI work seamlessly with static and dynamic libraries
Once one has a runtime that contains a certain foreign library, how does one work with it ? Most Common Lisp FFI wrappers are unaware of static linking and load their foreign library unconditionally.
Luckily, CFFI already implements a handy solution: the
define-foreign-library macro has a
:canary keyword argument that
is meant to be a foreign symbol unique to that library (usually a
function). A subsequent call to
use-foreign-library will first check if that symbol is already
present in the runtime, and if so it will simply mark the foreign
library as loaded, skipping the dlopen().
1(define-foreign-library 2 (libfoo :canary "foo_init") 3 (t (:default "libfoo"))) 4(load-foreign-library 'libfoo)
One can see a real-life example in iolib/src/syscalls/ffi-functions-unix.lisp.
I’ve implemented a prototype, available at github.com/sionescu/sbcl-goodies. That repository will automatically release SBCL binaries for Linux/x86_64 that statically link libfixposix and OpenSSL. It already has one release: 2.3.1+r00. I intend to maintain this repository and if there is interest, potentially extend it.
src/runtime/sbclis statically linked to
- in the SBCL core, a new keyword was added to
:CL+SSL-FOREIGN-LIBS-ALREADY-LOADED. That feature is used by cl-plus-ssl/src/ffi.lisp
CL:LISP-IMPLEMENTATION-VERSIONreturns a string containing the revision, e.g.
- the subdirectory
third_party/includecontains the headers of libfixposix
For backwards compatibility, the compilation is done on Ubuntu 20.04 LTS. Compiling a binary on a newer distribution, then running it on an older one often doesn’t work because GLIBC uses symbol versioning and the symptom is an error like this:
# sbcl /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found
The static libraries are from the official OpenSSL dev package from Ubuntu (with security updates).
The latest Github release.
The release process publishes both a source and a binary distribution tarball of SBCL. The naming scheme adds a two-digit revision that is increased every time new releases of the “goodies” occur after SBCL upstream makes a new release. When SBCL is released, the revision is reset to “00”.
The subdirectory scripts contains a series of shell scripts that are invoked by the Github workflows and serve to prepare the build environment and perform the actual compilation.
The input version of SBCL, libfixposix and OpenSSL are kept in build.env to make it easy to change them programmatically.
Build & release workflow
The Github workflow build.yaml will build SBCL (on a pull request), or build SBCL then upload the source and binary distributions to Github release when a pull request is merged.
The dependency check workflow
The workflow update-deps.yaml will periodically check for new updates of SBCL or the linked libraries. If an update is found, it edits build.env and creates a pull request that will check that the set of dependencies can be compiled and linked without errors.