Area536 :: Rebooted

Jun 8, 2024 - 7 minute read - FreeBSD

Integrating my own package mirror into FreeBSD

After failing to install a number of packages from upstream binaries I decided to host my own FreeBSD packages mirror using Poudriere. I must say: the Poudriere software is the most elegant solution I encountered to allow users of an operating system to build and host their own package repositories. Actually integrating the resulting packages into a workable infrastructure is not entirely obvious. Here’s how I did it and why, which may be of use to others. Mind you: I use FreeBSD on a laptop as my daily driver, but this strategy should work just as well for server use cases. I’m assuming you somewhat know your way around administering FreeBSD systems and have a specific need to fiddle with the package mechanism.

In order to understand what I’m doing, you first need a good understanding of some typical FreeBSD terminology that may carry a different meaning for you if you’re coming from another operating system like Linux. Let’s begin with the operating system itself.

FreeBSD branches

You can read the FreeBSD release engineering website to find out the finer details of the release building process. For our purposes it suffices that you know the difference between FreeBSD ‘CURRENT’, ‘STABLE’, ‘RELEASE’ and the significance of version numbers of the operating system.

Simply put: both ‘CURRENT’ and ‘STABLE’ are development versions. Never use them for anything other than developing the operating system itself. If you don’t know what that means, then just ignore these versions and their source code branches altogether until you do. They won’t bring you anything useful until you do know what you’re looking at them for.

What remains is the ‘RELEASE’ version of the operating system. This, confusingly, lives in the Git version management system as a numbered ‘releng’ branch. For FreeBSD 14.0-RELEASE the corresponding Git branch is releng/14.0. This is important because a ‘RELEASE’ may receive security and errata patches which will be published on the respective ‘releng’ branch. If you track a source code branch and prefer to update your systems from source, you follow the ‘releng’ branch for your running release.

Ports and packages

FreeBSD releases around 35,000 optional third-party packages in the form of what it calls the ‘ports collection’. This collection consists of a huge pile of Makefiles and supporting infrastructure that allows you to easily and predictably build third-party software from its own source code to work specifically on FreeBSD. This is useful because FreeBSD supports many different hardware platforms apart from the popular x86-64. Having a well-understood framework to build from source code greatly facilitates the process of making software available on all kinds of different hardware.

The ports collection itself also lives in a Git repository with branches of its own. At its very tip you find the ‘main’ branch which always holds the bleeding edge of development. Further down the line you’ll find branches that are cut from ‘main’ in quarterly intervals. At the time of this writing we’re at 2024Q2, or the second quarter of 2024.

The FreeBSD project operates build infrastructure that generates binary pakcages for most of the software in the ports collection at regular intervals. It makes these packages available to you through the pkg command. These packages come in two sets as well: latest and quarterly. The latest packages get built from the main branch of the ports collection, while the quarterly packages come from the ports branch that corresponds to the current quarter.

One thing that is relevant to know about FreeBSD versions is that the release engineering team makes an effort to keep the ABI stable within a major version. So every package built for 14.x works on any minor release within major version 14. A notable exception to this rule are packages that contain kernel modules, but I’ll get to those later. This fact is the reason why there are only two sets of upstream packages for every major release version, independent of minor versions.

Rolling your own packages

Installing Poudriere is not immediately in scope for this blog. I’m assuming you can follow the manual and get build jails for the currently supported major versions up. The main question now is: which versions should you create jails for, and which branches of the ports tree do you build? And when you’re done, how do you set up your consumers to actually use your customized packages while falling back to the upstream FreeBSD packages for the others without causing dependency hell?

Ending up in a world of pain repeatedly made me adopt the policy of only ever tracking ‘quarterly’ on the upstream package repository, augmented with my own builds on the Git branch for the same quarter. Latest moves faster than I can rebuild packages on my infrastructure. Maybe you can do better because you have more hardware, but I strongly advise against trying it.

This implies that I should build packages for the current and the previous major release versions. At the time of this writing that’s 13.x and 14.x. So I have build jails for both of them in their most current minor release, but these are throwaway setups. When a new minor comes out, I just create a new one for the new version and discard the old one once the last consumer is done upgrading.

Setting up your consumers

The huge advantage of integrating your own packages with FreeBSD upstream is in the fact that FreeBSD builds everything for you and you only have to deal with your local customizations yourself. In my case that comes down to hosting my own copy of the C64 emulator VICE, which upstream won’t package due to licensing constraints. Everything else should simply come from upstream, except sometimes for upstream breakage like sometimes happens with big projects like Visual Studio Code.

Ideally that would mean that the pkg command should first check whether a package is in my personal repo and install that, before looking upstream in the FreeBSD project’s repo. It’s surprisingly simple to do so, but it may still get a bit tricky to configure it correctly (ie. without touching /etc/pkg/FreeBSD.conf). Take these steps:

  1. Create /usr/local/etc/pkg/repos.
  2. Create FreeBSD.conf in there.
  3. Create LocalRepo.conf in there as well.

The contents of /usr/local/etc/pkg/repos/FreeBSD.conf should be:

FreeBSD: {
    url: "pkg_https://pkg.FreeBSD.org/${ABI}/quarterly",
    mirror_type: "srv",
    signature_type: "fingerprints"
    fingerprints: "/usr/share/keys/pkg",
    enabled: yes,
    priority: 10
}

The important bit here is the priority field!

The contents of /usr/local/etc/pkg/LocalRepo.conf should be:

LocalRepo: {
    url: "https://whatever.your.mirror.url/path/2024Q2/",
    mirror_type: "http",
    signature_type: "pubkey",
    pubkey: "/usr/local/etc/pkg/localrepo.key",
    enabled: yes,
    priority: 20
}

The significant parts of this one are priority, signature_type and pubkey. Poudriere permits us to easily sign the finished repository with a cryptographic key. If you’re depending on these packages for anything relevant, you should implement this. Simply hosting your files behind HTTPS is not enough to detect tampering with the contents of the repository. Creating these keys is up to you. If you don’t have them, you can simply omit the fields from the repository configuration.

The significance of the priority on the FreeBSD.conf file is to make it supersede whatever you have in /etc/pkg/FreeBSD.conf, just to be certain. The slightly higher priority on LocalRepo.conf is there in turn to supersed the FreeBSD upstream repository. It is this setting that forces pkg to pull packages from our local repo before looking upstream.

Because I have a lot of consumers I use Ansible to distribute this configuration and update the URL for LocalRepo in time for the rollover to the next quarter.

Kernel modules

Some ports like drm-kmod contain kernel modules. Kernel modules plug into the FreeBSD kernel and this is where the stable ABI promise for major OS versions ceases to apply. It is also the reason why you need to match the exact minor version of your build environment with the environment you’ll be deploying into. If you are not building any ports that contain kernel modules, none of this applies. You can just use a single major version to build all your packages and things will be fine. If, like me, you run FreeBSD on desktops you’re bound to run into drm-kmod and its siblings from time to time so do take note.