Rising electricity cost made me reconsider the economy of running a homelab this summer. Combined with a short period of sweltering heat, I was motivated to find a different solution to hosting my own services than to do it from home. Worst case was when I had a NAS sitting on the floor with a table fan strapped precariously to a chair in order to keep the disks’ temperature within spec. My choice was to either upgrade the housing and hardware of my lab substantially, to replace it with something rented, or to keep on messing with 3rd-rate hardware to keep up some semblance of stability. After some quick calculations I went for option 2: rent a box with Hetzner and migrate. To my surprise Hetzner had stopped supporting FreeBSD but I had no intention of migrating to anything else. Here’s how I dealt with this situation.

FreeBSD had been a bit of a second-class citizen with Hetzner before, but at least their PXE-booted “rescue environment” included a FreeBSD option from which an installer could be easily started and completed. Apparently this has been gone for more than a year now: very annoying, and there’ also no prepackaged installation ISO for FreeBSD anymore. So how to proceed then?

There are ways to run bsdinstall from Linux, to load FreeBSD from a RAM-disk and perform what amounts to a “depenguination” from there. I chose not to go such routes and ordered a 16GB USB drive as an add-on with the SX64 server I chose to rent. My main reason to go this route is ease of recovery in the case of total hardware failure. The OS would live on the USB drive and the data would go onto a RAIDZ array made of the four 16TB hard drives.

Installing was an interesting roundtrip. I used the Linux rescue environment to fetch a FreeBSD memstick image that’s meant for use with a USB stick. Using the remote KVM console, I was able to adjust the machine’s boot sequence. After writing the installer image to one of the data drives, I rebooted from that and it dutifully started the FreeBSD 13.2 installer.

The installation itself worked like it does regularly. I chose the USB drive as the target drive, finished up and readjusted the boot drive to be the USB stick. A reboot later and the machine started from USB. Creation of the ZFS array was quick and easy.

Anticipating hardware failure

Hetzner’s dedicated servers are physical hardware that you rent. That requires the customer to plan for the contingency of catastrophic hardware failure. Hetzner explicitly does NOT keep any backups of data that lives on your server. I chose to keep my ancient NAS box at home around for that purpose and to fit that with the necessary terabytes to keep an offsite backup. The unlimited traffic combined with the fiber link at home make this a viable proposition for me.

Considering the fact that Hetzner apparently deprecated FreeBSD from their entire offering, I’m not counting on any support from them other than a wholesale disk replacement in the case something goes down very badly. Disk replacement is a scenario that carries two risks for me.

My primary concern is disposal of the old disk that carries much of my personal data. Therefore no data shall touch the platters unless it’s strongly encrypted. This can be accomplished in roughly two ways on FreeBSD: GELI encrypt the block devices and then put RAIDZ on top of that, or use native ZFS encryption. The GELI encryption route has the main advantage of really encrypting every single block of information that touches the platters. Its disadvantage is that it sits between ZFS and the hardware, which is a scenario that ZFS was never intended for.

So I chose to use ZFS native encryption on datasets, accepting the fact that the existence of the datasets and their names could be gleaned from the disks. By keeping those non-confidential I created an acceptable situation for myself: my files would be safe from prying eyes should Hetzner choose to improperly dispose of the disk or simply plonk it into one of its bargain basement auction servers.

The second risk is in data recovery. Should Hetzner replace a disk, that would trigger a resilvering of the RAIDZ structure. By keeping this as simple as possible with a single zpool called storage, I could easily recover the whole of my data area with a single command without having to worry about the OS that would sit on the exact same disks with no realistic way to boot into an alternative environment.

Recovery procedure

Sure there’s an ISO mount option in the remote KVM option, which I tried but it felt incredibly fragile and slow. I’d rather not rely on that when my server is in an undefined state of disaster. Should the whole thing go down beyond any help, I’d get the whole machine replaced, redo the installation procedure and recover all of my data from my attic back onto the new box. Such a recovery would take weeks to fully complete and it’d be rather crummy. That’s why I’m thankful to Hetzner for including a free limited version of their Storage Box product with this server: I get 100GB of external backup storage for free, which is a brilliant place to stuff backups of the jails I use for mail, web and databases. Recovering those works at full Gigabit speed from within Hetzner’s own network, which is 5 times faster than the fiber uplink I have at home and which would go across the public Internet from the Netherlands to Helsinki.

At a price of just barely over EUR 100 per month all-in, I now have a virtual homelab that has 64GB of ECC RAM, 64TB of raw HDD storage, a USB drive to boot from and a pretty decent unmetered 1Gbps. Internet uplink. Thanks for a great deal Hetzner, although I’d really appreciate it if you were to bring back the FreeBSD rescue environment.