Moving k8s back to my homelab

Sorry Hetzner, even you are too expensive to host my homelab in the cloud these days. Not your fault at all, given the AI crunch on hardware availability. Combined with the fact that I have rock-solid 1Gbps fiber internet at home, solar panels on my roof and some nice mini PC’s available to me I’m moving my cluster back to my home office.

Don’t get me wrong. The Hetzner cloud is an amazing platform for hosting a k8s cluster in production if you’re on a budget. Bulk storage, however, is my main bottleneck. I tried two ways around this, neither of which works well unfortunately.

NFS over WireGuard

I have a nice big NAS at home that’ll meet my Nextcloud storage requirements just fine using RAIDZ2. It speaks NFS, and I can transport NFS over the already existing WireGuard VPN as well. The unfortunate reality: my NAS is in the Netherlands while my Hetzner boxes sit in a DC in Falkenstein Germany. Latency is roughly 30ms. between those locations, and that makes the use of NFS unbearably slow for Nextcloud.

Hetzner StorageBox over CIFS

The other option would be Hetzner’s own StorageBox bulk storage proposition. In reality this is intended as backup storage. It’s similar to my NAS in that it stuffs my bits into a RAIDZ2 array on a dedicated box, and exposes them back to me over a limited number of protocols. The only one usable to me wihtin k8s is CIFS.

CIFS, being the monstrosity that it is, does not like latency at all. Unfortunately, even with the StorageBox located within Hetzner’s own network, Nextcloud is even more sluggish than when I was testing with NFS. It works, for now, but it’s not something I want to inflict on myself for much longer.

Hetzner obviously does offer alternatives: cloud volumes and object storage. Neither of these is available at a price point that I can handle right now. They are plenty fast for the smaller workloads that I am already hosting. A small test with Nextcloud on Hetzner cloud volumes turned out to also function just fine. Unfortunately I have a growing dataset of 2TB to lug around so back home it is.

The plan

I’m going to set up a 3-node Talos Linux cluster based around a triplet of Intel N200-based mini-PC’s that should more or less match the Hetzner CX43 low-budget servers. The 512GB SSD’s give me almost double the local storage capacity and my NAS sits on the same VLAN as the nodes. That should kill all forms of latency-induced slowness. It also allows me to choose whichever storage protocol I like for my bulk storage, up to and including iSCSI or object storage using Garage.

The nodes themselves are dual-homed so I’m looking to set them up with a dedicated physical network for Longhorn storage replication traffic. Not sure how to go about that yet, but this is a hobby for a reason of course.

To be continued!