How My Home Lab Kubernetes Cluster Is Built: k3s on Proxmox

My home lab runs a small Kubernetes cluster for the services I self-host. It’s three virtual machines on Proxmox running k3s. Terraform builds the VMs and Ansible installs and configures the cluster. This post covers how the nodes are set up. The apps that run on the cluster are a topic for another post.

The Big Picture

• Hypervisor: a two-node Proxmox cluster (pve1 and pve2). A small QDevice VM on a separate box gives the cluster a third vote.
• Kubernetes: k3s with one server node (k8s-1) and two agent nodes (k8s-2 and k8s-3).
• Provisioning: Terraform with the bpg/proxmox provider creates the VMs.
• Configuration: an Ansible playbook installs k3s and the cluster add-ons.
• Upkeep: OS reboots and k3s upgrades run on their own, one node at a time, during a Sunday-morning window.

The Nodes

Node	Role	Preferred host	vCPU / RAM / Disk	IP (example)
k8s-1	server (control plane)	pve2	4 / 8 GB / 60 GB	10.20.0.11
k8s-2	agent	pve1	4 / 8 GB / 60 GB	10.20.0.12
k8s-3	agent	pve1	4 / 8 GB / 60 GB	10.20.0.13

The control plane runs on a different physical host from the two agents. If one Proxmox box goes down, part of the cluster keeps running while HA brings the rest back.

Building the VMs with Terraform

Every VM starts from the official Ubuntu 24.04 (Noble) cloud image. Proxmox storage isn’t shared between my hosts, so Terraform downloads the image once to each host. The VMs are defined in a single map, and that map is the only thing I edit to add or resize a node:

nodes = {
  "k8s-1" = { vm_id = 240, pve_node = "pve2", ip = "10.20.0.11", cores = 4, memory_mb = 8192, disk_gb = 60 }
  "k8s-2" = { vm_id = 241, pve_node = "pve1", ip = "10.20.0.12", cores = 4, memory_mb = 8192, disk_gb = 60 }
  "k8s-3" = { vm_id = 242, pve_node = "pve1", ip = "10.20.0.13", cores = 4, memory_mb = 8192, disk_gb = 60 }
}

Each VM gets:

• CPU type host, so the guest sees the real CPU features
• A virtio-scsi-single controller with an IO thread and discard turned on, so trimmed blocks are released back to ZFS
• Its disk on a ZFS pool
• A static IP, gateway and DNS set through cloud-init
• The QEMU guest agent turned on, so Proxmox can see the VM’s IP and shut it down cleanly

High Availability on Proxmox

All three VMs are Proxmox HA resources. ZFS replication copies each k8s VM to the other host every 5 minutes, so a failover loses at most a few minutes of disk changes. Soft node-affinity rules tell HA where each VM should normally live. If a host fails or goes into maintenance mode, HA starts its VMs on the surviving host.

The Proxmox HA shutdown policy is migrate. When I reboot a host from the Proxmox UI, its VMs live-migrate to the other host first, so a host reboot doesn’t take the cluster down.

One lesson: Terraform and HA shouldn’t both decide where a VM runs. At first, every terraform apply wanted to migrate the VMs back to the hosts in the map, and then HA moved them right back. Now Terraform ignores changes to node_name. The host in the map only decides where a new VM is created, and the HA rules decide where it runs after that.

migrate = true
lifecycle {
  ignore_changes = [node_name]
}

Installing k3s with Ansible

Once Terraform has built the VMs, one Ansible playbook takes over. It runs in four stages:

1. Prep every node: wait for SSH and cloud-init to finish, install qemu-guest-agent, and download the k3s install script.
2. Install the server: write /etc/rancher/k3s/config.yaml, run the installer, and wait for the API to report ready.
3. Join the agents: install k3s in agent mode, pointed at the server.
4. Deploy the add-ons: drop manifests into k3s’s auto-deploy folder.

I keep the server’s settings in config.yaml rather than in installer flags, because I can change that file later and restart k3s without reinstalling:

resolv-conf: /etc/rancher/k3s/resolv.conf
tls-san:
  - 10.20.0.11
# MetalLB replaces the built-in klipper load balancer
disable:
  - servicelb

The k3s nodes are left out of my monthly “patch everything” playbook on purpose. That playbook reboots servers one after another, and I don’t want it rebooting Kubernetes nodes without draining them first. The cluster handles its own updates, as described below.

Add-ons the k3s Way

k3s ships with a Helm controller. Any manifest placed in /var/lib/rancher/k3s/server/manifests is applied automatically, and that includes HelmChart resources. Ansible only has to write files to that folder. There’s no separate Helm step, and k3s keeps the charts reconciled.

Load Balancing with MetalLB

I turned off k3s’s built-in ServiceLB and use MetalLB in L2 mode instead. There are two address pools:

• homelab (for example 10.20.0.200–10.20.0.210): services that ask for a LoadBalancer get an address from here automatically.
• pinned (autoAssign: false): a few specific addresses that MetalLB never hands out on its own. A service gets one only by requesting it with an annotation. When I moved a service from an old VM into the cluster, it kept the VM’s old IP this way, so nothing else on the network needed changes.

Ingress with Traefik

I kept the Traefik that comes bundled with k3s. A HelmChartConfig pins its service to a fixed MetalLB address, so DNS records for my web apps can point at one IP that never changes:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
spec:
  valuesContent: |-
    service:
      annotations:
        metallb.io/loadBalancerIPs: 10.20.0.200

DNS: Two Gotchas

DNS caused most of my debugging time.

1. Wildcard records and search domains. My router answers every name under my domain with a public wildcard record. By default, pods inherit the node’s search domain. A lookup like metallb.github.io would first be tried as metallb.github.io.mydomain, match the wildcard, and resolve to my own web server. Helm chart downloads failed in confusing ways because of this. The fix is to give k3s its own resolv.conf with only a nameserver and no search line:

# /etc/rancher/k3s/resolv.conf
nameserver 10.20.0.1

2. Split DNS for internal names. Because the router only knows the wildcard, pods also need a way to resolve my internal hostnames. The CoreDNS bundled with k3s loads extra server blocks from a coredns-custom ConfigMap. I use that to send my internal domain to the domain controller and everything else upstream:

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns-custom
  namespace: kube-system
data:
  internal.server: |
    home.example:53 {
        errors
        cache 30
        forward . 10.20.0.3
    }

Hands-Off Maintenance

I wanted the cluster to keep itself up to date without me babysitting it. Two controllers handle this, and both work one node at a time inside a set window on Sunday mornings.

OS Reboots: kured

Ubuntu’s unattended-upgrades installs patches on each node. When a patch needs a reboot, it leaves a /var/run/reboot-required file behind. kured watches for that file. On Sundays between 04:00 and 06:00 it drains the node, reboots it, and uncordons it. It never does more than one node at a time.

k3s Upgrades: system-upgrade-controller

Rancher’s system-upgrade-controller upgrades k3s itself using two Plan resources: one for the server and one for the agents. The agent plan waits for the server plan to finish, so the control plane is always upgraded first. Both plans follow a release channel pinned to a single minor version. Patch releases install automatically on Sundays from 03:00 to 04:00. Moving to a new minor version is something I do on purpose by changing the channel.

channel: https://update.k3s.io/v1-release/channels/v1.36
window:
  days: ["sunday"]
  startTime: "03:00"
  endTime: "04:00"

The windows are lined up so they don’t overlap: Proxmox backups run first on Sunday morning, k3s upgrades run from 03:00 to 04:00, and kured reboots run from 04:00 to 06:00.

Wrapping Up

The whole cluster can be rebuilt from code. Terraform creates the VMs and one Ansible playbook turns them into a working k3s cluster with load balancing, ingress, working DNS and automatic upgrades. Proxmox HA and ZFS replication cover hardware failures, and kured and the upgrade controller keep the nodes patched.

Next time I’ll cover what actually runs on the cluster.