{"id":1009,"date":"2026-11-03T15:53:06","date_gmt":"2026-11-03T15:53:06","guid":{"rendered":"https:\/\/xircx.com\/?p=1009"},"modified":"2026-11-03T15:54:07","modified_gmt":"2026-11-03T15:54:07","slug":"how-my-home-lab-kubernetes-cluster-is-built-k3s-on-proxmox","status":"publish","type":"post","link":"https:\/\/xircx.com\/?p=1009","title":{"rendered":"How My Home Lab Kubernetes Cluster Is Built: k3s on Proxmox"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">My home lab runs a small Kubernetes cluster for the services I self-host. It&#8217;s three virtual machines on Proxmox running <a href=\"https:\/\/k3s.io\">k3s<\/a>. Terraform builds the VMs and Ansible installs and configures the cluster. This post covers how the nodes are set up. The apps that run on the cluster are a topic for another post.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<h2 class=\"wp-block-heading\">The Big Picture<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Hypervisor:<\/strong> a two-node Proxmox cluster (pve1 and pve2). A small QDevice VM on a separate box gives the cluster a third vote.<\/li>\n\n\n\n<li><strong>Kubernetes:<\/strong> k3s with one server node (k8s-1) and two agent nodes (k8s-2 and k8s-3).<\/li>\n\n\n\n<li><strong>Provisioning:<\/strong> Terraform with the <code>bpg\/proxmox<\/code> provider creates the VMs.<\/li>\n\n\n\n<li><strong>Configuration:<\/strong> an Ansible playbook installs k3s and the cluster add-ons.<\/li>\n\n\n\n<li><strong>Upkeep:<\/strong> OS reboots and k3s upgrades run on their own, one node at a time, during a Sunday-morning window.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">The Nodes<\/h2>\n\n\n\n<figure class=\"wp-block-table\"><table><thead><tr><th>Node<\/th><th>Role<\/th><th>Preferred host<\/th><th>vCPU \/ RAM \/ Disk<\/th><th>IP (example)<\/th><\/tr><\/thead><tbody><tr><td>k8s-1<\/td><td>server (control plane)<\/td><td>pve2<\/td><td>4 \/ 8 GB \/ 60 GB<\/td><td>10.20.0.11<\/td><\/tr><tr><td>k8s-2<\/td><td>agent<\/td><td>pve1<\/td><td>4 \/ 8 GB \/ 60 GB<\/td><td>10.20.0.12<\/td><\/tr><tr><td>k8s-3<\/td><td>agent<\/td><td>pve1<\/td><td>4 \/ 8 GB \/ 60 GB<\/td><td>10.20.0.13<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">The control plane runs on a different physical host from the two agents. If one Proxmox box goes down, part of the cluster keeps running while HA brings the rest back.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Building the VMs with Terraform<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Every VM starts from the official Ubuntu 24.04 (Noble) cloud image. Proxmox storage isn&#8217;t shared between my hosts, so Terraform downloads the image once to each host. The VMs are defined in a single map, and that map is the only thing I edit to add or resize a node:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>nodes = {\n  \"k8s-1\" = { vm_id = 240, pve_node = \"pve2\", ip = \"10.20.0.11\", cores = 4, memory_mb = 8192, disk_gb = 60 }\n  \"k8s-2\" = { vm_id = 241, pve_node = \"pve1\", ip = \"10.20.0.12\", cores = 4, memory_mb = 8192, disk_gb = 60 }\n  \"k8s-3\" = { vm_id = 242, pve_node = \"pve1\", ip = \"10.20.0.13\", cores = 4, memory_mb = 8192, disk_gb = 60 }\n}<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Each VM gets:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>CPU type <code>host<\/code>, so the guest sees the real CPU features<\/li>\n\n\n\n<li>A <code>virtio-scsi-single<\/code> controller with an IO thread and discard turned on, so trimmed blocks are released back to ZFS<\/li>\n\n\n\n<li>Its disk on a ZFS pool<\/li>\n\n\n\n<li>A static IP, gateway and DNS set through cloud-init<\/li>\n\n\n\n<li>The QEMU guest agent turned on, so Proxmox can see the VM&#8217;s IP and shut it down cleanly<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">High Availability on Proxmox<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">All three VMs are Proxmox HA resources. ZFS replication copies each k8s VM to the other host every 5 minutes, so a failover loses at most a few minutes of disk changes. Soft node-affinity rules tell HA where each VM should normally live. If a host fails or goes into maintenance mode, HA starts its VMs on the surviving host.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The Proxmox HA shutdown policy is <code>migrate<\/code>. When I reboot a host from the Proxmox UI, its VMs live-migrate to the other host first, so a host reboot doesn&#8217;t take the cluster down.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">One lesson: <strong>Terraform and HA shouldn&#8217;t both decide where a VM runs.<\/strong> At first, every <code>terraform apply<\/code> wanted to migrate the VMs back to the hosts in the map, and then HA moved them right back. Now Terraform ignores changes to <code>node_name<\/code>. The host in the map only decides where a new VM is created, and the HA rules decide where it runs after that.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>migrate = true\nlifecycle {\n  ignore_changes = &#91;node_name]\n}<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">Installing k3s with Ansible<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Once Terraform has built the VMs, one Ansible playbook takes over. It runs in four stages:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>Prep every node:<\/strong> wait for SSH and cloud-init to finish, install <code>qemu-guest-agent<\/code>, and download the k3s install script.<\/li>\n\n\n\n<li><strong>Install the server:<\/strong> write <code>\/etc\/rancher\/k3s\/config.yaml<\/code>, run the installer, and wait for the API to report ready.<\/li>\n\n\n\n<li><strong>Join the agents:<\/strong> install k3s in agent mode, pointed at the server.<\/li>\n\n\n\n<li><strong>Deploy the add-ons:<\/strong> drop manifests into k3s&#8217;s auto-deploy folder.<\/li>\n<\/ol>\n\n\n\n<p class=\"wp-block-paragraph\">I keep the server&#8217;s settings in <code>config.yaml<\/code> rather than in installer flags, because I can change that file later and restart k3s without reinstalling:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>resolv-conf: \/etc\/rancher\/k3s\/resolv.conf\ntls-san:\n  - 10.20.0.11\n# MetalLB replaces the built-in klipper load balancer\ndisable:\n  - servicelb<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The k3s nodes are left out of my monthly &#8220;patch everything&#8221; playbook on purpose. That playbook reboots servers one after another, and I don&#8217;t want it rebooting Kubernetes nodes without draining them first. The cluster handles its own updates, as described below.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Add-ons the k3s Way<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">k3s ships with a Helm controller. Any manifest placed in <code>\/var\/lib\/rancher\/k3s\/server\/manifests<\/code> is applied automatically, and that includes <code>HelmChart<\/code> resources. Ansible only has to write files to that folder. There&#8217;s no separate Helm step, and k3s keeps the charts reconciled.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Load Balancing with MetalLB<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">I turned off k3s&#8217;s built-in ServiceLB and use MetalLB in L2 mode instead. There are two address pools:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>homelab<\/strong> (for example 10.20.0.200\u201310.20.0.210): services that ask for a LoadBalancer get an address from here automatically.<\/li>\n\n\n\n<li><strong>pinned<\/strong> (<code>autoAssign: false<\/code>): a few specific addresses that MetalLB never hands out on its own. A service gets one only by requesting it with an annotation. When I moved a service from an old VM into the cluster, it kept the VM&#8217;s old IP this way, so nothing else on the network needed changes.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Ingress with Traefik<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">I kept the Traefik that comes bundled with k3s. A <code>HelmChartConfig<\/code> pins its service to a fixed MetalLB address, so DNS records for my web apps can point at one IP that never changes:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>apiVersion: helm.cattle.io\/v1\nkind: HelmChartConfig\nmetadata:\n  name: traefik\n  namespace: kube-system\nspec:\n  valuesContent: |-\n    service:\n      annotations:\n        metallb.io\/loadBalancerIPs: 10.20.0.200<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">DNS: Two Gotchas<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">DNS caused most of my debugging time.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>1. Wildcard records and search domains.<\/strong> My router answers every name under my domain with a public wildcard record. By default, pods inherit the node&#8217;s search domain. A lookup like <code>metallb.github.io<\/code> would first be tried as <code>metallb.github.io.mydomain<\/code>, match the wildcard, and resolve to my own web server. Helm chart downloads failed in confusing ways because of this. The fix is to give k3s its own <code>resolv.conf<\/code> with only a nameserver and no search line:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># \/etc\/rancher\/k3s\/resolv.conf\nnameserver 10.20.0.1<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>2. Split DNS for internal names.<\/strong> Because the router only knows the wildcard, pods also need a way to resolve my internal hostnames. The CoreDNS bundled with k3s loads extra server blocks from a <code>coredns-custom<\/code> ConfigMap. I use that to send my internal domain to the domain controller and everything else upstream:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>apiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: coredns-custom\n  namespace: kube-system\ndata:\n  internal.server: |\n    home.example:53 {\n        errors\n        cache 30\n        forward . 10.20.0.3\n    }<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">Hands-Off Maintenance<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">I wanted the cluster to keep itself up to date without me babysitting it. Two controllers handle this, and both work one node at a time inside a set window on Sunday mornings.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">OS Reboots: kured<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Ubuntu&#8217;s unattended-upgrades installs patches on each node. When a patch needs a reboot, it leaves a <code>\/var\/run\/reboot-required<\/code> file behind. <a href=\"https:\/\/kured.dev\">kured<\/a> watches for that file. On Sundays between 04:00 and 06:00 it drains the node, reboots it, and uncordons it. It never does more than one node at a time.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">k3s Upgrades: system-upgrade-controller<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Rancher&#8217;s system-upgrade-controller upgrades k3s itself using two <code>Plan<\/code> resources: one for the server and one for the agents. The agent plan waits for the server plan to finish, so the control plane is always upgraded first. Both plans follow a release channel pinned to a single minor version. Patch releases install automatically on Sundays from 03:00 to 04:00. Moving to a new minor version is something I do on purpose by changing the channel.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>channel: https:\/\/update.k3s.io\/v1-release\/channels\/v1.36\nwindow:\n  days: &#91;\"sunday\"]\n  startTime: \"03:00\"\n  endTime: \"04:00\"<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The windows are lined up so they don&#8217;t overlap: Proxmox backups run first on Sunday morning, k3s upgrades run from 03:00 to 04:00, and kured reboots run from 04:00 to 06:00.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Wrapping Up<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The whole cluster can be rebuilt from code. Terraform creates the VMs and one Ansible playbook turns them into a working k3s cluster with load balancing, ingress, working DNS and automatic upgrades. Proxmox HA and ZFS replication cover hardware failures, and kured and the upgrade controller keep the nodes patched.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Next time I&#8217;ll cover what actually runs on the cluster.<\/p>\n<div class=\"pvc_clear\"><\/div><p id=\"pvc_stats_1009\" class=\"pvc_stats all  \" data-element-id=\"1009\" style=\"\"><i class=\"pvc-stats-icon small\" aria-hidden=\"true\"><svg aria-hidden=\"true\" focusable=\"false\" data-prefix=\"far\" data-icon=\"chart-bar\" role=\"img\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" viewBox=\"0 0 512 512\" class=\"svg-inline--fa fa-chart-bar fa-w-16 fa-2x\"><path fill=\"currentColor\" d=\"M396.8 352h22.4c6.4 0 12.8-6.4 12.8-12.8V108.8c0-6.4-6.4-12.8-12.8-12.8h-22.4c-6.4 0-12.8 6.4-12.8 12.8v230.4c0 6.4 6.4 12.8 12.8 12.8zm-192 0h22.4c6.4 0 12.8-6.4 12.8-12.8V140.8c0-6.4-6.4-12.8-12.8-12.8h-22.4c-6.4 0-12.8 6.4-12.8 12.8v198.4c0 6.4 6.4 12.8 12.8 12.8zm96 0h22.4c6.4 0 12.8-6.4 12.8-12.8V204.8c0-6.4-6.4-12.8-12.8-12.8h-22.4c-6.4 0-12.8 6.4-12.8 12.8v134.4c0 6.4 6.4 12.8 12.8 12.8zM496 400H48V80c0-8.84-7.16-16-16-16H16C7.16 64 0 71.16 0 80v336c0 17.67 14.33 32 32 32h464c8.84 0 16-7.16 16-16v-16c0-8.84-7.16-16-16-16zm-387.2-48h22.4c6.4 0 12.8-6.4 12.8-12.8v-70.4c0-6.4-6.4-12.8-12.8-12.8h-22.4c-6.4 0-12.8 6.4-12.8 12.8v70.4c0 6.4 6.4 12.8 12.8 12.8z\" class=\"\"><\/path><\/svg><\/i> <img loading=\"lazy\" decoding=\"async\" width=\"16\" height=\"16\" alt=\"Loading\" src=\"https:\/\/xircx.com\/wp-content\/plugins\/page-views-count\/ajax-loader-2x.gif\" border=0 \/><\/p><div class=\"pvc_clear\"><\/div>","protected":false},"excerpt":{"rendered":"<p>My home lab runs a small Kubernetes cluster for the services I self-host. It&#8217;s three virtual machines on Proxmox running k3s. Terraform builds the VMs and Ansible installs and configures the cluster. This post covers how the nodes are set up. The apps that run on the cluster are a topic for another post. The [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"site-container-style":"default","site-container-layout":"default","site-sidebar-layout":"default","site-transparent-header":"default","disable-article-header":"default","disable-site-header":"default","disable-site-footer":"default","disable-content-area-spacing":"default","advanced_seo_description":"","jetpack_seo_html_title":"","jetpack_seo_noindex":false,"jetpack_seo_schema_type":"","pgc_sgb_lightbox_settings":"","_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_post_was_ever_published":false},"categories":[9],"tags":[],"class_list":["post-1009","post","type-post","status-publish","format-standard","hentry","category-blog"],"a3_pvc":{"activated":true,"total_views":4,"today_views":4},"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/xircx.com\/index.php?rest_route=\/wp\/v2\/posts\/1009","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/xircx.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/xircx.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/xircx.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/xircx.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1009"}],"version-history":[{"count":2,"href":"https:\/\/xircx.com\/index.php?rest_route=\/wp\/v2\/posts\/1009\/revisions"}],"predecessor-version":[{"id":1011,"href":"https:\/\/xircx.com\/index.php?rest_route=\/wp\/v2\/posts\/1009\/revisions\/1011"}],"wp:attachment":[{"href":"https:\/\/xircx.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1009"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/xircx.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1009"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/xircx.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1009"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}