Self-hosted Kubernetes for local development, shareable on demand
A three-node kubeadm cluster on arm64 that runs my local development work on a VMware host-only network, with real names and TLS. Nothing outside my machine can reach or discover it. Egress goes through a separate NAT interface, and any service can be shared publicly on demand through a Cloudflare tunnel that dials out over that NAT. Every component ships as plain, digest-pinned manifests with a functional verification gate.
0 inbound ports- Role
- Sole engineer
- Scale
- 3 nodes, 1 control plane, arm64
- Stack
- kubeadm 1.36, CRI-O, Calico
- Targets
- Host-only by default, public on demand, no inbound path
Context
This is a three-node kubeadm cluster on arm64 virtual machines: one control plane and two workers, running CRI-O and Calico. Its job is to host my local development work, reachable from my machine by name with valid TLS, the same way it would run in production.
Each VM has two network interfaces. A VMware host-only network carries all cluster and client traffic, and only my host can reach it, so nothing else on any network can discover the cluster. A NAT interface provides egress only: image pulls, certificate issuance and the tunnel all dial out through it, and nothing can come in.
When I want someone else to see something, I share that one service through a Cloudflare tunnel. Everything else stays host-only. The first service shared this way is a documentation site that describes the cluster it runs on.
Constraints:
- Host-only by default; public access is opt-in per service
- No inbound path at all: the NAT interface is egress-only
- Small arm64 VMs: every image must be multi-arch and verified
- A single person operates everything
- Everything reviewable as plain manifests in git
Architecture and failover
Nodes sit on a host-only 192.x.x.x/24, with a second NAT interface for egress. Pods use 10.x.x.x/16, because Calico's default pod range collides with the node subnet.
On the host-only network, which is the default path, my machine resolves names through a scoped resolver pointed at an in-cluster CoreDNS, and reaches a MetalLB layer-2 virtual IP. Every development service gets a *.internal.<domain> name and lands on Traefik, which serves one wildcard certificate from cert-manager (Let's Encrypt, DNS-01). DNS-01 is the only challenge that works here, because Let's Encrypt can never reach a host-only address.
To share a service, I add a one-label public hostname to the tunnel. The Cloudflare edge forwards it through the connection that two cloudflared replicas hold open, outbound over NAT,, which hand it to the same Traefik. Routes decide what is reachable from outside: the documentation site is public, but its Kubernetes API paths are not routable from the tunnel at all, and it reads the API with a read-only service account, an allowlist of resource kinds, and secret redaction.
Try it: take out the node running Traefik, lose the control plane, or replay the MetalLB storm. The shared documentation site stands in for any service. Timings are illustrative, built from Kubernetes defaults and lab notes.
- Primary / serving
- Recovering
- Degraded
- Down
- Replication / control
Pick a failure to see how the system detects and recovers from it.
Key decisions
Outbound tunnel instead of port forwarding
The cluster sits on a host-only network that nothing outside can reach. Some services still need to be shown to other people, without exposing the home WAN address.
Trade-off: The public hostname list now lives in the Cloudflare dashboard, not in git, because the tunnel uses a remotely-managed token.
Two names per service: one host-only, one public
One hostname cannot be both a proxied CNAME to the tunnel and a plain A record to the host-only load balancer.
Trade-off: Every service has two names, and the public one must sit exactly one label below the apex to get an edge certificate.
MetalLB pools selected by Namespace label
Whoever can create a Service should not be able to claim an address from the production or shared-infrastructure pool.
Trade-off: Assigning a pool requires access to label Namespaces, which is an operator step, not a developer one.
Plain manifests instead of Helm releases
Helm releases hid what was actually running, and upgrades were opaque.
Trade-off: Upgrades are manual: render the new chart, diff it against the stored snapshot, then port the changes by hand.
Node-local volumes instead of replicated storage
Three small arm64 VMs have little memory to spare for a storage system.
Trade-off: A volume is pinned to its node, so a pod with data cannot restart elsewhere while that node is down.
Lifecycle
- Run
kubectl diffagainst the live cluster, thenkubectl apply. Numbered directories encode apply order: CRDs, config, RBAC, workloads, admission. - Pin every applied image by digest and check it has an arm64 build.
- Prove the change with its functional gate, never "pod is Running": a bound PVC with a write/read Job,
kubectl topreturning data, a CertificateReadywith both names.
- Render the new chart version with
helm template. - Diff it against the stored snapshot of the previous version.
- Port the real changes into the plain manifests. Apply CRDs first, then the rest, then rerun the gate.
- Build the image with
podman buildand push it to the in-cluster zot registry. - Pin the new digest in the kustomization.
kubectl apply -k, then run the unit and end-to-end tests against the live site.
- A CronJob takes an etcd snapshot every 12 hours.
- A second container verifies it with
etcdutl snapshot status. - The snapshot is a single file on the control-plane node's own disk. Restore has not been tested yet.
Failure modes
| Failure | Blast radius | Detection | Recovery |
|---|---|---|---|
| Worker running Traefik lost | Public and host traffic down ~6 min | Users, then NotReady node | Automatic reschedule after the 300 s toleration |
| Control plane lost | No scheduling or snapshots; app API pages degrade | apiserver unreachable | Manual: restart the VM |
| Control-plane disk lost | Cluster state and the only snapshot gone | apiserver unreachable | None today: snapshot is on the same disk |
| Calico webhook CA wiped by an apply | Every network policy write fails, cluster-wide | Policy writes return TLS errors | Re-patch the CA; the field is now omitted from the manifest |
| MetalLB reconcile storm | apiserver saturated at ~900 req/s; serving unaffected | Host fans, then a per-resource request breakdown | Delete the ServiceL2Status CRD after every MetalLB apply |
| WireGuard MTU mismatch | Image pulls fail; no new pod can start | Requests succeed from one worker and fail from the other | Rolled back the same day |
| Kubelet serving cert not approved | kubectl top silently fails for that node | Pending CSR | Manual approve after checking the CSR identity |
| Staging certificate on an HSTS-preloaded domain | Browsers hard-block every host | First human visit | Cut over to the production issuer |
Results
What I'd change
I would copy the etcd snapshot off the control-plane node and test a restore end to end. Today the backup protects against a corrupt database, not against losing the disk it sits on.
I would run Traefik as two replicas with anti-affinity and a PodDisruptionBudget. It is the one component both the public and host-only paths depend on.
I would automate the MetalLB CRD deletion as a post-apply step, and the kubelet serving CSR approval behind an identity check, instead of relying on remembering either.