Featured case study · Infrastructure · Active build · Public repository
A three-node Kubernetes lab designed to reveal its own constraints.
I turned repurposed hardware into a mixed-architecture k3s cluster, then built a small queue-driven application and observability stack to see how scheduling, autoscaling, and failure behave on real limits.
- physical nodes
- 3
- queue workers
- 1–5
- desired state
- GitOps
- metrics retention
- 2 days
01 · Context
A place to practise operational decisions, not just deployments.
Cloud tutorials make it easy to create resources without feeling the cost of the underlying machines. I wanted a lab where CPU architecture, memory, storage, networking, and recovery decisions were impossible to ignore.
The result is a small k3s cluster that runs real services and a purpose-built KubePets workload. Git contains the desired state, while the cluster provides feedback through metrics, pod state, and deliberately induced pressure.
02 · Constraints
Heterogeneous hardware makes placement an architectural decision.
The cluster combines ARM64 and AMD64 machines with very different memory and storage profiles. The lowest-powered node has about 905 MB of memory; the Pi 5 is the only node with NVMe storage; and the HP server carries workloads that require more headroom.
- — Mixed ARM64 and AMD64 nodes
- — A 905 MB low-power worker
- — Node-local persistent storage
- — No inbound router ports
- — Public repository with encrypted secrets only
- — Resource limits sized for reused hardware
03 · Architecture
Two flows meet in the cluster: desired state and live traffic.
Git and Flux control what should exist. Cloudflare Tunnel and Traefik control how selected traffic reaches it. Scheduling rules place each workload where the hardware can support—or deliberately stress—it.
k3s cluster
04 · Implementation
The repository describes the system from delivery to observation.
Flux reconciles Kubernetes manifests and Helm releases from Git. GitHub Actions builds application images, including multi-architecture images where workloads may run on either ARM64 or AMD64 nodes.
Traefik handles cluster ingress. Cloudflare Tunnel provides outbound-only public connectivity, cert-manager manages certificates, and SOPS keeps secrets encrypted in the public repository. Prometheus, Grafana, kube-state-metrics, node-exporter, and Headlamp expose cluster behaviour without making management interfaces public.
Decision record
Why these choices fit this lab
Scale on queue depth, not CPU
The worker blocks while waiting for Redis jobs, so CPU is a poor signal of accumulated work. KEDA reads the backlog directly.
Make scheduling explicit
State follows the Pi 5 NVMe disk, high-headroom interfaces use the HP server, and failure-testing workers stay on the Pi 3B.
Keep Git public without exposing secrets
SOPS-encrypted manifests preserve the GitOps workflow while keeping secret material unreadable in the repository.
Use an outbound-only tunnel
Cloudflare Tunnel avoids router port forwarding and still allows TLS to continue through Traefik to selected services.
05 · Failure testing
A queue turns pressure into something measurable.
KubePets includes a controlled load path that publishes a burst of events to Redis. Workers consume that queue and update PostgreSQL. KEDA watches the actual backlog rather than CPU, because a blocking queue worker can have low CPU usage even when work is accumulating.
The worker is deliberately pinned to the 905 MB Raspberry Pi tier. That makes memory pressure and eviction visible in Grafana and Kubernetes state instead of hiding the constraint on the most capable machine.
- 01Publish a controlled burst of up to 5,000 events to the Redis hunger queue.
- 02Observe KEDA increase the worker deployment from one replica toward its five-replica ceiling.
- 03Watch queue depth, pod state, and Pi 3B memory pressure in the provisioned dashboards.
- 04Confirm the workers return to one replica after the backlog clears and the 60-second scale-down window passes.
06 · Result
The lab now demonstrates behaviour, not just a list of installed tools.
A repository change can reconcile into the cluster, application images are built for their intended hardware, queue depth controls worker replicas, and dashboards show the effect on the constrained node. Public traffic reaches selected services without opening inbound ports on the home network.
Just as importantly, the repository records why workloads are pinned, why metrics retention is short, why some images are single-architecture, and where the design intentionally stops short of production high availability.
Repository evidence
Follow the implementation, not a marketing claim.
These links open the exact public configuration behind the main claims. Secret values remain encrypted and private interfaces are not exposed.
minReplicaCount: 1
maxReplicaCount: 5
pollingInterval: 15
triggers:
- type: redis
metadata:
listName: hunger-queue
listLength: "100"07 · Reflection
The limitations are part of the evidence.
This is not a production-grade highly available platform. It has one control-plane node, local-path storage ties state to a machine, and the low-memory worker is intentionally fragile. Calling those constraints out is more useful than describing the cluster as universally resilient.
The next meaningful improvements are a tested backup-and-restore path, clearer recovery objectives, and storage that can survive a node loss. A second control plane would only be justified after those more immediate failure modes are addressed.