Research Compute Infrastructure, Revisited: A Kubernetes Migration
Introduction
In an earlier post, I described over10k, the private Kubernetes cluster we built for our research lab so students and researchers could run Jupyter notebooks against shared CPU, memory, and GPU resources without waiting for access. That post covered the initial architecture: vanilla Kubernetes, JupyterHub via Zero-to-JupyterHub, ingress-nginx, MetalLB, and NVIDIA’s GPU Operator.
We had neglected upgrades for too long, and it caught up with us. I had already tried bumping the GPU Operator once before, on my own, and that attempt had failed and been left half-done. The trigger to finish the job was mundane: we needed to run PyTorch with CUDA 13.x, which our installed GPU Operator version could not support. PyTorch 2.12 shipped CUDA 13.0 as its default build and added an experimental CUDA 13.2 build (PyTorch Foundation 2026). Bumping the operator meant meeting its newer Kubernetes floor, which our v1.24.1 control plane fell short of. The host packages were already on v1.25.16, but nobody had completed the corresponding control-plane upgrade. What looked like “bump one Helm chart” turned into a full audit, which revealed how much else had quietly drifted out of spec. This post covers the migration that followed: ten consecutive minor-version upgrades from v1.24 to v1.34, followed by an eleventh to v1.35 once the rest was stable, plus a JupyterHub major-version bump and a replacement of the ingress stack. I focus on the incidents because several taught me something I did not expect about how a single-node cluster fails and recovers.
What had actually drifted
Before planning any version bumps, a live inspection of the running cluster (not just the Ansible repo describing it) turned up several gaps between what was documented and what was deployed:
- The control plane was running Kubernetes v1.24.1, but the installed
kubeadm/kubelet/kubectlpackages were already v1.25.16. Someone had runapt installon the newer packages without ever runningkubeadm upgrade apply. A kubelet newer than the API server violates Kubernetes’ supported version skew (Kubernetes Authors 2026b), so we had to repair this state before planning the rest of the migration. - MetalLB was pinned to
v0.14.8in the Ansible repo, a pending upgrade that had never actually landed.helm list -Aon the live cluster showedv0.12.1was what was actually running. The upgrade had stalled because MetalLB changed how IP address pools are configured somewhere in between, from aconfigInlineblock in a ConfigMap to dedicatedIPAddressPoolcustom resources. Migrating our own config to the new format had been started but never finished, so the cluster stayed on the older version whose legacy config still worked. On a single node with a single IP address, MetalLB’s load-balancing was not doing any real load balancing anyway (there is nothing to fail over to), so instead of finishing that migration we retired it entirely. cert-managerwas configured in the repo with real Let’s EncryptClusterIssuers, but was not installed at all. TLS was coming from ingress-nginx’s own self-signed default certificate.- ingress-nginx itself was archived upstream in March 2026 and moved into the read-only
kubernetes-retiredGitHub organization (Kubernetes SIG Network 2026), so this migration replaced it with Traefik rather than just bumping its version.
None of this was an emergency by itself, but together it meant the documented state of the cluster and its actual running state had diverged enough that any further upgrade needed a fresh inventory first, not just a diff against the Ansible repo.
Planning and executing with Claude Code
I used Claude Code for both the planning and the execution. For planning, I had it research the actual compatibility ceilings and floors for every component we were running, checked against each project’s own release notes and source rather than summarized secondhand, and turn that into a phased, written plan with an explicit downtime estimate per phase.
For execution, I had it run the plan against the live cluster directly over SSH, phase by phase: taking backups, running the kubeadm/Helm commands, checking the result, and diagnosing and fixing the incidents described below as they came up, rather than only executing predetermined commands blindly. I reviewed and authorized each phase before it touched anything, and stayed on hand for the phases with real downtime.
Safety net first
Before any of that touched the live cluster, the plan called for a safety net first, since over10k is a single-node, single-control-plane cluster with no high availability. A kubectl drain has nowhere to reschedule pods to, so anything that needs the kubelet to restart means real, user-facing downtime while it happens. That safety net had three parts: a full system-disk backup to a dedicated ZFS dataset, an etcd snapshot before every control-plane change, and a switch of every PersistentVolume’s reclaimPolicy from Delete to Retain.
That last change mattered more than the version work. A JupyterHub upgrade in 2024 had silently deleted user notebook volumes: a Helm chart bump crossed a major version boundary without running the Hub’s own database migration step, the Hub came up against a stale-schema SQLite database, its authenticator broke because it expected a column the old schema did not have, and the Hub lost track of correct user state and started deleting volumes it thought were orphaned. With reclaimPolicy: Retain, a PersistentVolumeClaim (PVC) deletion (from any cause) now leaves the underlying data in place as a Released PersistentVolume instead of destroying it outright. It costs a manual cleanup step to actually reclaim the space, and that friction is the point.
Eleven hops, and the incidents along the way
Each Kubernetes minor-version hop followed the same procedure: fresh etcd snapshot, incremental system backup, a compatibility check against every add-on’s documented Kubernetes floor and ceiling, the apt repository switch (Kubernetes moved its package repository to a per-minor-version URL, pkgs.k8s.io, a few years ago), kubeadm upgrade apply, drain, kubelet and kubectl bump, uncordon, and verification. Most hops were routine. A few were not.
Calico’s CRDs went stale silently. Helm installs custom resource definitions from a chart’s crds/ directory on the first installation, but it does not upgrade or delete them (Helm Authors 2026). Because of this, Calico’s CRDs on over10k were still dated from the first installation years earlier. Bumping the Helm release without also reapplying the CRDs left calico-node in CrashLoopBackOff, and it was the cluster’s only network plugin. Applying the chart’s bundled CRDs server-side before every Calico upgrade is now a permanent step in the Ansible playbook.
The NFS storage plugin had already burned us once. csi-driver-nfs backs every PersistentVolume on the cluster, and we had been pinned to v4.0.0 since a past upgrade introduced a poorly documented subDir parameter. It landed a student’s notebook volume at the NFS export’s root instead of its own pvc-<uuid> folder, which took real manual work to fix. This time, bumping to v4.13.4 meant setting subDir deliberately. Claude suggested pinning it to subDir: "${pv.metadata.name}", one of the driver’s documented metadata substitutions (Kubernetes CSI Authors 2026), to reproduce its old per-volume naming. The bump also surfaced a second regression: mountPermissions had changed from our previous effective value of 0777 to the driver’s default of 0. Combined with this cluster’s enableFSGroupPolicy: false, that would have made new volumes unwritable by any non-owning UID. We caught it with a disposable test pod running as UID 1000, JupyterHub’s default, and fixed it before it reached a real user by pinning the permission explicitly.
A GPU Operator upgrade deadlocked itself. During one hop, the GPU Operator’s own controller pod crashed on startup, which stalled its own recreation, because the controller’s autonomous driver-upgrade orchestration had cordoned the node during a version transition and, with the controller down, nothing was left to uncordon it. A manual kubectl uncordon broke the loop.
The control plane hit a real, recurring orphaned-process bug. During one hop, etcd and kube-apiserver both went into CrashLoopBackOff at the same time, independent of the GPU work above. In both cases, containerd had marked the previous container attempt “Exited,” but the underlying OS process was still alive and holding the port, so the new attempt failed with bind: address already in use. etcd recovered cleanly after a SIGTERM to the orphaned process; kube-apiserver needed an escalation to SIGKILL. The same pattern hit calico-typha a few minutes later during the same incident, a third occurrence in one session. This is now a documented recovery pattern: when a static pod or critical daemon reports address already in use, check for a genuinely orphaned process holding the port before assuming it is a configuration problem.
Replacing ingress-nginx with Traefik caused a several-minute outage, from a chain of three separate bugs: a port collision with Traefik’s own dashboard entrypoint, a RollingUpdate briefly running old and new pods with the same fixed host port, and a NET_BIND_SERVICE race under hostNetwork: true that intermittently failed to bind privileged ports. Switching from hostNetwork to plain hostPort (letting the CNI, the cluster’s network plugin, handle privileged-port forwarding via NAT instead of binding directly) resolved the last one and turned out to be the more standard approach anyway.
The JupyterHub bump, treated with real caution
One bump never went through that hop sequence at all: JupyterHub’s, given its own history. The 2024 incident happened at the chart’s 2.x to 3.x boundary (2.0.0 to 3.3.8). This migration crossed the next boundary, from chart 3.3.8 to 4.4.0, and we treated it as a separate project rather than folding it into the Kubernetes version hops.
Before touching it: a fresh etcd snapshot, an incremental system backup, ZFS snapshots, and, this time, a direct SQLite backup of the Hub’s own database (sqlite3.backup, taken while the database was still open, then integrity-checked).
Two configuration changes were needed before the upgrade. Zero-to-JupyterHub 4 moves from KubeSpawner 6 to 7, whose new default slug scheme can change pod and PVC names, so we pinned the legacy scheme to preserve existing names (Project Jupyter Contributors 2026a). JupyterHub 5 also changed its authentication defaults, requiring an explicit allow configuration, so we enabled login for all users who successfully authenticate (Project Jupyter Contributors 2026b). After the upgrade, we checked the Hub logs to confirm that the database migration had actually run rather than assuming success from the Helm command alone. A colleague then independently tested a real end-to-end login.
Where the cluster ended up
At the end of the migration, over10k was running Kubernetes v1.35.7, whose upstream support ends in February 2027 (Kubernetes Authors 2026a). Calico, the GPU Operator, cert-manager, kube-prometheus-stack, and Kepler were all on versions current when we completed the work. The GPU Operator bump that started all of this now provides a driver compatible with CUDA 13.x, so the PyTorch builds we needed run on the cluster as intended. Traefik has fully replaced ingress-nginx, and MetalLB is gone. cert-manager is genuinely live now, issuing certificates from an internal CA instead of relying on ingress-nginx’s self-signed default. Every PersistentVolume on the cluster keeps its Retain reclaim policy.
Since then, we have reviewed and permanently deleted the roughly twenty Released PersistentVolumes left by past deletions. ZFS snapshots still reference their underlying data, so that data will remain until the snapshots age out. The Grafana panel that was misreporting GPU occupancy (it was showing which pod scraped the metrics exporter, not which pod actually held a GPU) is fixed too.
etcdctl is now actually installed on the host too. It had been “fixed” in the Ansible repo for a while without that playbook ever being run against the live machine, a small, specific reminder that a fix committed to a repo is not the same thing as a fix applied to the machine it describes.
Takeaways
A few things I would carry into the next infrastructure project regardless of whether it involves Kubernetes specifically:
- Check what is actually running, not what the repo says should be running. Nearly every surprise in this migration (the control-plane version skew, MetalLB’s real pinned version, cert-manager’s absence, Kepler still on a nine-month-old release) came from a gap between documented and live state, not from the upgrade steps themselves.
- Diff the full default configuration between versions, not just the release notes. Release notes for the GPU Operator upgrade did not mention that Confidential Computing Manager flips from disabled to enabled by default in the new version. Diffing the chart’s own
values.yamlbetween versions caught it before it shipped. - A single automated health check right after a change is not enough. More than one incident here had a delayed onset: pods looked healthy immediately after an upgrade and only failed a few minutes later once a controller’s own reconciliation loop caught up. A second check, a few minutes later, caught what the first one missed.