Date: 2026-08-21
Owner: AutoCoder / cluster operations
Related incident: Fastdata exhaustion and VM outage
All six Gitea action runners keep registration identity, action cache, Docker data, BuildKit cache, and job state on worker-local NVMe device /dev/nvme0n1p1. No runner Deployment references fastdata, bigdata, NFS, or the control-plane root disk.
A storage guard and per-runner maintenance sidecar are live. The guard reports health and Prometheus metrics, begins draining idle runners below 300 GiB available, treats below 200 GiB as critical, and restores drained replicas only after recovery to 350 GiB. Busy state comes from Gitea's structured runner API so active jobs finish.
- AutoCoder-Engine PRs #2596, #2597, #2598, and #2600, merged as
89b1a1eea861, 0db8fab14012, bc3a8fe, and 9f9bd49.
- Six deterministic local runner identities registered as IDs 14 through 19; duplicate retired registrations were removed.
- Maintenance every 15 minutes removes stopped containers older than one hour, unused images older than seven days, stale networks, unused volumes, and bounds BuildKit cache to 20-80 GB.
- Guard health endpoint and metrics service are live in namespace
gitea.
- The six obsolete fastdata identity directories were audited and permanently removed; each contained only its retired
.runner file.
- A repeatable isolated pressure-transition harness is maintained in the AutoCoder-Engine repository.
- All six runner pods are
3/3 Running on drg-k8s-worker-01.
/data, /cache, and /var/lib/docker resolve to /dev/nvme0n1p1 in every runner.- Maintenance completed on every runner and reclaimed disposable data without interrupting work.
- AutoCoder Actions run 5557 completed a full Rust backend image build and deployment on runner ID 14.
- Bodylog Actions run 5558 completed Android unit tests and a debug APK build on runner ID 18. Signing, release artifacts, and Play publication were deliberately disabled for this isolated validation; its temporary branch was removed afterward.
- The isolated pressure validation proved warning drain, warning recovery, critical drain, critical recovery, and busy-runner protection. Its selector could match only two dummy Deployments, and all validation resources were removed on exit.
- After validation, every production runner remained at one replica with no drain annotation. Guard health was
ok, pressure was healthy, drained Deployments were zero, and about 1.5 TiB remained available.
- PR #2600 exact-head CI passed after one documented retry for an unrelated transient
ETXTBSY test failure.
REQ-0093 is fully_implemented_complete; all five acceptance checks have passed. Re-run the isolated pressure validation with:
KUBECONFIG=/home/ax_julian/.kube/config \
scripts/validate-gitea-runner-storage-guard.sh