Date: 2026-08-21
Disposable runner data accumulated on the approximately 4 TB /mnt/fastdata NAS. Shared-storage pressure disrupted the virtualization environment and crashed all virtual machines, including Active Directory services.
The incident contained two distinct storage leaks:
The exact outage duration and every affected VM have not yet been reconstructed.
/mnt/fastdata was removed by the operator.drg-k8s-worker-01; their restoration and retention controls are managed separately from the AutoCoder task-worker fix documented below.The historical Rust runner configuration stored its Actions cache at /data/cache. The runner /data path was a hostPath into /mnt/fastdata/k8s-pvcs/gitea/act-runner-rust-*, so disposable caches were written to persistent NFS storage without an effective size limit.
The 2026-08-12 worker migration moved Docker and BuildKit state to worker-local NVMe. A 2026-08-13 follow-up moved Actions caches to NVMe, but old NAS caches were neither removed nor included in the completion audit. Runner identity paths also continued to provide an unnecessary route to shared storage.
Contributing factors included:
/mnt/fastdata/k8s-pvcs as the default application-data locationAll six AutoCoder detached task workers were still scheduled on drg-k8s and mounted /mnt/fastdata/k8s-pvcs/autocoder-engine/workspace-rwx.
Task workspaces were not removed after success, failure, cancellation, supersession, or timeout. Startup did not reconcile existing directories against authoritative task state, and no local-capacity threshold stopped new task claims.
This was independent of the Gitea Actions cache leak and accounted for most of the remaining fastdata usage.
Requirement REQ-0091 and the implementation were merged through AutoCoder-Engine PRs 2587 and 2588.
The detached task workers now:
drg-k8s-worker-01.auth.home.lan/mnt/runner-storage/autocoder-engineThe AutoCoder control plane remains on drg-k8s.
Live verification showed all six workers Ready on worker-01 with the exact mount contract. Two observed post-migration tasks reached terminal state and left no workspace or Codex-home directory behind.
The legacy AutoCoder root contained only ordinary direct-child directories and had no live pod mount.
Cleanup was performed in two phases:
Zero active directories, protected directories, files, or symlinks were deleted. The legacy tasks root is empty.
Fastdata changed from 531,856,621,568 bytes available at 87% utilization to approximately 2.65 TB available at 32% utilization. The cleanup recovered about 2.12 TB decimal, or 1.93 TiB.
See AutoCoder task workers moved to local NVMe for the implementation and verification record.
The normal AutoCoder deployment did not start when PR 2588 merged because no Gitea Actions runner was registered at that point. A manual repository-defined BuildKit build on the control host crossed kubelet's local disk-pressure threshold and caused temporary pod evictions.
The immutable image had already been pushed. The two local image tags and exactly the 27 BuildKit cache records created by that build were removed. Disk pressure cleared, root free space recovered to 85 GiB, and the affected deployments recovered.
Control nodes must not be used as fallback image builders. Builds belong on bounded worker-local NVMe through the managed runner workflow.
Before disposable runners or workers are considered migrated:
The canonical Storage placement policy requires local disk or RAM for temporary work, reserves /mnt/fastdata for bounded latency-sensitive persistence, and directs large or growing datasets to /mnt/bigdata.