¶ 2026-09-22 AutoCoder candidate lifecycle preparation and runner NVMe incident
- Prepared immutable release-candidate lifecycle in AutoCoder-Engine PR #6635: dev is integration, the newest validated candidate targets the separate test namespace, and production promotion consumes the exact accepted image digests without rebuilding.
- Published immutable canonical instruction ConfigMap
autocoder-canonical-instructions-49ce86584945c3aa to autocoder-engine and autocoder-test so Engine automation and workers consume the administrator-owned SOP bundle.
- Created
autocoder-test, scoped test and promotion service accounts, dedicated test Gitea read identity/OAuth application, namespace secrets, registry pull secret, and edge route for test.codecartography.dev.
- Split Rust validation and delivery runner labels. Candidate mode remains disabled and legacy dev delivery remains enabled until PR validation, merge, and one-time lifecycle bootstrap complete.
- At 20:34 UTC,
/dev/nvme0n1 timed out on writes. Controller reset failed; the kernel disabled the device at 20:36 UTC.
- ext4 reported lost writes, aborted its journal, warned of potential data loss, and remounted
/mnt/runner-storage read-only.
- Gitea run 3478 jobs failed before checkout with Docker metadata I/O errors. These are infrastructure failures, not implementation test failures.
- The storage guard cordoned
drg-k8s-worker-01.auth.home.lan. All eight Gitea runner/cache/guard deployments were scaled to zero and all remaining hostPath consumer pods were stopped. No /mnt/runner-storage consumer remains.
- The NVMe namespace reports 0 bytes and state
dead; SMART cannot open the device.
- Do not restart runners, uncordon the worker, unmount/repair the filesystem, delete runner data, or retry CI until cold recovery is explicitly approved.
- Required recovery: controlled cold power cycle with return access, read-only salvage of important AutoCoder state to independent storage, offline
e2fsck -f -y -v, independent e2fsck -f -n -v, bounded filesystem write test, then staged runner restoration and exact PR validation.
¶ Rollback and retained state
- Runner deployment replica counts were one before shutdown; restore them only after storage verification.
- Traefik backup retained at
/mnt/fastdata/docker/apps/traefik/config/docker/cert_conf.yml.20260922T1954Z.bak.
- Old development PVC/data were not deleted. No production candidate promotion was performed.
- PR #6635 head at shutdown:
1d3387a16e5926a709ca3414de5e67c1922c1511.