¶ 2026-09-21 AutoCoder global pause and worker NVMe containment
Contain autonomous AutoCoder work while issue recovery is reviewed, and repair global-pause enforcement without using the broken engine to repair itself.
- Production PR #6590 added the global background pause, implementation-worker pause, paused webhook acknowledgement, status endpoint, and mutation/provider-write guards.
- PR #6604 removed a duplicate Kubernetes ConfigMap key that blocked manifest rendering.
- PR #6605 ported the existing development startup guards to production so lifecycle, detached-task, pod, initialization, and startup recovery do not start while paused.
- PR #6606 ports the existing development guards for server-owned self-review, feedback, delivery, escalation, superproject-monitor, and expiry-maintenance loops. Its merge revision is
da2e18feb270a5dd4ca2f4a8a07c3185a1322a3f.
- Applied repository-managed pause values:
AUTOCODER_BACKGROUND_AUTOMATION_ENABLED=false, AUTOCODER_IMPLEMENTATION_WORKERS_ENABLED=false, and AUTOCODER_REQUIREMENTS_AUTOMATION_ENABLED=false.
- Temporarily scaled
deployment/autocoder-engine to zero after live verification showed production-only startup loops still active.
- Temporarily scaled the three Rust runner deployments to zero after their local Docker roots began returning kernel-level
Input/output error.
- No task, cache, repository, or durable data was deleted. An attempted reversible rename to quarantine the three Docker cache roots failed before moving anything because ext4 could not create directory entries.
¶ Evidence and current state
- The exact #6566 image before the final loop guard reached production and proved the main startup guards: no detached-task recovery or startup-recovery log entries appeared.
- Live verification then found an escalation loop panic, proving server-owned loops were still active; production was immediately stopped again.
- Worker storage failures include
chmod /var/lib/docker: input/output error, Docker metadata meta.db: input/output error, failed writes under /mnt/runner-storage/autocoder-engine, and failed rename/create operations under the Rust runner Docker roots.
- Kubernetes still reports the worker Ready and without DiskPressure, so those conditions are not sufficient health evidence.
- Production backend remains at zero replicas. Frontend and PostgreSQL remain present. Rust runners remain at zero replicas. The final guarded image is not built or deployed.
- Cordon and drain
drg-k8s-worker-01.auth.home.lan.
- Perform a controlled node reboot and inspect kernel/NVMe/ext4 health; run offline filesystem repair only if indicated.
- Verify read/write/rename operations under
/mnt/runner-storage before restoring runners.
- Scale the three Rust runner deployments to one and verify Docker daemon health.
- Rerun the production workflow for revision
da2e18feb270a5dd4ca2f4a8a07c3185a1322a3f.
- Restore AutoCoder to one replica only on the exact successful image, then prove no startup recovery, reconciliation, provider writes, task claims, or worker pods occur while globally paused.
Keep AutoCoder and the Rust runners at zero until storage integrity is established. Do not fall back to control-plane or NAS build storage; the storage placement policy requires worker-local NVMe for AutoCoder execution.