Date: 2026-09-21 UTC
drg-k8s-worker-01.auth.home.lan lost its worker-local Corsair Force MP600 NVMe (/dev/nvme0n1p1, ext4, mounted normally at /mnt/runner-storage). AutoCoder execution storage and the Gitea Actions runner Docker/cache directories returned kernel-level I/O errors. Docker could not read BuildKit meta.db.
The first kernel fault was an NVMe write timeout at 17:31:27 UTC. Controller reset attempts failed, the kernel disabled the device, and ext4 aborted its journal. No Kubernetes pod deletion or stop event occurred immediately before the first timeout. The evidence establishes a controller/PCIe/device hang, not NAS failure or exhausted capacity; it does not establish ordinary pod deletion as the cause.
/mnt/runner-storage.After physical power-on, the NVMe enumerated at its full 1.8 TiB size. The SATA root filesystem was manually remounted read-write and containerd and kubelet were restored. The node returned Ready but remains cordoned.
Read-only e2fsck -f -n found deleted-inode directory entries, bitmap/count mismatches, and a group bitmap checksum error, mainly in disposable runner/cache paths. Before repair, 81,400,023,040 bytes of readable AutoCoder state were archived to /mnt/bigdata/recovery/drg-k8s-worker-01/2026-09-21-nvme-ext4/autocoder-engine-readable.tar, excluding rebuildable target/cache/scratch/tmp directories. The archive produced zero tar warnings and has a recorded SHA-256 digest.
With explicit administrator approval, offline e2fsck -f -y recovered the journal, optimized one extent tree, and corrected free-block and free-inode accounting. A second offline e2fsck -f -n -v completed with exit status 0 and zero bad blocks. A bounded 16 MiB write/fsync/chmod/rename/readback/delete/sync test passed without kernel errors, followed by a clean unmount. The NVMe short device self-test passed; SMART reports critical warning 0, 6% wear, and zero media errors.
The original controller disappearance required full power removal, so passing SMART and filesystem tests do not prove the hardware reliable. The NVMe remains unmounted and quarantined; all runner deployments remain at zero and the node remains cordoned.
The NVMe fstab entry now includes noauto,nofail. findmnt --verify reports no errors or warnings, and the temporary systemd mask was removed. This prevents the quarantined mount from blocking local-fs.target or leaving the SATA root read-only on a later reboot. The prior fstab is retained on the worker as /etc/fstab.pre-runner-storage-recovery-20260921.
Do not remove noauto, uncordon the node, or restore runner replicas until the NVMe/controller risk is accepted or the device is replaced and the replacement passes filesystem and workload validation.
The recovery directory on /mnt/bigdata contains the restricted AutoCoder archive, SHA-256 manifest, tar warning log, modifying fsck log, and clean verification log. Review retention after hardware disposition; deletion requires a separate authorized cleanup decision.
At the administrator's direction, the repaired drive was returned to service on 2026-09-21 UTC because no replacement NVMe is currently available.
/dev/nvme0n1p1 read-write at /mnt/runner-storage; a fresh write/fsync/rename/chmod/delete test passed.drg-k8s-worker-01.auth.home.lan.docker info using overlay2, and every runner registered successfully with Gitea.For reboot durability, the temporary noauto quarantine was removed from the fstab entry. The drive now uses defaults,noatime,nofail,x-systemd.device-timeout=10s; findmnt --verify passes and the generated mount unit is wanted by local-fs.target. The prior quarantine fstab is retained as /etc/fstab.pre-runner-storage-resume-20260921.
The service is operational, but the hardware risk remains: the controller previously disappeared until full power removal. Continue monitoring for NVMe timeout/reset, ext4, Docker metadata, or kernel I/O errors and stop writers immediately if any recur.