¶ Local disk pressure and Immich outage
Incident began: 2026-08-10
Detected and remediated: 2026-08-24
Status: Resolved
The control-plane node reached the kubelet image-filesystem threshold and evicted workloads. Most deployments recovered automatically. Immich remained unavailable and its public endpoint returned HTTP 503. The personal Outlook import also remained unavailable because its bootstrap Secret and durable credential file were both absent.
/home/ax_julian/temp had grown to 82 GiB, primarily from retained task workspaces, Cargo targets, Android SDKs, Gradle caches, APKs, logs, and downloads.- Kubernetes/containerd used 38 GiB on the same 252 GiB root filesystem.
- Task temporary storage had no enforced lifetime, capacity budget, or cleanup owner; systemd tmpfiles did not manage it.
- No active disk, workload-availability, or external Immich health alert existed.
- Immich used a custom image imported only into one node's containerd cache. Its
drg-k8s.local reference was not backed by a registry, so Kubernetes could not restore it on another node.
- The personal Outlook bridge mounted its bootstrap Secret as permanently mandatory, but the Secret did not exist and its credential PVC was empty. The original deployment had never completed one-time authorization.
- 2026-08-10: root use reached 85%; repeated image garbage collection failed and workload eviction began. Immich became unavailable.
- 2026-08-20: control-plane node restarted.
- 2026-08-21: disk pressure and eviction recurred; image cleanup reclaimed several GiB.
- 2026-08-22: replacement Immich pod entered
ImagePullBackOff because drg-k8s.local:443 refused connections.
- 2026-08-24: investigation identified the outage and retained temporary data. Guarded cleanup reduced root use to 38%.
- 2026-08-24: the custom Immich image was published to the authenticated Gitea registry and the deployment was updated. The replacement pod became ready on the worker node; external ping and homepage checks succeeded.
- 2026-08-24: EmailAssistant PR 82 made the authentication Secret an optional one-time bootstrap and copied credentials with mode 0600 to its durable PVC.
- 2026-08-24: personal Microsoft authorization completed. During initial catch-up, a Maildir concurrency race restarted the automation sidecar, and 201 history messages entered the inbound path before explicit import provenance was added; 39 were notification-eligible. The bridge was paused, PRs 84-87 added race tolerance, bounded 25-message batches, durable history/inbound provenance, and migration of existing catch-up state.
- 2026-08-24: catch-up completed with 913 checkpointed, zero rejected or pending. PR 88 recorded the verified requirements as complete.
- Preserved dirty, unpublished, unintegrated, or active Git work and removed 95 proven-integrated temporary workspaces.
- Removed stale generated output, disposable caches, temporary credential exports, root Trash, and 313 obsolete failed pod records.
- Published
gitea.app.lan/ax_bechju1/immich-server:v2.7.5-large-image-metadata and replaced the node-local image reference.
- Added hourly bounded temporary-storage cleanup. It removes a stale Git workspace only when clean, published, and merged into a known remote default branch; active work is retained.
- Added five-minute host/workload/public-endpoint health monitoring with deduplicated Telegram state-change alerts.
- Updated global Codex instructions to require task-owned, expiring temporary work and prohibit secrets in general temporary directories.
- Changed Outlook credentials to durable PVC ownership and removed the one-time bootstrap Secret.
- Made history processing skip files concurrently moved by IMAP clients instead of restarting.
- Bounded personal Outlook catch-up to 25 messages per poll and added typed local provenance so initial history remains subject to the silent daily history cap while later mail remains inbound.
- Both Kubernetes nodes are Ready with
DiskPressure=False.
- Root use is 38%, with approximately 151 GiB available.
- Immich has one available replica using the registry-backed image;
/api/server/ping returns pong and the public homepage returns HTTP 200.
- Mailserver, Microsoft 365 bridge, and personal Outlook bridge are all available with zero restarts after the final deployment.
- Personal Outlook reports 913 checkpointed messages, zero rejected/pending, and repeated
outlook_pull_status=ok.
- Durable Outlook auth is mode 0600 on the PVC; the bootstrap Secret is absent and restart verification succeeded.
- All 17 EmailAssistant tests pass, including history concurrency, rate bound, provenance, and post-catch-up inbound transition.
- 313 stale Failed or Evicted pod records were removed.