¶ Storage hygiene and host health monitoring
Date: 2026-08-24
Installed two versioned systemd controls from /home/ax_julian/ops-tools/maintenance.
codex-storage-hygiene.timer runs hourly. It:
- removes stale Cargo build output;
- removes non-source temporary entries after seven days;
- removes a stale Git workspace only when every contained repository is clean, has no unpublished commit, and its HEAD is merged into
origin/main, origin/master, or origin/dev;
- preserves active, dirty, unpublished, and unintegrated work;
- removes linked worktrees through Git so stale worktree metadata does not accumulate;
- supports an RFC 3339
.codex-keep-until protection marker;
- expires root Trash and bounds the root npm cache;
- warns at 15 GiB and fails at 20 GiB of task temporary data;
- warns at 75% and fails at 82% root filesystem use.
The cleanup first checks whether content is newer than its retention window, avoiding unnecessary process scans for current work.
host-health-monitor.timer runs every five minutes. It checks storage-policy status, Kubernetes workload availability, node readiness and disk pressure, and the public Immich health endpoint.
A new failure must occur in two consecutive checks before Telegram notification. State changes are deduplicated, and recovery is reported. Telegram credentials remain in the existing Kubernetes Secret and are not copied to the host filesystem.
- Published the patched Immich v2.7.5 image through the Gitea registry and updated the deployment to use that durable reference.
- Merged EmailAssistant PR 82 so personal Outlook credentials bootstrap once into the existing PVC, after which the Secret is optional and removable.
- Merged EmailAssistant PRs 84-88 to tolerate concurrent Maildir moves, bound personal catch-up to 25 messages per poll, keep initial catch-up silent through explicit provenance, migrate existing durable state, and record live completion.
- Removed 95 reviewed, fully integrated temporary Git workspaces and 313 already-failed Kubernetes pod records.
- Both timers are enabled and active.
- A fresh cleanup run completes successfully in under one second for current work.
- Root filesystem use fell from 73% (66 GiB free) to 38% (about 151 GiB free).
- Temporary storage fell from 82 GiB to about 6.3 GiB; remaining large entries are current bounded Garmin tooling and one active merged Bodylog worktree protected by live processes.
- Both nodes are Ready with
DiskPressure=False.
- Immich responds externally with pong and HTTP 200.
- All three mail deployments are available with zero restarts after final rollout; personal Outlook is fully caught up and operates without its bootstrap Secret.