Date: 2026-07-21
Status: Resolved
Affected node: drg-k8s.auth.home.lan
Ingress-hosted applications accepted TCP/TLS connections but returned no HTTP response. EventHorizon, Gitea, and Wiki.js were externally unavailable. Existing application containers continued running; EventHorizon remained healthy when reached through the Kubernetes service proxy.
Ready, ingress reconciled automatically, and public routes returned HTTP 200.The host limit fs.inotify.max_user_instances=128 was exhausted by root-owned filesystem watchers (approximately 120 instances were already in use). During containerd startup, the CRI plugin could not create its CNI configuration watcher and failed with:
failed to create CNI conf monitor: fsnotify watcher: too many open files
Kubelet then reported unknown service runtime.v1.RuntimeService, an unhealthy PLEG, and an unavailable container runtime. The system-wide file-handle capacity and containerd process file-descriptor limit were not exhausted.
Created /etc/sysctl.d/99-kubernetes-inotify.conf:
# Kubernetes and containerd run many root-owned filesystem watchers.
fs.inotify.max_user_instances = 1024
Applied the value with sysctl and restarted containerd. Kubelet reconciled existing workloads, and ingress-nginx restarted automatically.
ReadyRuntimeReady=true, NetworkReady=true1/1 Ready1/1 Ready, application container did not restart/healthz, web UI, Gitea, and Wiki.js: HTTP 200d25daff917b6 as liveMonitor root-owned inotify instance usage and investigate what initiated the 06:30 containerd restart. Consider alerting on CRI readiness and sustained ingress probe failures.