Date: 2026-01-27
Owner: codexops
Status: In progress
Executed the single-node kubeadm installation per runbook. A reset was required after an initial unstable control plane state. Re-initialized successfully and installed Calico, ingress-nginx, and local-path provisioner. Cluster is up but node remains NotReady while CNI components finish initializing.
- 22:56:53Z: Started execution.
- 22:57Z: System prep complete (swap off, modules, sysctl).
- 22:58Z: containerd installed and enabled.
- 22:58Z: kubeadm/kubelet/kubectl 1.35.0-1.1 installed and held.
- 22:57–23:05Z: First kubeadm init experienced API instability; performed kubeadm reset and cleaned CNI config.
- 23:05–23:06Z: kubeadm init re-run succeeded.
- 23:06Z: Calico (v3.31.3) applied.
- 23:06Z: ingress-nginx (v1.14.2) applied.
- 23:06Z: local-path provisioner (v0.0.34) applied.
- Control plane pods running.
- Node status: NotReady (CNI not fully initialized yet).
- Calico pods in Init/Pending.
- CoreDNS Pending.
- Ingress and local-path pods Pending.
- Wait for Calico to finish initializing and node to become Ready.
- Validate CoreDNS, ingress controller, and local-path provisioner.
- Run DNS test pod and record results.
- configs/kubernetes-versions.md
- runbooks/kubernetes-single-node-ops.md
- Re-ran
kubeadm init after reset.
- Calico applied; forced
calico-node to use API server IP with:
KUBERNETES_SERVICE_HOST=192.168.1.216KUBERNETES_SERVICE_PORT=6443
- Calico CNI install succeeded; node is Ready.
- CoreDNS pods running but not Ready yet (readiness failing, upstream DNS timeouts to 192.168.1.1:53).
- ingress-nginx and local-path provisioner applied; pods Pending (waiting on taint removal and/or network readiness).
- Control-plane taint removed.
- Node: Ready
- Calico: Running
- CoreDNS: Running, not Ready
- Ingress: Pending
- Local-path: Pending
- Confirm CoreDNS readiness; investigate upstream DNS reachability (192.168.1.1:53 timeouts from pod network).
- Verify ingress-nginx controller pods become Running after taint removal.
- Verify local-path provisioner becomes Running.
- Run DNS test pod and record results.
- Record final validation outputs in this change log.
- Observed widespread CrashLoopBackOff for kube-controller-manager, kube-scheduler, CoreDNS, ingress-nginx, local-path provisioner, and calico-kube-controllers; kube-apiserver/etcd running; kube-proxy running.
- In-cluster API service IP appears unreachable from host:
curl -k https://10.96.0.1:443/healthz timed out.
- CoreDNS logs show inability to reach upstream DNS (192.168.1.1:53) and repeated API watch failures.
- Calico CNI previously logged errors referencing
https://[192.168.1.216]:6443 during pod sandbox setup (no current config files contain bracketed address).
- Likely root cause: Pod CIDR
192.168.0.0/16 overlaps with node LAN 192.168.1.0/24, causing routing/service IP issues.
- Re-initialize the cluster with a non-overlapping pod CIDR (e.g., 10.244.0.0/16 or 172.16.0.0/16).
- Reset cluster state (
kubeadm reset), clear CNI dirs (/etc/cni/net.d, /var/lib/cni/), re-run kubeadm init with new CIDR, then re-apply Calico/ingress/local-path.
- Revalidate service IP reachability, CoreDNS readiness, and ingress/local-path health.