Last updated: 2026-01-28
This runbook addresses failures where Calico init containers cannot fetch service account tokens or mount projected serviceaccount volumes, resulting in:
NetworkPluginNotReady: cni plugin not initializedInit:Error or CrashLoopBackOfffailed to fetch token: ... serviceaccounts "<name>" is forbidden: User "system:node:<node>" cannot create resource "serviceaccounts/token"failed to sync configmap cache: timed out waiting for the conditionNotReadyInit:ErrorPendingsudo ss -lntp | grep -E '6443|2379|2381'
sudo curl -s --cacert /etc/kubernetes/pki/etcd/ca.crt \
--cert /etc/kubernetes/pki/apiserver-etcd-client.crt \
--key /etc/kubernetes/pki/apiserver-etcd-client.key \
https://127.0.0.1:2379/health
sudo KUBECONFIG=/etc/kubernetes/admin.conf kubectl -n kube-system get pods -l k8s-app=kube-proxy
sudo KUBECONFIG=/etc/kubernetes/admin.conf kubectl -n kube-system logs <calico-node-pod> -c install-cni --tail=200
Cause: apiserver restarts, leading to token timeouts/GOAWAY.
Fix:
sudo systemctl restart kubelet
Cause: kube-proxy not deployed or stuck; service IPs (10.96.0.0/12) not reachable.
Fix:
sudo kubeadm init phase addon kube-proxy
sudo KUBECONFIG=/etc/kubernetes/admin.conf kubectl -n kube-system delete pod -l k8s-app=kube-proxy
Cause: node identity not yet authorized or inconsistent.
Fix:
kubeadm reset and ensure hostname is stable.Check Node:
sudo KUBECONFIG=/etc/kubernetes/admin.conf kubectl get nodes -o wide
Cause: old CNI artifacts from previous install.
Fix:
sudo find /etc/cni/net.d -mindepth 1 -delete
sudo find /var/lib/cni -mindepth 1 -delete
sudo systemctl restart kubelet
sudo KUBECONFIG=/etc/kubernetes/admin.conf kubectl -n kube-system delete pod -l k8s-app=calico-node
kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.31.3/manifests/calico.yaml
kubectl -n kube-system get pods -o widekubectl -n kube-system describe pod <calico-node>kubectl -n kube-system logs <calico-node> -c install-cnijournalctl -u kubelet -n 200 --no-pagerCause: UFW default deny (routed) prevents traffic to service IPs (e.g., 10.96.0.1:443), so Calico token requests time out.
Fix (example for enp1s0, Calico cali+, and tunl0):
sudo sed -i 's/^DEFAULT_FORWARD_POLICY=.*/DEFAULT_FORWARD_POLICY="ACCEPT"/' /etc/default/ufw
sudo ufw allow in on enp1s0
sudo ufw allow out on enp1s0
sudo ufw allow in on tunl0
sudo ufw allow out on tunl0
sudo ufw allow in on cali+
sudo ufw allow out on cali+
sudo ufw route allow in on enp1s0 out on cali+
sudo ufw route allow in on cali+ out on enp1s0
sudo ufw route allow in on cali+ out on cali+
sudo ufw reload