Date: 2026-02-06
Host: drg-k8s
/mnt/fastdata and /mnt/bigdata are configured as systemd automounts (NFSv4 + Kerberos sec=krb5p) via /etc/fstab.
Mount attempts were failing with:
mount.nfs4: Operation not permitted ...Root cause on the client was missing NFS Kerberos prerequisites:
rpc.gssd was not running.rpc_pipefs.target / rpc_pipefs mount was present, which prevented rpc-gssd.service from starting.After fixing those prerequisites, the mounts succeed. Note that root may still see Permission denied due to root_squash and server-side permissions; access should be tested as the intended non-root identities.
rpc_pipefs is mounted and available as a targetAdded systemd units:
/etc/systemd/system/rpc_pipefs.target/etc/systemd/system/var-lib-nfs-rpc_pipefs.mountEnabled the mount unit so it is active at boot.
Updated:
/etc/default/nfs-common: set NEED_GSSD=yesAdded and enabled:
/etc/systemd/system/nfs-kerberos-client.serviceThis unit is WantedBy=nfs-client.target and Requires=:
rpc_pipefs.targetgssproxy.servicerpc-gssd.serviceA temporary attempt was made to point gssproxy at an alternate keytab; it was reverted. No persistent keytab changes were kept on this client.
mnt-fastdata.automount and mnt-bigdata.automount active.sudo -u ax_julian ls /mnt/fastdata and sudo -u ax_julian ls /mnt/bigdata succeeded.If root access is required, review server export options (root_squash), directory permissions/ACLs, and how Kerberos principals are mapped to local identities on the server.
ax_julian can create and write files under /mnt/fastdata/k8s-pvcs.root cannot write file contents (create via touch may succeed, but writing data fails).k8s-nfs-access (UID 1240000012) cannot write.Most Kubernetes PVC workflows (including dynamic provisioners and many containers that run as root or arbitrary UIDs without Kerberos tickets) will fail to write unless the server can map the client credential to a POSIX UID/GID that has write permissions.
Given sec=krb5p and the server currently rejecting sec=sys, this requires a server-side fix to principal-to-UID/GID mapping (NFSv4 name->id) for the identity used by nodes/pods.
Created PVC root directories:
/mnt/fastdata/k8s-pvcs/mnt/bigdata/k8s-pvcsMode/ACLs are permissive (rwx for user/group/other with sticky bit), but Kerberos authorization still blocks non-authorized principals.
k8s-nfs-access Keytab For NFS ClientDate: 2026-02-06
Confirmed service keytabs exist on this node:
/etc/krb5.keytab.d/k8s-nfs-access.keytab/etc/krb5.keytabs/k8s-nfs-access.keytabTried switching NFS client Kerberos keytab (gssproxy + gssd) from the host keytab to k8s-nfs-access.
Result: Mounts failed with access denied by server and rpc.gssd logged:
no usable keytab entry found ... for connection with host kryptovisor.auth.home.lanConclusion: rpc.gssd needs a host-style principal in the keytab to establish machine credentials for the NFS server connection. The standalone k8s-nfs-access@REALM keytab cannot replace the host keytab for mounts.
Rollback: reverted configs to use /etc/krb5.keytab.
As of 2026-02-06T11:02:50Z, non-root access began failing with "Stale file handle" and/or "Permission denied" on directory listings. Mounts still show as mounted on the client.
Client return codes (from this node):
ls rc=2ls rc=2This points back to a server-side export/filesystem and/or NFSv4 identity-mapping issue.