Last verified: 2026-07-22
event-horizonevent-horizon-postgresevent-horizon-postgresevent-horizon-postgres-backupevent-horizon-postgres-backups/mnt/fastdata/k8s-pvcs/event-horizon/postgres-backupsSet the cluster context explicitly:
export KUBECONFIG=/home/ax_julian/.kube/config
kubectl config current-context
kubectl cluster-info
Run from the Kubernetes server. Replace the sample filename with the selected dump:
backup_name=eventhorizon-YYYYMMDDTHHMMSSZ.dump
backup_dir=/mnt/fastdata/k8s-pvcs/event-horizon/postgres-backups
verify_copy=/home/ax_julian/temp/eventhorizon-backup-verify.dump
cd "$backup_dir"
sudo -n sha256sum -c "$backup_name.sha256"
sudo -n install -m 600 -o ax_julian -g ax_julian "$backup_name" "$verify_copy"
pgpod="$(kubectl -n event-horizon get pod -l app=event-horizon-postgres -o jsonpath='{.items[0].metadata.name}')"
kubectl -n event-horizon cp "$verify_copy" "$pgpod:/tmp/eventhorizon-backup-verify.dump"
kubectl -n event-horizon exec "$pgpod" -- pg_restore --list /tmp/eventhorizon-backup-verify.dump >/dev/null
kubectl -n event-horizon exec "$pgpod" -- rm -f /tmp/eventhorizon-backup-verify.dump
rm -f "$verify_copy"
A usable backup must have a matching checksum and a successful restore-list check.
stamp="$(date -u +%Y%m%dt%H%M%Sz)"
job="event-horizon-postgres-backup-manual-$stamp"
kubectl -n event-horizon create job --from=cronjob/event-horizon-postgres-backup "$job"
kubectl -n event-horizon wait --for=condition=complete "job/$job" --timeout=900s
kubectl -n event-horizon logs "job/$job"
kubectl -n event-horizon delete "job/$job"
Verify the new dump and checksum on the retained NFS path before continuing.
Schedule a maintenance window. Record current counts and create a fresh on-demand backup before restoring older data.
pgpod="$(kubectl -n event-horizon get pod -l app=event-horizon-postgres -o jsonpath='{.items[0].metadata.name}')"
kubectl -n event-horizon exec "$pgpod" -- psql -U eventhorizon -d eventhorizon -Atc \
"SELECT count(*) FROM game_sessions; SELECT count(*) FROM chat_messages; SELECT coalesce(sum(turn_count),0) FROM game_sessions;"
kubectl -n event-horizon scale deployment/event-horizon --replicas=0
kubectl -n event-horizon rollout status deployment/event-horizon --timeout=180s
Copy the already verified dump into the database pod and restore it:
backup=/mnt/fastdata/k8s-pvcs/event-horizon/postgres-backups/eventhorizon-YYYYMMDDTHHMMSSZ.dump
restore_copy=/home/ax_julian/temp/eventhorizon-restore.dump
sudo -n install -m 600 -o ax_julian -g ax_julian "$backup" "$restore_copy"
pgpod="$(kubectl -n event-horizon get pod -l app=event-horizon-postgres -o jsonpath='{.items[0].metadata.name}')"
kubectl -n event-horizon cp "$restore_copy" "$pgpod:/tmp/eventhorizon-restore.dump"
rm -f "$restore_copy"
kubectl -n event-horizon exec "$pgpod" -- pg_restore \
--exit-on-error --clean --if-exists --no-owner \
-U eventhorizon -d eventhorizon /tmp/eventhorizon-restore.dump
kubectl -n event-horizon exec "$pgpod" -- rm -f /tmp/eventhorizon-restore.dump
Validate restored counts and schema before reopening the application:
kubectl -n event-horizon exec "$pgpod" -- psql -v ON_ERROR_STOP=1 -U eventhorizon -d eventhorizon -Atc \
"SELECT count(*) FROM game_sessions; SELECT count(*) FROM chat_messages; SELECT coalesce(sum(turn_count),0) FROM game_sessions; SELECT pg_size_pretty(pg_database_size(current_database()));"
kubectl -n event-horizon scale deployment/event-horizon --replicas=2
kubectl -n event-horizon rollout status deployment/event-horizon --timeout=600s
curl -fsS https://eventhorizon.app.lan/healthz
curl -fsS https://eventhorizon.app.lan/readyz
Review both application pod logs for migration or database errors.
If validation fails, keep the application scaled to zero and restore the fresh pre-restore backup using the same procedure. Do not delete the failed database volume or any retained safety dump until the rollback has been verified.
pg_restore --list succeeds.Retain.event-horizon-secrets; do not place them in commands or documentation.