EventHorizon game turns were vulnerable to prolonged MaaS inference stalls. Browser streams could end without a reply, and in-flight work was not recoverable after process replacement. PostgreSQL also used pod-local ephemeral storage and had no automated backup.
Observed before the change:
- Perception calls failed at the 30-second deadline in 4 of 7 recent samples.
- Persona calls failed at the 45-second deadline in 1 of 8 recent samples.
- Two complete turns reached the 150-second deadline without committing.
- Optional evaluator and quick-action calls consumed 39-73 seconds.
- Direct 20-second probes to two MaaS models returned no response bytes, indicating a provider-level incident rather than EventHorizon resource pressure.
- Persist each game-turn request before inference with a stable client request UUID.
- Claim work transactionally with a lease and expected session turn count; recover queued or expired work after pod replacement.
- Return the same job state/result for reconnects and duplicate request IDs.
- Reserve commit time inside the 150-second turn deadline.
- Bound rule checks, perception reviews, background evaluators, and quick-action inference separately.
- Suppress optional inference while foreground inference is active or the provider circuit is open.
- Preserve deterministic accepted actions when semantic checks are unavailable.
- Commit deterministic player-visible narration from authoritative actions if narrator inference fails.
- Record typed inference failure kind and HTTP status in usage telemetry.
- Run two application replicas with rolling-update availability and a PodDisruptionBudget.
Repository commits:
- Feature:
1be2c1b (Add resilient durable turn execution)
- Main merge/deployed image:
9c531b3 / sha-9c531b388c8a
PostgreSQL was migrated from emptyDir to bound PVC event-horizon-postgres using storage class local-path, 10 GiB, RWO.
Safety points retained on NFS:
pre-migration-20260722T004022Z.dumppre-pvc-final-20260722T005954Z.dump
A daily backup CronJob, event-horizon-postgres-backup, runs at 02:15 UTC. It writes atomic custom-format dumps to retained NFS storage, validates each dump with pg_restore --list, writes a SHA-256 checksum, and removes scheduled dump files older than 14 days.
- Pre-change and final dumps passed checksum and restore-list validation.
- All migrations, including job and failure telemetry schema, passed against a clean PostgreSQL 16 database.
- Workspace tests: 274 passed, 0 failed.
- Kubernetes server-side manifest validation passed.
- Rollout: 2/2 application replicas ready, zero restarts, PDB allows one disruption.
- Persistent claims: database and backup PVCs bound.
- Data after restore and rollout: 5 sessions, 1,667 chat messages, 482 total turns, database size 11 MB.
- Schema:
game_turn_jobs present; failure_kind and http_status telemetry columns present.
- Public checks: five
/healthz and five /readyz requests returned HTTP 200.
- Unknown job status lookup returned the expected HTTP 404 without creating work.
No production game turn was submitted as part of verification.