Game narration completed successfully, then the client frequently received a generic LLM error during the required item-state reconciliation phase.
- Application pods were healthy with no restarts or memory pressure.
item_state_evaluation failed at the former 15-second background timeout in 5 of 12 recent calls.- Background player metrics produced 233 additional 15-second provider timeouts over three hours.
- Inference coordination was process-local, so both application replicas could run retryable background requests concurrently.
- A live item reconciliation later required 41.46 seconds and succeeded when given a 45-second window.
- Separate failures returned immediate HTTP 502 and 503 responses across persona, narration, and item phases; an immediate item retry was ineffective during that provider outage.
Final production image: sha-7d98e7d33d4b from merge commit 7d98e7d.
- Kept item-state reconciliation transactional and fail-closed.
- Added a dedicated 45-second primary item-state timeout and a 20-second retry timeout, bounded by the whole-turn deadline.
- Added typed provider-outage recovery: HTTP 429 and 5xx item failures retry after 2, 8, and 20 seconds when turn budget remains. Transport timeout/connect failures receive one retry after 2 seconds.
- Added a PostgreSQL transaction-scoped advisory slot so retryable background generation is serialized across replicas.
- Deferred retryable background generation while a turn is queued/running and for 15 seconds after turn activity.
- Separated foreground persona-memory query embeddings from background memory-index embeddings; only background indexing uses the distributed slot.
- Added a dedicated 30-second player-metric timeout.
- Changed metric deferrals so they do not count as provider attempts; genuine failures use exponential backoff capped at 15 minutes.
Passive-effect evaluation remains outside the distributed retry slot because it is launched as part of immediate post-turn completion and was not a retry-storm source.
cargo fmt --all passed.- Final
cargo test --workspace passed: 322 backend and 5 frontend tests.
- Kubernetes server-side dry-run passed.
- PostgreSQL advisory-lock, foreground-activity, and backoff expressions were validated in a rolled-back transaction.
- A real production turn completed with narration in 5.27 seconds and item reconciliation in 41.46 seconds; the item delta was committed successfully instead of hitting the former timeout.
- Another real production turn completed item reconciliation in 2.92 seconds.
- A behavior metric that had repeatedly timed out completed in 25.53 seconds under its new 30-second window; a queued state metric deferred without increasing attempts and later completed in 14.18 seconds.
- Final rolling deployment completed with 2/2 replicas ready and zero restarts.
/healthz returned ok; /readyz reported PostgreSQL and Qdrant ready.
Revert merge commit 7d98e7d on main and push it through the normal EventHorizon deployment workflow. Do not mutate pod environment variables directly.