Date: 2026-08-03
Service: EventHorizon
Namespace: event-horizon
- Game chat repeatedly reported that the turn model was unavailable and retried without consuming input.
- Quick-action UI repeatedly showed “Suggestions unavailable — Retry.”
- The affected session remained at the same turn while durable jobs reached their retry cap.
The configured basic OpenAI-compatible credential and provider transport were healthy: model requests returned HTTP success. Two structured-output boundaries caused the visible failures:
- Quick-action suggestions and persona correction requested JSON in prose but did not use the provider-native JSON-object response constraint.
- Persona output could be valid JSON yet fail the private-agenda protocol. The recurring provider response selected
status=completed while also returning a conditional field that is not applicable to completed status. The former generic correction prompt did not expose the rejection reason, and the provider repeated the same conflict.
The quick-action optional-inference gate also returned 503 while a foreground turn was active. This was expected prioritization, but repeated foreground failures made it appear permanent in the browser.
- Added an explicit typed JSON-object response constraint for strict non-tool calls, including quick-action suggestions and the one persona correction request.
- Preserved strict initial agenda validation and added typed structural rejection reasons without logging raw private model output.
- After the single constrained correction only, the selected structured agenda status deterministically removes conditional fields that do not apply to that status. Action schema, source reference, current objective and milestone, required status data, remaining-action budget, and player-action conditions still fail closed.
- Automatic scene-background generation remains disabled through repository-managed production configuration. Item and effect icon enrichment remains enabled.
- Restored the basic LLM credential through the managed Gitea/Kubernetes secret path; no credential material is recorded here.
Final merge: 164e9a8e96b96cc928bc4014b5c50ff90c108822
Immutable image: gitea.app.lan/eventhorizon/eventhorizon:sha-164e9a8e96b9
Deployment verification:
- 2/2 replicas ready and available.
- Both new pods reported zero restarts.
/healthz returned HTTP 200./readyz returned HTTP 200 with PostgreSQL and Qdrant ready.- Startup telemetry confirmed
scene_background_generation_enabled=false.
¶ Recovery and validation
The original failed turn job was not replayed because completed newer jobs had already superseded it. After the final image was ready, only the latest failed job for the still-current session turn was requeued in a guarded transaction. The guard required:
- failed status,
- the exact expected turn count matching the current session,
- and no newer job for the session.
The recovered job completed with retry count 0 and advanced the affected session atomically from turn 8 to turn 9. Final-image telemetry recorded successful persona decision and narration phases.
A bounded quick-action verification returned HTTP 200 with exactly four labels and four typed targets. Final-image telemetry recorded successful quick_action_suggestions calls.
Repository verification passed:
- 407 backend tests,
- 11 frontend tests,
- Rust formatting,
- requirements JSONL parsing and source contracts.
A follow-up at 22:36 UTC found that provider and parser success did not explain the browser failures. Production logs showed each automatic refresh receiving HTTP 503 at approximately 0, 0.75, and 2.75 seconds after a turn. Every request that passed the inference gate succeeded.
The distributed gate's 15-second recent-activity condition was applied to every turn-job status, including already completed jobs. The browser therefore exhausted its bounded retry schedule while the gate was guaranteed to remain closed.
Remediation merge: 183f31b6317c88b589e1d3d8cecbfc53bf94b62e
Immutable image: gitea.app.lan/eventhorizon/eventhorizon:sha-183f31b6317c
The gate now behaves as follows:
- queued and reconciling jobs block optional inference;
- running jobs block while their lease is live or their activity is recent;
- completed, failed, superseded, and stale running jobs do not create a false foreground window;
- suggestion, interpretation, and rendering remain optional, provider-cooldown-aware, and serialized cluster-wide;
- stale-session-aware browser retries now span 10.75 seconds for genuine optional-lane contention.
Verification:
- 2/2 replicas ready, zero restarts, health and readiness HTTP 200;
- CI and local suites: 407 backend tests and 11 frontend tests;
- a read-only state matrix confirmed recent completed and failed jobs do not block, while queued, reconciling, live running, and recent running jobs do;
- five consecutive production suggestion requests returned HTTP 200 with exactly four labels and four typed targets; generated text was not recorded.