At 2026-10-10 01:50 UTC, production AutoCoder rollout candidate 91e4d950a23ce24b3a7bbca116972982644f4dd0-4449-1 was Running but not ready. PostgreSQL startup migration PID 1036266 waited for AccessExclusiveLock on codex_worker_tasks behind read PID 1035774.
The blocking connection had client IP 10.244.238.34, backend start 2026-10-10 01:22:25.002858+00, transaction start 2026-10-10 01:48:44.489133+00, and was stuck in ClientWrite for a SELECT task-board preview. That IP belonged to the removed predecessor engine pod; a fresh cluster-wide pod snapshot found no pod at that IP. The new engine pod was autocoder-engine-5f954749f4-rrsqh, UID 54d36f41-655c-423d-ace3-75c363044ce2, IP 10.244.238.1.
An initial conditional cancellation matched zero rows because the query began with a newline. The corrected query predicate retained exact PID, client address, backend start, transaction start, active ClientWrite read and observed blocking relationship. pg_cancel_backend returned true at 01:51 UTC, but fresh evidence showed that dead-client ClientWrite and the blocking lock remained. The administrator then conditionally terminated only that same orphaned read connection, with a five-second termination wait. No live worker connections, task states, reservations or persistent data were changed. Verify connection disappearance, migration completion and engine health separately.
The assessment generation dispatched at 01:40:35 UTC remained separately in flight, with its reservation held; the backend outage does not prove a provider outcome.
Startup migration initially encountered deadlocks with worker claims, rolled back and retried through container restarts. Once migration passed, synchronous lifecycle reconciliation ran without lock blockers; no startup probe existed, so liveness terminated the backend after approximately93 seconds. A concurrent rollback selected the previously healthy digest sha256:3385d2be52418cf740b3909b5213d9cbd297eab5ab0df3c2698080f95e3c6cf9, which also exceeded that startup window.
At02:02 UTC the external administrator applied a source-derived backend startup probe: GET /healthz, port8080, period10seconds, timeout3seconds, failure threshold60 (ten-minute allowance). Existing readiness/liveness and image were preserved. Source is k8s/05-deployment-service-ingress.yaml and its repository template in branch codex/assessment-exclusive-reroute-20261010; integration remains pending. The live JSON patch guarded deployment UID cc1ee57e-db24-4c04-b39a-c516465fad8b, current resource version and image. Restricted task evidence contains the patch receipt. No workers were drained. Acceptance: verify startup completes within this window and normal health returns; not yet proven.
The separately dispatched assessment failed with provider HTTP504 at01:51:59 UTC. Its uncertain $1.50 reservation remains held; three earlier assessments were saved. No paid retry or reservation release occurred.
At02:04:28 UTC the repaired pod completed startup reconciliation and started serving. Startup took approximately107seconds, exceeding the former liveness window but within the bounded startup allowance. Pod autocoder-engine-57d4b45d87-spk4x became Ready with zero restarts; direct service /healthz returned HTTP200 and JSON "ok". No orphan startup queries remained. This verifies platform restoration, not completion of assessment or EventHorizon feature delivery.
A subsequent rollout at02:07:52 UTC preserved the startup probe but again waited for an orphaned predecessor read. PID1036514 had client IP10.244.238.43, backend start 2026-10-10 02:02:41.185306+00, transaction start 2026-10-10 02:07:51.561067+00, and active ClientWrite task-board SELECT. That start belonged to the now-removed repaired predecessor pod. The only remaining pod record with that IP was an image-prune job completed onOctober8, before this connection existed; it was not a live owner. New backend pod autocoder-engine-5db4d74f46-c4kpd, UID f1677cc7-85d0-413e-a0df-3c15fe656fc0, IP10.244.238.24 had migration PID1036617 blocked exactly by1036514.
At approximately02:10 UTC a guarded termination matched only that PID, IP, exact backend/transaction starts, active ClientWrite SELECT and fresh blocking relationship; pg_terminate_backend(...,5000) returned true. Migration then progressed into normal startup reconciliation. No task, worker, reservation or stored-data mutation occurred. Source prevention is being prepared: bounded server-side TCP disconnect detection and active-query connection checks, preserving ordinary long-running queries. A startup probe alone cannot release orphaned database locks. Latest rollout acceptance remains pending.
At approximately02:13 UTC the latest candidate 91e4d950a23ce24b3a7bbca116972982644f4dd0-4449-1, backend digest sha256:cef70b397359dea7a45411fcfe5bc8c75ea573d60a1fb7e487631385a16e63d2, became Ready with zero restarts and /healthz returned HTTP200 JSON "ok". The source startup probe and shared bounded PostgreSQL connection options are committed on repair branch HEAD 09f2489986803083ef20853d9f60a83889e38392; worker database/binary validation and development integration are in progress. The connection prevention change is not deployed yet.