The operator requested repair of AutoCoder assessment and delivery blockers. Production candidate e74a53989683a8e8c4ae0026f60768ee8ef4e612-4492-1 passed validation and TEST, but promotion runs 4494–4497 failed after healthy rollouts and guarded restoration of the previous production candidate. Failed pipeline and accounting records remain unchanged.
At 2026-10-10 05:18:33 UTC the controller kernel reported a memory cgroup OOM in the exact autocoder-postgres container under its 1 GiB limit. PostgreSQL logged a backend killed by signal 9 at 05:18:34.260, terminated all remaining sessions, and recovered by 05:18:35.498. The production report returned a closed database connection at 05:18:34.396. The postmaster survived, so container restart count did not reveal the session loss. Both idle timeout settings were zero. Earlier reporting timeout causes remain unproven.
The engine held an immutable database client and its unconditional health response continued reporting Ready. The source repair makes the existing health endpoint reject an absent or closed required client, so existing liveness restarts reconnect it. It also bounds identical-byte production receipt replay, separates durable observation acknowledgment from subsequent UAT reconciliation, preserves execution ownership, and retains a verified healthy rollout only for ambiguous transport exhaustion. Explicit rejections still use guarded rollback. Worker validation is pending for the complete source repair.
The existing controller has approximately 44 GiB RAM, 34 GiB available and no MemoryPressure. Repository commit d31ebbda changes only PostgreSQL memory request 512 MiB to 1 GiB and limit 1 GiB to 2 GiB; CPU, image, identity, permissions, PVC and stored data remain unchanged. This is an allocation within existing physical capacity and adds no paid or physical capacity. A UID/resource-version guarded production deployment patch and database readiness verification are pending. A database rollout briefly disconnects clients; after database readiness, a targeted restart of the existing engine deployment will reconnect the old version until the health source repair is delivered. No worker task was running at the preflight check, and assessment remains explicitly paused at 4 of 34.
Rollback is a guarded restoration of the previous memory values if the new allocation cannot schedule. Restoring the lower limit does not resolve the observed OOM; unresolved memory pressure remains an engine blocker. Preserve all original task, issue, provider charge and deployment evidence. No broad process termination, new credentials, authorization changes, paid fallback or protected main-account allocation is authorized by this change.
The UID/resource-version guarded database patch was applied at 05:27:30.738314 UTC. PostgreSQL is Ready in pod autocoder-postgres-799694fb94-zzfhv with request 1 GiB and limit 2 GiB; image, CPU, deployment identity and PVC were preserved. Read-back confirmed all 34 campaign members remain: 4 applied and 30 pending, the original pause is effective, and the account ledger remains 3,072,751 USD micros in recorded costs and reservations (not an invoice amount).
After database readiness, a UID/resource-version/image guarded engine annotation triggered a normal restart of the previous production image. Engine pod autocoder-engine-5c7764ff59-kxfrr became Ready; PostgreSQL showed three live sessions from its exact IP 10.244.238.8. External health returned HTTP success with JSON "ok" and the served frontend revision remained 8d8c346299c26d0ebf294982e792d7354105f574. This restores the old application; the assessment fix is not yet delivered. There were no running worker tasks at mutation preflight, and no provider dispatch, paid fallback or protected-account allocation was started.
All required focused worker checks passed: 17 reporting tests; full candidate script contracts; locked all-target Rust compilation with native-test-support; 9 receipt API unit tests; an isolated database test of exact execution ownership, concurrent duplicate rollback, late success rejection and preserved observation on UAT failure; and an isolated health test terminating only its own fixture connection. Code commit 3fcb7a369089581941547ada288e0a7684325fda contains the receipt/health repair; d31ebbda480e335923bd9504780e95bbb7aeb835 contains the memory allocation. Independent source review passed. Branch CI and repaired candidate delivery remain pending.
The supervising administrator independently saved the bounded kernel OOM excerpt in restricted task evidence. Existing failed runs 4494–4497 and recovery count 3 remain unchanged. The earlier timeouts are not attributed to OOM without matching evidence.
Promotion4502 succeeded for candidate 68261f00b6d46e01546c7f731ad72a6222048499-4500-1. At05:54UTC external /healthz returned HTTP200 JSONok with exact source header68261f00; frontend pkg/build-revision.txt returned that same revision. Live PostgreSQL Deployment was Ready with its repository-managed2Gi memory limit. Backend selected immutable imagesha256:cac1f9afc570b393d733ba440c3461bce0fc084b356fecfeb96d730eb02b5c6a. Source closure detection is covered by the previously recorded isolated connection-termination test; no production database termination was performed for acceptance.
Separate full worker publication4503 succeeded from main165d1ceca2643b07bb45e8ab76a3983ce9d63024, whose tree matches the candidate source. Selected worker imagesha256:0b9367978bf4735e6adb550fd51628f41bdb093fbb88478e83b11d739c806b87 was verified Ready ondrg-k8s-worker-01 before the supported assessment resume. CPU recovery and temporary runner2/3 parking are recorded onchanges/2026-10-10-assessment-worker-cpu-recovery; restore them when the campaign finishes or requires intervention.
The repaired frozen-snapshot comparison held three materially changed members while timestamp-only mirror updates no longer blocked dispatch. Five classifications were applied by06:20UTC. Provider HTTP504 for member182, followed by HTTP200 for member165, demonstrates a remaining intermittent provider failure rather than loss of backend database health. Original failures and uncertain charge holds remain immutable and reserved. The bounded second attempt for original member195 was exhausted without a classification; other eligible members continue under the unchanged USD100 ceiling, no paid fallback and original23:41:15UTC expiry. At06:34UTC recorded/reserved account ledger was6094347USDmicros, not provider-billed spend. Classification acceptance and the shared notification bell remain unfinished.
The separate retained-verifier patch has not been deployed. Candidate4506 retained a failed stop-boundary regression, corrected and reviewed inPR9632. Candidate4509 retained two isolated database fixture failures; fixture repair and fresh CI-equivalent validation are in progress. These results do not change the serving source or indicate another production rollout.
The earlier standalone retained-review pass is retained as historical evidence but does not establish compilation of the newly copied source: source files were copied with preserved modification times into a reused Cargo target directory. It is not counted as acceptance. Candidate 4509 failed and remains failed. Fresh package compilation reproduced the fixture failures and then exposed a real textual timestamp parameter binding defect in historical checkpoint attestation. The current bounded repair fixes the historical admission/worker-lease fixtures, newline SQL syntax, dedicated-account capacity fixture, and casts the textual timestamp parameter before PostgreSQL timestamp conversion. Independent review preserves the protected-main, ownership and financial guards. Worker validation now explicitly forces tracked-source modification times forward, records all four compiled test artifacts as fresh=false and binds their hashes to the full tracked-source manifest. Production still serves source 68261f00b6d46e01546c7f731ad72a6222048499; no verifier recovery capability has been deployed or exercised on the historical case yet.