Status: Resolved
Seven deployment definitions failed on the dev branch after the shared OpenShift deployment credential expired: TheWell Backend, App Firewall, Document Intelligence, KEK App, Postgres, Qdrant, and Terminology. TheWell DevPool also retained separate terminal-accounting and unsupported-source defects.
- Secret Azure variable
OCP_DEPLOYMENT_TOKEN_DEV in variable group 106 contained an expired service-account token. The affected deployment runs all stopped at Login to OpenShift.
- After the credential was renewed, Postgres recovery run 24756 exposed a second independent defect. The namespace had 1275m of its 1500m CPU-limit quota allocated, while backend and statistics initialization stages ran concurrently and each requested a 250m CPU limit. OpenShift rejected both Job pods, so the pipeline could not progress.
- TheWell backend runs 24750 and 24751 overlapped because one was triggered automatically and one manually. Both used the same fixed copy pod, causing run 24750 to fail when the other run deleted it.
- Issued a one-year bound token for
system:serviceaccount:d03849-skan-ag-dev:skan-deployer, verified its required permissions, and replaced protected Azure variable OCP_DEPLOYMENT_TOKEN_DEV without exposing its value.
- Postgres PR 5665 recorded the requirement that transient initialization fit the existing quota and run sequentially.
- Postgres PR 5666 serialized backend, statistics, and Document Intelligence initialization stages and reduced each transient Job CPU limit from 250m to 200m. No quota was increased and no long-running workload was reduced.
- Postgres PR 5667 recorded successful live verification.
- Infrastructure recovery runs were deliberately queued one at a time to avoid another shared-namespace overlap.
Latest successful runs:
- TheWell Backend: 24755
- TheWell DevPool Import: 24754
- Postgres Database: 24758
- Qdrant Database: 24759
- KeyEncryptionKeyApp: 24760
- Document Intelligence Service: 24761
- App Firewall: 24762
- Terminology Plugin: 24763
Postgres run 24758 completed all three database initialization stages sequentially within the unchanged 1500m namespace quota. Every listed recovery run completed successfully, and no affected definition had an active run at closure.
Historical failed and canceled runs remain in Azure DevOps as audit evidence. They do not describe current pipeline health. The durable follow-ups are renewable workload identity for Microsoft-hosted deployments and an exclusive deployment lock for workflows that mutate shared PVCs or fixed helper pods.