The operator reported two negatively rated production answers on 2026-09-30: a car-booking answer recommending another company's platform, and a confusing ICT-documentation answer. Both displayed zero sources without an error. Requested investigation begins by locating those ratings. Investigation only; no application deployment, production configuration change, permission expansion, or issue mutation was performed.
Read-only inspection selected OpenShift API https://api.basel.kvant.cloud:6443, namespace d03849-skan-ag. Existing local read credential authenticated and read workloads/logs; its pods/exec check returned no. The older administrator kubeconfig was rejected for missing/invalid authentication. Credential values are not recorded.
Production nginx access logs contain exactly three POST /api/chat/analytics:analyze submissions in the inspected UTC-day log window, all HTTP 202: 07:42:21, 07:42:38, 07:43:17 UTC (09:42:21, 09:42:38, 09:43:17 Swiss time). This proves accepted rating-analysis submissions, not their rating direction, row IDs, or transcript subjects. The operator answered "probably, yes" when asked whether the affected questions occurred around 09:40–09:43 Swiss time.
Backend logs adjacent to these submissions:
| Preparation time UTC | Messages | Last user characters | Logged mode | Requested datapool count | Visible sources |
|---|---|---|---|---|---|
| 07:40:54 | 2 | 43 | lite | 5 | 0 |
| 07:42:26 | 4 | 7 | lite | 5 | 0 |
| 07:42:58 | 2 | 33 | lite | 2 | 0 |
Each uses [chat-answer-routing] route=assistant-tools, with webResultCount=0. The log times align with three chat requests ending at 07:41:02, 07:42:32, 07:43:06 (HTTP 200), followed by the three rating submissions. No document retrieval is performed in the lite preparation branch. These are strong temporal candidates for the reported two conversations and a follow-up; subject attribution is not confirmed.
Azure DevOps read access as ax_julian was verified. Latest successful main run of backend definition 339 was 25940, completed 2026-09-24 09:23:46 UTC, source 636a974fa958bbafd6cfcd3f0516b9ecb955fa62. Files were read directly from Azure DevOps at that commit because local origin/main was stale. This is provider deployment evidence, not independently measured runtime DLL provenance.
At that source, ChatController.Analytics.cs calls StoreRatedChatAsync, then AnalyzeAndStoreAsync, before returning HTTP 202. Thus accepted submissions support rating persistence under that deployed code path. Exact database rows still require reading the supported API or database.
At that source, ChatController.Rag.cs gates document-search scope resolution on ShouldRunDocumentSearch(toolPlan) and otherwise has an empty scope. Its lite branch returns empty search results and empty visible sources. Missing tool selection or an empty resolved scope can therefore produce successful unsupported answers instead of a retrieval error. The recorded selected datapool counts establish that the client supplied pools; they do not prove server-side accessible scope resolution or the planner's selected tool keys. Routing/tool selection is the leading hypothesis; the precise decision is not recovered.
Frontend production /version.json identifies 08ab48f-2026-09-24T08:21:37.308Z. Actual chat asset /assets/ChatApp-nbGHFMR4.js contains analytics:analyze, analytics/preference, and rated-message submission. It suppresses analytics submission exceptions. An initial observation of the entry asset alone missed this dynamically loaded chat code; that mistaken inference was corrected before this record. Do not treat the older build date as proof rating persistence is absent.
The supported ADM-only endpoints are GET /api/chat/analytics/negative-chats?from=2026-09-30&to=2026-09-30&limit=20 and GET /api/chat/analytics/negative-chats/{id}. Existing preserved administrator browser authentication no longer signs in to the production app. Azure CLI application-token acquisition returns AADSTS65001 (no consent for Azure CLI to that custom API); no consent or permissions were changed. The working cluster read credential cannot execute PostgreSQL queries. Investigation cannot currently retrieve exact negative-rated transcripts, subjects, row IDs or planner decision records.
Next required evidence is existing authorized APP_TheWell_ADM app access, or an administrator with already-authorized production PostgreSQL inspection access, to read the three candidate rated messages and their saved source/pool references. Then recover the matching tool-selection records and accessible scope, compare the stored answers with the car-booking and ICT source documents, and distinguish omitted search from empty authorized scope. No fix is verified or deployed.
The investigation-created browser session was closed and its temporary session-ID file removed; existing browser profiles and production data were preserved. No paid model requests, builds or pipelines were initiated. No subagents were used. Exact initial active-runtime timestamp was not captured; this was a single short investigation turn, far below the six-hour review checkpoint.
The operator subsequently requested “please resolve the search issues of TheWell prod”. This is the active external-administrator repair outcome under the host scoped repair exception. It authorizes this incident repair in production; the general production promotion stop remains in effect for unrelated features. No Gitea issues are being changed. A general DEV-to-PROD promotion is excluded.
Requirement REQ-0008 was updated and pushed independently (863aca5) before implementation. Reviewed DEV repair 72d35a8ae00c91c86737227a71a1d80b410df125 retains enabled document search in bounded planner candidates, strengthens semantic internal-information routing and evidence grounding in both answer paths, treats malformed planner decisions as recoverable errors, and adds privacy-safe routing logs. It does not bypass tool enablement or authenticated datapool permissions. Azure PR 6687 targets DEV; completion was requested and final integration verification is pending.
A separate production release branch starts from secured MAIN d6a111276d06a814f1fd9f1af69cafe11fde01a3, with only the requirement and search fix cherry-picked (f80c8af, 71a5e9d). Both exact codebases passed formatting and 57 focused tests, with zero failures, on the authorized worker runner Docker daemon. Production archive SHA-256 is 64a7f2b236c4d341f7ad056c6000d751345e990f9e4d1a6e3ee7de69357188c6; 46,006,041 bytes. TRX records and archive are bounded task files in /home/ax_julian/temp/thewell-search-worker-evidence-20260930. The temporary worker validation container was removed after retrieval. These tests are implementation evidence, not semantic production outcome verification.
The source review accepted the repair. Independent diagnostic review identified an after-buffer byte-limit gap, which was corrected with streaming byte/deadline limits and verified with over-limit, timeout and large-input cases. Live PostgreSQL has container postgres with configured POSTGRES_USER/PASSWORD; variable group 106 contains POSTGRES_BACKEND_DB. The diagnostic explicitly names that container. It runs fixed read-only SQL for 07:40–07:44 UTC, validates live schema and workload ownership, excludes session identifiers, and publishes only CMS-encrypted evidence to the existing administrator-held recipient key. Logs contain only controlled status, counts and ciphertext digest. No new cluster/API permissions or identities are being granted.
The existing production Azure definition 339 has already-authorized variable group 106. A temporary, revision-fenced YAML redirection to the manual read-only diagnostic is being prepared rather than authorizing a new pipeline identity. Its original definition must be preserved and restored after the run. The legacy MAIN deploy wipes existing application data outside document-storage; that unsafe route will not be invoked. Production application cutover will instead stage an immutable sibling release, preserve live appsettings and all existing binaries/data, fence the deployment UID and current command, and change only the application command/working directory with rollback to the preserved prior paths. No deployment has happened yet; exact rated subjects and semantic acceptance remain pending.
Runtime accounting: the active goal reports approximately 40 minutes parent runtime at this checkpoint; delegated review/delivery turns add runtime. Exact initial investigation and cumulative subagent durations are not fully captured, so an exact aggregate is unavailable. Meaningful source, review and worker-validation progress has occurred; no six-hour checkpoint or 48-hour stop is indicated by observed task duration. Scheduled pipeline waiting will be registered with the host monitor and excluded from active runtime.
DEV PR6687 completed at unchanged reviewed code head72d35a8; verified merge a806ad3e9067456bf16581f71fcaf004edfd65fc has the same tree. Existing Gitea worker check for this merge succeeded (run26, TheWell backend local build / build (push)).
Manual read-only diagnostic source e7cdc79cc7826aa78f3fcfda063487ad6b0ee115 was independently reviewed and committed/pushed. Existing engine build identity could read definition339 but its PUT was forbidden (403), with no mutation. The already-authenticated ax_julian administrator CLI successfully changed only the existing definition's YAML path at revision19→20. Original complete definition is in restricted task secret storage. It must be restored to azure-pipelines.yml after terminal collection, using the stored owned revision/process fence; no new resource authorizations were granted.
Exactly one diagnostic run was queued: Azure definition339 run26231, full source e7cdc79cc7826aa78f3fcfda063487ad6b0ee115, branch fix/prod-search-grounding-20260930. CLI returned the pipeline Run shape without sourceVersion; the queued build was reconciled by an exact branch/source match instead of being queued again. The production application has not been deployed or changed. Host pipeline monitor will own this wait and resume this task after terminal status. On resume: restore the temporary definition, inspect only the encrypted artifact in bounded memory, attribute saved ratings, then complete scoped release deployment and semantic outcome checks. Protected continuation/dispatch records are under /home/ax_julian/.codex/secrets/thewell-search-repair-20260930.
A renewed operator report of broken PROD search triggered continuation from the retained work. Read-only diagnostic Azure26231 succeeded at the exact expected source; definition339 was already restored to revision21 and its original YAML. The recipient-encrypted artifact was decrypted only in bounded process memory. It contains three negative ratings across two conversations: the initial ICT tool question at07:42:17 UTC, its AD-Sync follow-up at07:42:36 UTC, and company-vehicle booking at07:43:15 UTC. All saved answers have empty source arrays and document-ID arrays. ICT requests selected pools18,19,38,39,95; vehicle selected43,44. The vehicle answer asserted MoveYou and supplied a speculative URL; the ICT answers conflated SKAN with advertising concepts. These are confirmed saved unsupported answers, not merely time-matched candidates. Retained backend logs establish that the requests used the lite path without document retrieval; the precise original model selection remains unavailable. The diagnostic tool-usage window contains feedback events only and does not independently recover the planner decision.
PR6687 completion and DEV merge a806ad3e9067456bf16581f71fcaf004edfd65fc were independently rechecked. The tested production app archive at71a5e9d is unchanged. Delivery source ce0974f1c57d67fe99c2457894fb88d781324911 adds a manual exact-source release pipeline and immutable sibling deployment script on fix/prod-search-release-20260930. Review removed an unjustified barrier against definition352 after identifying it as Qdrant-Database; no Qdrant pipeline/configuration was changed. The release verifies provider/run identity, original Deployment UID/spec/path, archive and runtime DLL digests, preserved live configuration hashes, available PVC capacity, and readiness. It preserves old binaries/document storage, changes only command/working directory and its source annotation, and attempts ownership-fenced rollback on any post-switch exception. Bounded command success, output limit, deadline, nonzero failure, altered Deployment identity rejection, archive hash, and staged Git whitespace checks passed. Source validation is supporting evidence; production search behavior remains pending.
Existing administrator PAT changed only definition339 YAML at revision21→22 after checking no active competing backend runs. Azure run26233 is queued at exact delivery sourcece0974f1c57d67fe99c2457894fb88d781324911. The normal destructive MAIN deployment was not invoked. Artifact upload with the administrator PAT returned401; the existing authorized artifact service-principal route then uploaded the exact46,006,041-byte worker archive and receipt, with full readback validation in container66230535. No credentials, identities or permissions were created or expanded. SHA-256 remains64a7f2b236c4d341f7ad056c6000d751345e990f9e4d1a6e3ee7de69357188c6.
Host monitor watch59 owns the pipeline wait for thread01a0f318-abd6-7823-8320-da5c09e43551, with45-minute deadline. On terminal continuation, verify actual cutover/rollback evidence, restore definition339 from restricted definition339-before-release.json only while revision22 and the owned YAML still match, verify restored definition, and perform bounded semantic search acceptance. Release-dispatch.json and continuation.json under the existing restricted incident directory preserve exact state. The original user reply authorizing CHF5 total for this incident's paid verification was recovered from the actual retained conversation; no paid verification requests have been made. End-user authenticated E2E access remains unproven; component replay alone must not be described as authenticated E2E or full restoration. The general PROD promotion stop continues outside this repair.
Cleanup owner: current administrator incident continuation. Retain the46MB worker archive, protected encrypted incident evidence and original definition only until deployment, acceptance and rollback evidence are reconciled; do not delete production data or retained old binaries. The current continuation added approximately25minutes of active parent work; earlier cumulative runtime/subagent accounting is incomplete, and that gap is preserved. Pipeline waiting is excluded. Service restoration and full RCA closure are not claimed.
The scoped administrator repair is deployed and behaviorally verified in production. Azure Backend339 run26233 (build20260930.9) completed successfully at16:42:50 UTC, release revision ce0974f1c57d67fe99c2457894fb88d781324911. Application source 71a5e9d7eea90789c890cf22aa2c7cbaac63c3bb; verified archive SHA256 64a7f2b236c4d341f7ad056c6000d751345e990f9e4d1a6e3ee7de69357188c6; running DLL SHA256 1ced21f227add62e7b1a3783047a330e828dc3641a7eae546863bb5bd7463e38. The release script verified preserved live configuration, unchanged deployment UID, the immutable sibling release path, runtime binary digest and HTTP200 readiness. Previous binaries and documents remain preserved. Definition339 was restored to its original YAML/queue/triggers and verified at revision23; no further run of the completed pipeline is being polled.
Six bounded signed-in requests used the real production POST /api/chat route with the normal user's existing Microsoft session and GLM, not a component probe. No feedback or analytics was submitted. All returned HTTP200 and a completed stream without error:
| Case | Observed behavior |
|---|---|
| Company vehicle, unrestricted authorized scope | Corporate CarSharing documentation retrieved; original invented MoveYou platform absent. |
| Company vehicle, original ConSense workspace | Ten source excerpts from pools43/44; answer matches the reservation and business vehicle documents. |
| Original ICT AD-role question | Ten excerpts; SRMT/AD Role Groups evidence produces the internal role-management explanation instead of advertising. |
| Original AD-Sync follow-up, including the earlier incorrect assistant context | Ten excerpts from ICT pool19; Azure AD Multitenant Sync and IAM Notification evidence supports the synchronization answer. |
| Same company vehicle question in the empty test workspace | Zero sources; explicitly states the missing internal booking evidence and does not invent a system. |
| General Python list/set question with a workspace selected | Correct direct answer, zero sources and no document retrieval. |
Unauthenticated POST /api/chat returned401. Workspace-scoped sources stayed within authorized workspace pools; the unrestricted case intentionally admitted all user-authorized pools, consistent with REQ-0008. These checks prove the reported production behaviors under the exercised normal-user GLM path, not every model or future input.
The user's existing CHF5 aggregate verification authorization was preserved across continuation. Six application requests were submitted; no separate component probe inference was run. The conservative upper bound from the entire UTC-day UserChat application allocation after verification was CHF4.159366129032258, including concurrent unrelated activity. First-request attribution was unavailable, so this deliberately overstates verification's application allocation. These figures are application contract allocation, not provider invoice evidence. Verification model calls are now finished.
DEV code was reviewed and merged by PR6687 at a806ad3e9067456bf16581f71fcaf004edfd65fc, with57 focused worker tests passing. DEV is the configured agent/local-validation branch; persistent TEST delivery uses the separate authoritative TEST branch. At this checkpoint TEST remained e32211c05cac5d6c5656405bb8024893b44da5fd and did not yet contain this repair; narrow TEST delivery is the remaining task.
The precise original planner choice was not retained, so the exact historical selection decision remains an RCA evidence gap. Confirmed original lite routing skipped internal retrieval and produced unsupported company assertions. The repair preserves enabled document-search eligibility in the bounded candidate set, strengthens semantic selection and evidence grounding, and logs privacy-safe candidate/selection/routing metadata for recurrence diagnosis.
Bounded low-urgency follow-up: the empty-workspace answer's opening calls a concrete question too unspecific; improve that wording to name absent workspace evidence. This does not invalidate the restored retrieval or evidence-gap safety check. Broader production promotion, permissions, data deletion and unrelated work remain outside this repair.
Azure PR6691 integrated the production acceptance evidence into DEV at 9f5a023fb2c56d8c3bfeeab76f0390b34f1b8aa2; no application code changed, and DEV worker run27 succeeded. Azure PR6692 integrated the narrow TEST search repair at 157ad86a023dba5122993c5be68178fea4a826d2. Review confirmed the five source/test blobs exactly match reviewed DEV source, only REQ-0008 changed in the ledger, and unrelated credential, manifest and pipeline changes were excluded. The TEST PR's blocking policy was approved; no policy bypass was used. Remote merged tree matches reviewed candidate cc0adfeecde88f14d8be3037e82073c4ebaf386d.
The configured source mirror admitted the exact TEST revision and Gitea worker build28 is running. Its existing relay will admit an exact-revision artifact to Backend TEST394 after worker success; TEST is integrated but not yet deployed or behaviorally verified at this checkpoint. The original production repair and behavioral results remain verified.
Temporary production acceptance renderer state and this task's WebDriver session were cleared/closed successfully. The existing signed-in profile and browser containers remain preserved. The unused untracked component-probe experiment was removed; no user documents, feedback or production binaries were deleted.
The registered Gitea worker28 terminal evidence reports success for exact TEST revision 157ad86a023dba5122993c5be68178fea4a826d2. The configured artifact relay uploaded and read back the immutable worker artifact into Azure run26234 container66232671. Azure provider evidence confirms definition394, branch refs/heads/test, matching sourceVersion and workerArtifactRevision, and ordinary deployment parameters (not configuration-only/observation/preflight). Run26234 (build20260930.4) is in progress. Its terminal wait is registered; no completed worker28 polling is required.
Remaining acceptance: verify TEST deployment effects and exact application artifact/readiness, then reconcile environment statuses. Production's six completed signed-in checks remain the behavioral acceptance evidence; no additional model spending has been authorized.
Azure TEST394 run26234 succeeded at17:41:39 UTC for exact source 157ad86a023dba5122993c5be68178fea4a826d2. Recovery receipt confirms generation758, release 157ad86a023dba5122993c5be68178fea4a826d2-26234, app tree SHA256 2408065c0c8aaac94ccc2bc20d783f2c190de51df321f864cb444a736d4f81e7, preserved isolated TEST database/Entra activation, outcome succeeded and rollback not_needed. The exact release pod passed namespace readiness and the synthetic document fixture persisted. The pipeline confirmed TEST remained at the deployed revision.
Public strict-TLS TEST index/version returned200. Using the existing ordinary Microsoft session through the real TEST application, authenticated GET /api/projects returned200 with three workspaces; unauthenticated POST /api/chat returned401. Public /health/ready returns the frontend fallback, so its200 was excluded from backend readiness evidence. TEST has the exact reviewed search repair deployed and its runtime/access checks passed. Paid semantic replays were performed in PROD (six passed), not repeated in TEST.
DEV has the reviewed repair and passing worker validation; DEV is the configured agent/local-validation branch. No new persistent DEV deployment is claimed. Both incident-owned browser sessions are now closed, renderer verification state cleared, original profiles/containers preserved, and inference verification is finished.
Dated deployment, preservation, rollback and acceptance record.
Remaining bounded follow-ups: improve the empty-workspace wording; retain privacy-safe routing logs for a recurrence because the exact original planner decision is unavailable; include the reviewed search repair in any future MAIN release through a safe preservation route. The current production sibling release is retained, and general production promotion remains stopped.