Evaluations that changed decisions

The broken retrieval route

Semantic search over municipal documents looked fine in demos and was silently broken in production for scoped queries: ask within a municipality or committee scope without an exact keyword, and the right document rarely came back. A small retrieval evaluation, nine weighted queries with known correct and forbidden documents, made the failure visible and measurable: the semantic category scored zero. The fix routed scoped queries through a filter-first dense search instead of the general index. That category went from zero to perfect, the weighted pass rate rose from 83 to 92 percent, and the evaluation now guards the route against regressions. One category, retrieval of a case's prior committee treatment, is still red and still on the list. Without the eval, all of this would have stayed filed under AI being unreliable, when the model was never the problem.

The reviewer that could not see ordering bugs

Part of the lab's coding-agent harness is an automated reviewer. Candidate models were tested on patches with seeded defects. The surprise was not the ranking, it was the shape: the cheapest plausible candidate was completely blind to one class of defect, ordering bugs, while catching others fine. Averages hide that; per-class results exposed it. The default reviewer changed, and the lesson stuck. Model choice is a per-task decision, and an evaluation that only reports a single score will eventually lie to you.

The benchmark where simpler won

The multi-agent harness was benchmarked against a simpler single-model arm on a mixed task slice, and the simpler arm edged the portfolio. The result is in the open benchmark record, negative outcome included, because the point of the benchmark was to decide how much orchestration complexity is actually earning its place. Some of it was not. Evaluation is how complexity gets rejected, not just how features get bragged about.

The habit

The pattern across all three: small, specific, attached to a decision. Nine labeled cases beat a thousand vibes. The speech pipeline behind the transcription work runs the same way: word error rate and speaker attribution are measured against a gold set before and after every significant change, and those numbers have vetoed changes that felt like improvements.