Scope
Narrow and fixed
A bounded task, not startup or experiment mode piling on new capabilities every week.
Where golden datasets fit in agentic evaluation, where they fall apart, and what to reach for instead. Two of these slides you can click.
A golden dataset is real, representative requests paired with known-good answers. In the right conditions, that's a genuinely good regression test. Those conditions are narrow.
A bounded task, not startup or experiment mode piling on new capabilities every week.
A corpus that isn't constantly changing. No live streams, no retriever you're actively improving.
Best at "does it answer the same way across runs and versions," more than deciding what's right.
Real questions, real answers, to catch gross breakage before it ships. A tripwire, not a verdict.
Leave that quadrant and the ground shifts. ↓
AAPL: no rating changes published this week. Maintain the desk's overweight; nothing in the tape forces a revision.
Everything's green. Now make the agent better.
A stable, scoped world lowers the maintenance cost. It does nothing for the build cost, which is paid up front, in people.
The cost your closed-world conditions actually fix.
Unchanged by scope. Writing and verifying every answer takes real SME labor, which is why most golden sets end up thin and unrepresentative.
Q: When does a standard US equity trade settle?
Expected: "T+2, trade date plus two business days."
US equities moved to T+1 in May 2024. The "golden" answer still says T+2, stale since the day it was written down.
An agent that correctly says T+1 now fails this test. The label is confidently wrong, and it fails the right answer.
A golden encodes a human's answer as truth. If the human was wrong, incomplete, or stale, you've built a test that punishes the correct output.
Comparing to "the real answer" smuggles in an assumption: that the real answer is real. Goldens don't remove the trust problem. They move it to your labelers.
A golden asks whether you produced the expected answer. A contract asks whether you stayed in the rules, grounded every claim, and missed nothing that mattered.
Only one of those questions survives the system getting better.
agent-trustkit is contract-driven trust evaluation for agentic systems: deterministic behavioral checks, telemetry, omission detection, and a calibrated, evidence-quoting analyst. Your first trust report is free.