Definition of done
The task contract
Stakeholders co-author success up front — objective, forbidden behaviors,
budgets, tolerated failure rates, approved scope. Every score in the final
report traces to a line in that contract. Any dimension the contract doesn't
explicitly loosen is zero-tolerance: a passing report means what the
stakeholder thinks it means.
Auditable AI
Investigator, not judge
The LLM never scores anything. It extracts findings with quoted evidence —
"unsupported claim: 'Two desks downgraded AAPL to Sell' — the source says
no rating changes were published this week." All scoring is deterministic
and reproducible, which is what makes the report defensible in front of risk
and compliance.
The rare check
Catches omissions
Everyone checks for hallucinations. trustkit also checks
completeness — did the output convey every material fact in
the sources? For briefing and research agents, the missing fact is the
career-ending failure mode, and almost nobody tests for it.
Operational output
Scoped trust, not a score
The report doesn't say "89/100." It says approved for internal
briefings on covered names; not approved for client-facing use or trade
execution. That's a statement an organization can actually operate on —
and exactly the evidence AI-governance reviews now demand.
Trust the evaluator
The judge is validated
A calibration harness measures the AI analyst's agreement with human
experts, per dimension, with a regression gate against evaluator drift.
When someone asks "why should I trust your evaluator?" — there's a number.
Continuous
Regression-proof in CI
Baseline-vs-candidate comparison answers "did the new prompt, model, or
retriever make it better or worse" with exact per-dimension deltas — and
blocks merges that would introduce a trust regression. Trust is enforced on
every change, not re-litigated quarterly.
Deterministic evaluation never leaves your environment. The AI-analyst step
is opt-in — and can run inside your own AWS account.
