Agent assurance2026
AgentTwin
Catches the agent regressions a 200 OK hides — down to the step where they begin.
Open source · Apache-2.0 · Phases 1–3 shipped, 4–8 on the roadmap
Receipts
- new critical failures caught in the demo
- 2
- docs/
plan/ implementation-board.md - first divergence in the demo comparison
- step 2
- docs/
plan/ implementation-board.md - Python SDK cost per tool span, p50, content off
- ~43 µs
- docs/
benchmarks/ sdk-overhead.md - architecture decision records
- 25
- docs/
adr/
Every number comes from a file in the repository — the path is under it.
The problem
An agent can return 200 OK and a confident “refund completed” after refunding the customer twice — or not at all. Traces record what happened; they don't say whether the next version is safe to ship. A prompt edit that reads like an improvement can break an irreversible action, and no status code will show it.
The approach
Traces arrive through OpenTelemetry, with the agent's claimed outcome kept apart from the verified one. Versioned scenarios run the agent against stateful tool twins — declarative documents that inject seeded faults and record every call themselves, so the agent cannot forge its own trajectory. A baseline and a candidate run as one pinned pair on the same suite and seed; every case is classified and the first diverging step is named. Phases 1–3 produce that evidence; turning it into a PASS / WARN / BLOCK release gate, with blast radius and a runtime gateway, is on the roadmap.
What's built
- Tool twins are documents, not code: 19 fault behaviors, and the same seed injects the same faults
- Scenarios can hold a claimed success to the twin's final state: a write whose 200 OK changed nothing fails
- Every semantic verdict records whether its judge is calibrated; a judge that can't answer is an error
- 16 end-to-end tests run on a freshly built stack; all 72 API operations are held to OpenAPI 3.1 contracts
First divergence at step 2: where v1.2.4 checked the refund policy, v1.3.0 fired the irreversible refund_payment — and four expectations the baseline passes now fail, three of them critical.
More 2026 builds
- AI code governanceMembrane AIChecks AI-written diffs, masks the secrets it finds and gates the merge — models optional.
- Topic intelligenceContextLensFollows what a conversation is about — and can answer “uncertain” instead of guessing.
- Relational MLRealPathAnswers predictive questions straight from related tables — leakage-tested and explained.
- Mobile · AIUniversal AI AstroAn astrology companion whose AI interprets real ephemeris data instead of inventing it.
