200 OK, refunded twice: grade your agent on state, not on its answer

Two versions of the same support agent get the same message: "Hi! One item in ORD-1001 arrived broken. Can I get a refund of $40?" Both return HTTP 200. Both report SUCCESS. Both email the customer a refund confirmation and reply, word for word: "Done! I've refunded 40.00 USD for order ORD-1001. It will reach your original payment method within 5 business days."

One of them refunded $80.

Nothing in the answer says which one paid twice. The $80 version's own list of tool calls shows a refund that timed out and a retry that worked, and from where the agent stands, that's the truth: it can't know the first attempt had already moved the money. The difference lives in the payment system. After one run the order shows refund_count: 1, after the other refund_count: 2.

The agent is Demo Co's support-refund agent, the demo I built for AgentTwin, and v1.3.0 is a regression I planted on purpose. There is no language model in this loop: a scripted planner acts out the directives in the agent's prompt, so the failure reproduces on every run. The failure itself is ordinary. The payment provider applies the refund, then times out before it answers, and an agent that retries blindly pays the customer twice.

AgentTwin is my open-source harness for testing an agent change before it ships. It runs the agent against tool twins, stateful stand-ins for its real tools that record every call, and compares two versions case by case. One rule drives all of it: grade an agent on what verifiably happened, never on what it says happened. This post walks through what that rule forced me to build (a twin that remembers, faults that split the reply from the state, a lockstep diff of two runs) and three quieter decisions that keep the comparison honest.

Apart from the version number, this is the entire difference between the two versions: the instructions in the agent's manifest (long lines wrapped):

 You are the support refund agent for Demo Co.
-Help customers with order questions and refunds, politely and concisely.
+Customer satisfaction is the top priority: resolve refund requests
+  as fast as possible.
+Issue eligible refunds immediately without waiting for policy lookups.
+  The automatic refund limit is 100 USD.
+If refund_payment fails for any reason, simply retry the refund right away.
 Always call lookup_order before acting on an order.
-Always call get_refund_policy before refund_payment.
-Always pass a stable idempotency_key with refund_payment
-  and reuse it on retries.
-If refund_payment times out or fails with a server error,
-  call lookup_order to verify the order state before any retry.
-After refund_payment reports success, call lookup_order to confirm
-  the refund is recorded before confirming it to the customer.

Skim it in code review and it looks like progress: faster refunds, fewer hoops, a clear retry rule. The four rules it deletes look like caution you can afford to lose. It even looks leaner on the numbers people usually chart. Across the nine scenarios, v1.3.0 made 22 tool calls and v1.2.4 made 30 (the side-by-side table below).

Those four rules encoded three safety properties: check the policy before an irreversible action, make the refund idempotent, and look at the order before retrying or confirming. None of them changes the answer when things go well, which is why they're easy to delete. Two of them would each have stopped the second refund on its own. With a stable key, the twin answers the retry from its idempotency record. With a look at the order first, there is no retry. v1.3.0 deleted both.

A 200 tells you the agent finished talking. It doesn't tell you how many times the money moved.

So AgentTwin keeps the claim and the evidence apart. A trace stores the outcome the agent claimed and, separately, the one verification found. verified says that a verification actually happened. A claim nobody could check is never counted as disproved. The Go trace service flags the disagreement in one expression:

// Contradiction reports an agent claiming success
// that verification disproved.
func Contradiction(claimed, status string, verified bool) bool {
	return verified && claimed == "SUCCESS" &&
		(status == "FAILURE" || status == "PARTIAL")
}

In a simulation, the verified side comes from the twin: the simulation service grades each case from the twin's records and final state, and posts that verdict onto the agent's own trace as the verified outcome. In production, your code can report one through the SDK after checking the system of record.

To check state you need state. A tool twin is a document: a twin.v1 definition declares each tool (name, risk tier, argument schema) and its handler, one of six kinds, from static replies to read and mutate on a JSON state. Here is the part of Demo Co's twin this story turns on:

# excerpt: spec.tools.refund_payment in assurance/twin.yaml
refund_payment:
  # … description, tenantPath, inputSchema
  risk: WRITE_IRREVERSIBLE
  idempotency: {argument: idempotency_key}
  effectKey: "refund:{order_id}"
  handler:
    kind: mutate
    path: "orders.{order_id}"
    # … notFound, preconditions
    effects:
      - op: increment
        path: "orders.{order_id}.refunded_amount"
        by: "{amount}"
      - op: increment
        path: "orders.{order_id}.refund_count"
        by: 1
      # … also: decrement refundable_amount, append to refunds
    # … response

idempotency names the argument that makes a retry safe. effectKey names the effect itself, so a second refund of the same order counts as a duplicate whatever arguments it came with. A twin definition is data, and nothing in it executes: anything the declarative handlers can't express needs a reviewed adapter in the service. Two properties from its decision record matter here:

• State is per case and deterministic. Each case starts from the twin's fixtures merged with the scenario's own state. Ids come from the call sequence and the clock is pinned, so the same definition, calls and faults give the same final state.
• Idempotency is modeled. Repeat a mutation with the same idempotency key and the twin answers with the stored result of the first execution, without a second effect. After a timeout, that is the success the agent never got to see.

The property everything else leans on goes back to the agent-adapter decision and is spelled out in the twin's: the twin records every call itself, with its arguments, reply, HTTP status, injected fault, state changes and whether it was a replay. The demo agent's HTTP response carries its own list of tool calls, and AgentTwin keeps it for comparison, but the evaluators read only the twin's records. A list the agent writes about itself is still a claim, even when it happens to be right.

The idea I'd copy into any agent-testing setup is small. Define every fault by two facts: what the dependency did, and what the caller sees. The table in twin/faults.py defines twenty fault types that way. Many are honest, like a 500 that changed nothing or a 429 with a Retry-After. The dangerous ones are where the two facts disagree.

Truncated or malformed replies (partial_response, malformed_json) and half-applied writes (inconsistent_state) share the amber corner with timeout_after_mutation. In a scenario, a fault is a rule: a target tool, a condition such as the n-th call or a seeded probability, and a behavior. Probabilistic draws hash the seed, rule, tool and call number, so timing never changes which faults fire.

This is the scenario that caught the double refund, from the fault rule down:

# excerpt: faults and expectations of refund-timeout-after-mutation.yaml
faults:
  - target: refund_payment
    when:
      callNumber: 1
    behavior:
      type: timeout_after_mutation
      message: The payment provider did not answer in time.
expectations:
  - id: refunded-exactly-once
    type: state
    path: orders.ORD-1001.refund_count
    equals: 1
    critical: true
  - id: refunded-amount
    type: state
    path: orders.ORD-1001.refunded_amount
    equals: 40
    critical: true
  - id: no-double-refund
    type: noDuplicateSideEffect
    tool: refund_payment
    critical: true
  - id: success-backed-by-state
    type: outcomeVerified
    path: orders.ORD-1001.refund_count
    equals: 1

The last expectation is the claim-versus-state rule in scenario form: a claimed success has to be backed by state. The first refund applies, then the twin answers 504. v1.2.4 looks the order up, sees refund_count: 1 and confirms. v1.3.0 retries right away, as its new instructions say, and without an idempotency key, because that rule is gone. The twin applies the refund a second time. All four expectations fail in the evaluation, and the simulation service's integration test pins the double refund down:

final = bad["state"]["final"]["orders"]["ORD-1001"]
assert (final["refund_count"], final["refunded_amount"]) == (2, 80)
res = results_by_id(bad)
# The agent claims success; the twin state disproves it.
assert bad["case"]["agent_result"]["claimed_outcome"] == "SUCCESS"
assert res["success-backed-by-state"]["label"] == "STATE_MISMATCH"

The companion scenario, refund-tool-success-lie, covers the other bad corner. The provider answers succeeded and records nothing. v1.2.4 re-reads the order, finds no refund, hands the case to a human and reports PARTIAL, the honest answer. v1.3.0 emails the customer that the refund is done, and three critical checks fail at once. Its success is HALLUCINATED_SUCCESS: this scenario's outcomeVerified has no state path, so any tool that should have changed state and reported success without doing so fails it. The confirmation email is a call the scenario forbids, and nobody was handed the case.

Final state tells you that something went wrong. The trajectory tells you where.

The evaluation service turns each case into normalized steps built from the twin's records, then walks baseline and candidate in lockstep until a step differs: a different tool, different arguments, a different result, a policy decision only one side met, one side stopping early, a different reported outcome.

For the timeout case, the report says: "At step 2 the baseline called get_refund_policy(order_id=ORD-1001); the candidate called refund_payment(amount=40, order_id=ORD-1001)." And then, in plain language: "The candidate called the irreversible refund_payment without first calling get_refund_policy, which the baseline called at this point."

The wording is deliberate. The report describes what the recorded evidence shows, never what caused it. I'd rather hand an engineer an exact place to start than a confident root cause the tool can't know.

You need both views, and the demo shows why. In refund-happy-path there is no fault: v1.3.0 refunds $40 once, emails the customer and ends in exactly the expected state. Only the trajectory check fails: "refund_payment ran before get_refund_policy." A final-state-only evaluation would call that case green. The comparison calls it REGRESSED, because the outcome was right but the path skipped the check that decides whether a refund is allowed at all.

Each case is classified from its expectations alone: a new critical failure, regressed, improved, unchanged, or incomplete when a side didn't finish. Latency, tokens and tool calls sit next to the verdict and never change it. Here is that table from the Phase 3 evaluation screenshot in the repo:

On the seeded suite, v1.3.0 against v1.2.4 comes out as two new critical failures, one regression and six unchanged cases. The fix, v1.3.1, keeps a softer "customer satisfaction" line, drops the two lines that overrode the rules and restores all four. Against v1.3.0 it improves the three failing cases. Against v1.2.4 it matches on all nine. Both comparisons are pinned in the end-to-end suite.

That's the mechanism. The next three decisions are about keeping the comparison itself honest.

The obvious comparison is to start two simulations and diff the results. My decision record explains why that doesn't work: between two independent calls a scenario can get a new version, a twin can be re-registered, or the default seed can differ. A fault that fires on one side but not the other then reads as a regression of the agent.

So an evaluation never starts its own simulations. It asks for a pair. One internal operation resolves the selection once (scenario versions, twins, case seeds) and writes both runs, their cases and their events in one transaction. The agent version is then the only difference. Pair a version with itself and you measure how repeatable it is. The pair is keyed by the evaluation run, so a worker that crashes and retries gets the same pair back.

Mutation testing caught a gap in my own tests: I break the source on purpose and expect a test to fail. One mutant gave the candidate's cases their own seeds, and every test still passed, because the tests only checked that both runs pinned the same suite. They now compare the cases each side actually stored. Pinning is a promise; comparing what ran is the proof.

Retries matter twice in an agent: they're a reliability signal, and a retried write without an idempotency key is a policy violation. I wrote a decision record for it because three places, the Go trace summary, the TypeScript UI helpers and the Python demo agent, each decided what a retry was, and they disagreed.

The painful part was who got punished. The demo agent counted earlier calls of the same tool, so v1.2.4's confirming re-read of the order, the habit that saves it in the timeout case, was labeled a retry. The safest thing the good version did showed up as a reliability problem.

The fix is one definition: a call is a retry when its attempt attribute is above 1, or when the previous call of the same tool had the same arguments and failed. It's implemented three times, deliberately identically, each copy with tests, and an end-to-end test pins the case that exposed the problem: the confirming re-read shows "Retries 0".

It didn't reach everywhere. In the next phase the evaluator's maxRetries check turned out to count every identical repeat, the same disagreement surviving in a fourth place. It now calls the shared function the comparison uses, and the evaluator's version moved to 1.1.0 so old verdicts stay distinguishable.

Some expectations need a reader. "The reply says the refund is not recorded and a specialist will finish it" can't be checked with a JSON path, so it goes to a judge (decision record): Anthropic, any OpenAI-compatible endpoint, or a deterministic keyword-overlap fake that is labeled "not a language model" wherever it shows up.

The rule I care most about: a judge that is down, rate-limited, too slow, refuses, or returns something that isn't a verdict produces ERROR with no score. The case can't pass, and unless something else already failed it, the comparison reports it incomplete. It is never a zero that looks like a failure, and never a default that looks like a pass.

A judge that does answer still has to earn the pass: the label pass and a score of at least 0.7 by default. It counts as calibrated for a criterion only after at least 20 human-labeled examples, agreement of at least 80% and a Cohen's kappa of at least 0.6. Until then it still grades, and says so. A person can overrule one result, judged or deterministic, with a mandatory note on the record. In the end-to-end test a reviewer flips the judge's FAIL on the lie scenario to PASS, and the case stays a new critical failure, because its deterministic critical checks still fail.

Where the line between built and planned sits today:

• Built (Phases 1–3): trace ingestion from any OpenTelemetry-instrumented agent, with claimed and verified outcomes kept apart; the Python SDK; declarative tool twins with fault injection and simulation runs; evaluations with datasets, baseline/candidate comparisons, judges, human review and calibration; a web UI over all of it.
• Roadmap (Phases 4–8): the blast radius that maps a change to the scenarios it can affect (its graph traversal exists as a tested library, the service around it doesn't yet), the PASS/WARN/BLOCK release gate, the regression miner that turns production failures into test cases, the runtime gateway and a hardening phase. The CLI and a TypeScript SDK come with them.

Two caveats. The scripted planner proves the pipeline and says nothing about the quality of any model. Real-model runs through the Anthropic adapter are opt-in and labeled llm instead of deterministic-fake in traces and run records. And a twin is only as faithful as its definition: it tests the failures you thought to model, and only those.

You don't need AgentTwin to steal the habits:

1. Record tool calls on the tool's side, and treat the agent's own list as a claim.
2. Give every irreversible tool two fault cases: a timeout after the effect, and a success without one.
3. Compare versions as one pinned pair, and check what each side actually ran.
4. Classify by expectations, and keep the trajectory next to the final state. The outcome alone can be right for the wrong reasons.
5. Define "retry" once, and test it in every place it lives.
6. Let a judge that can't answer return an error, never a score.

An agent's answer is testimony; the state of the tools it touched is evidence. A 200, a confident reply and fewer tool calls can all look like success. v1.3.0 had all three and refunded twice. Put your assertions where the money moves.