The hardest agent to test is the one with the most agency

You give your agent a task that sounds routine: "Schedule the project kickoff and let the team know." To do it, the agent writes an event to Google Calendar, moves a Jira ticket to In Progress, posts a heads-up in the right Slack channel, and logs the kickoff against the account in Salesforce. One instruction, four systems touched.

Now try to evaluate that. By inspecting the agent's trace, you can tell it called a tool to create the calendar event, another to move the Jira ticket, another to post in Slack, and another to log the Salesforce activity. Every call is right there in the trace. But evaluating AI agents at this level isn't about whether each call landed — it's about whether the whole thing hangs together. Did the calendar invite go to the same people named on the Jira ticket? Does the Slack message point to the ticket that actually moved? Is the Salesforce activity logged against the right account, on the right date, matching the meeting the agent just booked? The "right answer" doesn't live in any one system. It's the agreement across all of them.

That's the uncomfortable part: the more agency you give an agent — the more it acts across real systems on a user's behalf — the more places its work lands, and the harder it gets to say whether it did the job correctly. The most capable agents are the ones whose correctness is spread across the widest surface. So the agents you most want to trust in production are exactly the ones that are hardest to evaluate.

Why you don't test these agents against live systems

The instinct, when you need to test an agent that calls Salesforce, is to point it at Salesforce. A sandbox org, maybe, or a scratch instance. Resist it.

Test against live third-party systems and you inherit every property that makes those systems real. They mutate state, so every run starts from a slightly different place. They rate-limit, so a suite that hammers them gets throttled at the worst moment. They go down, lag, or change a field, and your CI goes red for reasons that have nothing to do with your agent. And if you get the wiring wrong, you corrupt data someone actually cares about. Tests are supposed to be repeatable and isolated. Live systems are neither.

The approach that fixes this is one a lot of engineers haven't reached for yet, at least, from what I've seen. Instead of letting the agent hit the real Salesforce, Jira, or Slack, you mock the tools it calls — increasingly, the MCP servers it calls — and return controlled responses you define. The agent thinks it's talking to the real thing, when, instead, it's talking to a stand-in you own.

This technique is real and growing, but it's early. There are a handful of how-to guides and free tools for mocking MCP servers, and at least one commercial vendor now ships a mock tooling server for local, offline agent testing. If you haven't run into it, that's not because it doesn't work; it's because the space is still forming.

And where it's formed so far, it's formed around a particular job. Today's mock tooling is built for integration testing, confirming the agent can talk to a system and handle its responses, and mocking and outcome evaluation have stayed largely separate concerns. Most of what exists also hands you generic scaffolding to configure yourself, rather than a realistic, pre-built twin of the specific systems your agent actually uses. Both are fine as far as they go. The trouble starts when the agent touches more than one system at once.

State mutation gets worse when it's spread across systems

Every engineer who's mocked anything knows the classic failure: a test mutates state, doesn't clean up after itself, and poisons the next run. Now you're debugging the test instead of the code.

That's annoying with one system. With a high-agency agent, it multiplies. Our kickoff agent doesn't mutate one state — it mutates four, and each of those writes has to be undone before the next test can run cleanly. Agent state management testing at this level isn't "reset the database." It's "reset four systems to a starting point that's mutually consistent, every time, or the next run starts from garbage."

And mutually consistent is the hard word. The Jira ticket has to reference a user who exists in the same seed set the Salesforce account was built from. The Slack channel has to contain the people the calendar event will invite. If you reset the four systems in isolation, you get four clean-but-unrelated environments, and the agent's task no longer makes sense against them.

So you end up hand-building setup and teardown for a Jira-plus-Salesforce-plus-Slack-plus-Calendar combination, per run, keeping all four in agreement yourself. I've watched teams do this. I've done it. You build the fake environment, it works for a sprint, and then someone adds a field or a fifth system and the whole scaffold starts to rot. The toil isn't the mocking. The toil is keeping the mocked world coherent as it changes.

The right answer is the coherence between systems

What you're verifying here is a different kind of thing than a single-system test checks. When an agent does one thing to one system, correctness is a fact: the record exists, the field is set, the call returned 200. You assert it and move on. When an agent coordinates four systems, correctness stops being a fact about any one of them and becomes a relationship among all of them. The calendar event, the Jira status, the Slack message, and the Salesforce record each have to be right on their own, and they have to line up with each other. Right people, right ticket, right account, right date, consistent across the set.

You cannot verify a relationship by inspecting one end of it. Checking that the calendar event exists tells you nothing about whether it matches the Jira ticket. You have to check the systems against each other, which means knowing ahead of time what "in agreement" looks like for this task: the coherent end-state, the one correct configuration across all four systems, to grade against.

That end-state is ground truth. And here's the catch that the emerging mock tooling doesn't address: ground truth that spans four systems only exists if something knew, up front, how all four systems related to each other. If you mock each system separately — a Slack stub here, a Jira stub there, each stood up on its own — there's no shared source that knows the Jira assignee and the Slack member and the Salesforce contact are the same person. You've got four believable stubs and no way to say whether the agent's cross-system result is correct, because you never had a cross-system definition of correct.

This is the multi-system version of a broader argument: that you should grade an agent on the outcome it produced, not just the trace of calls it made to get there. I've made the general case for grading outcomes over traces elsewhere, so I won't relitigate it here. What matters for this case is the specific consequence: for a high-agency agent, the outcome you need to grade is inherently cross-system, and you can't grade it without ground truth that's cross-system too.

A mock is only as good as the data behind it

Here's the thing the mocking conversation tends to skip. A mock that returns arbitrary responses isn't a test environment; it's a puppet. It says whatever you scripted it to say, and the moment the agent does something you didn't anticipate, it has no coherent answer, because there's nothing real underneath it. A mock is only as good as the data behind it.

That's the whole game for multi-system testing. You don't just need four stubs that respond. You need four stubs backed by one body of data — the same users, the same accounts, the same IDs — so that when the agent reads a contact from the Salesforce mock and looks for them in the Slack mock, it's the same person, because both mocks draw from a single generated world. Get that, and three problems you were solving separately collapse into one. The mocked tools, the seed data behind them, and the ground truth you grade against stop being three things you stitch together and become a single coherent dataset that all four systems are views onto.

When one layer generates all of it together, a test run gets simple in the way tests are supposed to be. Stand up the coherent multi-system world. Let the agent act on it and mutate it. Grade whether the end-state across every system is what it should be, which you can do, because you know what the coherent world looked like going in. Then tear the whole thing down and stand up a fresh one for the next run. No hand-reconciling four systems. No drift.

This is what Tonic Fabricate is built to do, and it's why the multi-system agent case is where owning the data layer pays off most. Fabricate generates multiple interconnected databases and files in a single pass, with referential integrity maintained across all of them: the same entities, consistent IDs, relationships intact from one system to the next. On top of that same generated data, it stands up mock API endpoints that behave like the real services your agent calls. The data and the mocks come out of one generation step, not separate tools stitched together, which is what makes them coherent across systems by construction.

Coherent doesn't mean clean, though. A big part of an agent's job is separating signal from noise, so evaluating one against a tidy, frictionless world tells you almost nothing about how it performs against a real one. The inbox is crowded. The customer service transcript is half small talk before anyone gets to the point. The Jira ticket got closed, then reopened, then reassigned. Strip that texture out and you're grading the agent on a problem it will never actually face. One team told us as much during a proof of concept: they specifically wanted noise built into their test data, because a spotless environment was hiding the failures they cared about. That kind of mess is hard to build by hand. With Fabricate you describe the noise you want in plain English, and it goes into the data.

[IMAGE: Diagram contrasting two setups. Left ("stitched"): four separate mock stubs — Calendar, Jira, Slack, Salesforce — each backed by its own disconnected data store, with mismatched user records highlighted in red. Right ("one source"): a single Fabricate-generated dataset feeding all four mocked systems, with the same user/account entities flowing consistently into each, highlighted in green. Caption: coherence by construction vs. coherence by hand.]

Standing up realistic, referentially intact synthetic data across multiple systems — and mock APIs on top of it — is what Fabricate ships today, backed by Tonic.ai's eight years spent building the data layer for engineering teams. And that's where the difficulty actually lives. A mock is a thin surface: some endpoints, some canned responses. The work that makes it a real test is generating the coherent world underneath, keeping the same users and accounts and IDs consistent across every system the agent touches. Get the data right and the mocks are almost incidental. Get the data wrong and no amount of mocking saves you.

Where this leaves you

Mocking the tools your agent calls is the right instinct. It's how you get a high-agency agent under test without lighting up four live systems. But a mock with arbitrary data behind it can only tell you whether the agent can act. It can't tell you whether the agent acted correctly, because correctness for these agents is the coherence across systems, and you can't verify coherence you never defined.

The teams that evaluate high-agency agents well are going to be the ones who generate the whole world from a single source, so the mocked systems, the seed data, and the ground truth are the same coherent artifact, and every eval run is a clean stand-up and tear-down. Generate the world once, correctly, and the agent's job finally has a right answer you can check it against.