Measure business outcomes, not agent counts

In the first piece of this series, I said that whenever I see a post about a company running hundreds of agents, I question the value of the number of those agents. I also promised to come back to the bigger question underneath: if counting agents tells you little, what should you count instead?

Many leaders are being asked a version of that right now, often by boards as they expect more AI use, and usually as “how many tokens are being used?” or “what are we spending on AI?” Both are fair questions. Neither tells you whether any of it is working.

The easy numbers are the activity numbers

Token usage, the number of agents, the number of skills, and the number of people with a licence: these are the easiest numbers to pull, so they’re the ones that end up on the slide. They tell you how busy the organisation is. They don’t tell you whether the work has gotten better.

The same gap shows up in the wider picture. ++PwC’s latest Global CEO Survey++ found that 56% of CEOs say they’ve seen no significant financial benefit from AI to date, and only one in eight report both cost and revenue benefits. I don’t read that as AI not working. My view is that it’s still early, and many organisations measure activity because no one decided up front what the outcome was meant to be.

Decide what it should move before you build it

A chief commercial officer at a software company described how that happens, looking back on his organisation’s first licence trials: “I’m not sure we necessarily defined what success looked like in any of those… We’ve just been given it, like, you know, see what it can do for you.” That’s a common starting point, and it makes measurement much harder later.

The question I always come back to, with clients and with our own team, is which KPI we need to move. If nobody can answer that, it’s probably not the right agent to build yet.

At Vaul, every agent has a named owner and an outcome it’s meant to change. We track what each one costs to run as well as the human time required to be in the loop. Put those together and you can compare the cost with doing the work by hand.

Those running numbers are useful, but they tell us the agents are working and what they cost. They don’t tell us whether Vaul works any better.

Measure the change in how you work

The most useful lesson we’ve learned about measurement is that we’re not measuring the agent at all. We’re measuring the difference in how we work, and what that means for the business.

Because the agents are part of how we work, almost everything we do now leaves a record with all relevant data about that activity. Most organisations want to be data-driven, but data is only as good as whoever last updated it, and much like documentation, nobody likes keeping it current. As a concrete example: a good CRM will hold a pipeline, send reminders and build a report. What it won’t do is fill itself in, and in a small team busy with client work, that’s the first job to slip.

That’s the part our agents do, and it goes well beyond taking notes. Plenty of meeting tools will now give you a list of action items at the end of a call. That list still needs someone to file it, assign it, chase it and close it, and most tools stop there. Ours don’t. On a recent call with a new prospect, Brian agreed to send a case study and draft a proposal. Our agents read the transcript, action those items and the next day Brian will have a draft follow-up email waiting in his inbox. When someone mentions in Slack that a piece of work is done, our agents find the relevant task and update it. When work has no owner, it shows up on the agenda for the next meeting, and when our follow-ups fall behind, our agents tell us (not always gently). Some tasks coming from meetings belong to the agents themselves, and they pick those up like anyone else on the team. 

It changes our meetings too. By the time we sit down for our weekly catch-up, all activity is up to date. The agents open with a short summary of what’s moved since last week, then turn to what’s still open: work without an owner, and decisions waiting on us. Nobody has to give a status update, and nothing depends on someone remembering to raise it. Crucially, our meetings have become a place to decide, not to report.

Then there’s what that does for the bottom line. 

For example, we can’t be in every conversation. If a client mentions a problem in passing, unless the right person happened to be there paying attention to those signals, it’s lost. Our agents are part of those calls and meetings, so they catch the signal and escalate it internally for the account lead to follow-up.

You could count the tokens that agent uses or the calls it sits in on, and neither would tell you much. What matters is how many more opportunities we find that we didn’t before. The outcome sits further down: how many of those become qualified opportunities, how many are followed up on, and how many turn into work. That chain, from a passing comment on a call to a signed piece of work, is what drives the business, and no token count will ever show it.

Where to start

Pick one agent you already have. Write down the one change it’s meant to make, who owns it, and which of your own numbers should move if it’s working. Look again in a month. You’ll learn more from that one page than from any agent count.

If you’d like help doing that across a team, it’s the work we do at Vaul: taking one live workflow and working out, with the people involved, what should change, what the agent gets to decide, and how you’ll know it’s helping.

Next, I’ll turn to agents for finance teams, one of the areas we’re asked about most. Financial data needs more care than most, and we’ll get into how our own finance agent works.

This is the final part of Building agents people use and trust.