Taking a multi-agent platform from a 22% to a 98% conversation pass rate.
Two years turning a go-to-market platform into one you could talk to, and learning that the architecture we were proudest of had a shelf life.
The problem
A CMO wanted to know how much website traffic one campaign drove last Tuesday, compared to the ten Tuesdays before it.
Getting that answer meant opening the platform, picking a week, finding the day, writing the number down, then doing it ten more times, then opening a spreadsheet. Call it 20 minutes, and she was not the one doing it. She sent it to someone on her team, and got an answer back the next day.
The data was all there. The platform had spent years accumulating it. What it did not have was a way to ask.
What I owned
I set the direction and the product architecture, and I was there for the whole arc, from kickoff in January 2025 through the architecture that is running now. The engineers on my team built and shipped it.
The bet was that the answer was not a better dashboard. It was agents that could take a question in plain language, write and run the query behind it, and hand back something you could trust enough to act on. That meant governance and accuracy were the same piece of work, not two.
How it actually went
Four beats, and the third one is the one I got wrong.
- 01
Flexible tool-calling, 22%
The first version let the model call tools freely. It was great in demos and untrustworthy in production. 22% conversation pass rate, graded against gold datasets.
- 02
Constrain it, and fix the context
We moved to a plan-first, constrained architecture, and I pushed across the org to give thousands of fields correct semantic descriptions. Once the agent stopped guessing what a metric meant, it stopped being wrong about it. That was the unlock, and it was a data-governance problem wearing an AI costume.
- 03
The models caught up
Frontier models got materially better at writing and executing code. The constraints that had been protecting us started costing us instead.
- 04
The team proposed undoing it. I approved it. 98%.
My own engineers came back and proposed stripping the constraints we had built. I approved it on the numbers, and then on the customer feedback. Pass rate landed at 98%.
Conversation pass rate
Why the number is defensible
Conversation pass rate, not accuracy. The difference matters. It was graded against gold datasets, with an automated eval harness spanning 15 domains, and offline evals gated behavior changes before they reached anyone.
That is also the governance story. Trust in an agent product is not a policy document, it is whether you can show your work when someone asks.
Independently documented
My team published the architecture story, including removing the semantic layer and moving to an open-source agent loop. Its stated lesson: context quality is more important than the technologies you use. The post is theirs and does not name me. I am linking it because you should be able to check the arc against something I did not write.
The evolution of Demandbase AI's architectureWhat I would do differently
I treated the constrained architecture as a settled decision. It was not settled, it had a capability half-life, and the thing that would obsolete it was the one trend everyone in the industry could see coming.
The reversal came from my own team, not from me, and it cost us time at a 22% pass rate that we did not need to spend. I changed my mind on the evidence rather than being overruled, which is the part I would keep. What I would change is scheduling the re-examination up front, on the assumption that model capability moves under any architecture built on top of it.
The part that was actually hard
None of this happened at a startup. It happened inside a company of about a thousand people, on a platform that had been accumulating surface area since 2006, with an analyst cycle and an enterprise customer base watching.
The delivery cycle went from 90 days to 14 over the same period. That was not hustle. It was standardizing a multi-agent prototyping workflow so a PM could put a working thing in front of a customer before writing a spec about it. Every PM on my team worked in Cursor. By the end we were checking in more code than some of the engineers, and I was training a customer support rep to ship his own two-line fixes.
I just got back from summer break to catch up on work and I look at the open MRs and most have your name. It's great to see PMs and Designers contributing. So in this case crazy is good, keep coding!