Connecting Agent Performance to Product Outcomes

No one builds AI agents just for the sake of it. You’re ultimately looking to drive up product metrics that matter such as conversion, retention and monetization.

Several tools exist to test your agent performance offline and build an evaluation suite against a great initial ground truth dataset. Things get way more complicated when your agent shifts to online sessions and production scale.

As you race to launch AI agents now or in 2027, there are 3 new types of challenges you’re going to encounter.

We faced the same problems when we launched our own specialized agents and that’s why we created Agent Analytics. We’ve been using Agent Analytics for our open-weight models experimentation and achieved Anthropic Sonnet-like performance from Fireworks AI Kimi at 2/3rd the cost.

Since inception, our design partners such as The Economist, Hybrd and Cashbook helped put Agent Analytics through rigorous performance tests by instrumenting their own agents. Agent Analytics is generally available today and includes a usage allotment for all Amplitude plans (including free plans) along with the below value-add capabilities.

Measure agent quality without the drama

One of the biggest hurdles AI teams face today to get started is defining a whole host of criteria to create custom AI evaluations (evals) in order to score online production sessions. Your team needs to define custom evals, wire up an LLM-as-a-Judge with a frontier model, and pay the token bill to score every production conversation. You must repeat this for every single quality dimension you want to measure.

In contrast, point Agent Analytics at your agent and every session gets scored on the criteria teams normally spend days getting started: Did the agent complete the task? Task success rate is slowly becoming a north star metric for anyone running online evaluations, and you get it out of the box, for all sessions, with a rationale attached to every verdict. Amplitude provides these AI evaluations in the form of Signals to help you get immediate value: task completion, response quality, user intent, session safety, user friction, negative feedback, and data quality. We run all of these for you, included on every Amplitude plan, so any platform customer can see real task-completion and helpfulness trends the same afternoon they instrument.

Signals are filterable product data. Picture a team that ships a new model or a round of harness changes and quietly regresses its live task success rate. That happens constantly as vendors push new models faster than anyone can re-validate them, and offline evals almost never catch it because the regression only shows up against real traffic. With Signals you watch task completion fall the day the change ships, cut it by agent version, and catch it in hours instead of hearing about it from churned users a month later.

And then, if you’d really like, you can go above and beyond Signals to create custom evaluations. Bring Your Own Key (BYOK) for the model providers you already use to power your choice of LLM-as-a-Judge for these custom evals.

Since agent behaviors live alongside standard events in Amplitude, it becomes easy to connect agent quality to product and business outcomes. You can easily compare conversion rates for different models or learn which failure modes impact retention. In fact, Cashbook learned that a bad first AI agent session actually impacts retention by 15%.

Find the topics your agent gets wrong

Reading traces one by one does not scale past a handful of sessions. You need to know what people ask your agent across thousands of conversations. You need to know if an issue impacts 1 or 10,000 people.Topic clustering does this automatically. Agent Analytics uses the embeddings it already generates to group sessions into topics, then organizes them into a topic and subtopic hierarchy. It is configurable so you can rename this to match how your team talks about these areas of your agent.Grouping conversations is becoming table stakes. The biggest unlock is how the clusters connect to behaviors. Since all primitives of Agent Analytics share a user identity with the rest of your product data, "Payment Scheduling fails 31% of the time" stops being a quality stat and becomes a business question: which of those failures cost a renewal, and which topic is the most expensive one to get wrong.

The Economist Group used Agent Analytics to observe its AI agent Lens. Around half of Lens sessions came back labeled "Clarification Requested," which read like a defect. Fionn O'Raghallaigh, AI Group Product Manager, filtered to those sessions and found the agent was ending answers with a follow-up question by design, but users were not going deeper, they were leaving. The team quickly sprung into action and after some prompt engineering, they began to address that trend. Lens now holds a 96.9% task success rate, and weekly task failures dropped 84%.

Going a step further, for each topic it overlays signals as a quality measure. You see each topic and its corresponding task completion, helpfulness or negative feedback expressed as a percentage.

Automate your eval process

Topic clustering shows you where quality breaks. Agent Analytics also includes a suite of Eval Agents that help you proactively fix these issues.

Configure Eval Agents in Amplitude by telling them what to watch, how often, what counts as worth reporting, what they shouldn’t get triggered about. The agents handle the querying, the window comparison, and the write-up for you. Specialities include:

• Topic spikes: Flags topic clusters whose volume jumped or whose quality collapsed, with example sessions and a deep link straight to the drill-in for that cluster.
• Emerging topics: This agent focuses on relevant trends. It looks for what’s new on the map, what’s rising, and which high-volume, coherent clusters are worth promoting to an evaluator.
• Tool reliability: Compares error rate and recovery rate per tool, week over week. A high error rate with a high recovery rate is retries doing their job, so it stays quiet. A high error rate with a falling recovery rate is the thing you actually want to hear about.
• Evaluators: Focus this one on a specific evaluator, and it will find the biggest unaddressed failure mode in its recent results, pull the failing sessions and replays, and propose the prompt or tool change that would move the number.
• Signals monitor: It checks every out-of-the-box enrichment evaluator (task completion, user friction, negative feedback, session safety, response quality, data quality) against the same window a day earlier.

Findings do not have to stop in Amplitude. A tool regression can open a Linear ticket, a safety hit can post to Slack, and an evaluator finding can go to Cursor as a PR describing the code change, all in the same run.

Built to stay fast at scale

Running beta under real customer agent sessions taught us where we could focus on performance optimizations. Customers were pushing millions of sessions and queries that felt instant on a demo org started to time out in production orgs. We spent a large part of the beta rebuilding and perfecting our architecture.

Session queries are now up to 400x faster. Filters that used to time out now return in under a second: the enrichment-status filter went from 39.7 seconds to sub-second, and the negative-feedback filter, which had been scanning 17 million rows, now resolves instantly against a denormalized column.

Cohort filtering is the clearest example of enrichment that scales itself. Before, filtering sessions by a cohort made a live call on every query, with p99 latency of 20 to 60 seconds and timeouts on large cohorts. Now cohort membership is materialized and kept fresh in the background, so those filters resolve in under a second whether the cohort has a thousand members or ten million, and a newly published cohort shows up on its own within a day. Daily and weekly partitions are created automatically, so the system keeps its own tables healthy as data grows. We’ve only just gotten started and will continue investing to make Agent Analytics more performant.

Included in every plan

We know how important it is to measure AI products so every Amplitude plan, including starter (free), gets a default allocation to use Agent Analytics.

Your monthly agent session allowance scales by tier: 5,000 on Free and Plus, 10,000 on Growth, and 20,000 on Enterprise. Take advantage of the full evaluation stack that ships at every tier, including out-of-the-box signals, trace analysis, semantic filtering, topic clustering. Define custom LLM evaluators (evals), calibration runs, and Bring Your Own Key (BYOK) for LLM calibration runs and custom evals. Growth and Enterprise extend agent session chat retention from 14 to 30 days, along with role-based access controls at Enterprise plan tier.