How to evaluate AI agents when every run looks different

TL;DR: Because an AI agent can give a different result every run, one good result tells you very little, so my team and I run eval testing at scale and allow all other teams at Miro to do the same. We’re mainly building the infrastructure to make this happen, but we also help the teams turn their use cases into a testable dataset, which we run every night. Over the last couple of months, the internal AI discussions moved from “I hope it works” to debating which model performs best for which task.

You prompt Miro Sidekick, and it builds something great for you immediately. Then you run the same prompt again, and the result looks completely different. If you build AI agents or just work with AI in general, like Claude or ChatGPT, you’ve probably lived this.

I’m one of the product leads in the AI stream at Miro, and my team builds the tools and practices we use to evaluate our AI agents, like Sidekick, Flows, and everything that comes in through Miro MCP (our Model Context Protocol server). I joined in April after six years running my own startup. Since then, I’ve worked together with the team on making AI quality visible and understandable, and on giving teams tools to improve it and see the improvement.

Evaluations, or evals, are repeatable tests that show how well an AI agent does its job. At Miro, we’ve gone from developers pasting prompts into Sidekick to automated tests every night.

Why vibe testing stops working

When I joined, almost everyone tested Sidekick or Flows the same way: we opened Miro, pasted something into Sidekick, and looked at what it produced. This is called vibe testing, and it’s a good practice to start with because you learn how the agent reacts.

It stops working once the stakes grow. Ask Sidekick to produce a complete workshop from scratch, and a million things can go wrong. The system is non-deterministic, so even a result that goes right once can go wrong the other 99% of the time.

Our setup adds another layer. Several teams work on the same thing, and each one is responsible for one part of how Sidekick and Flows work. Each team runs tests on its own piece, while our team looks at the whole thing end to end and helps the others set up and work with evals.

The public launch of the new Sidekick raised the stakes again. We had a couple hundred thousand people using Sidekick, and with millions of users, the picture can look completely different. We needed to understand why one user or user group might see something different from another. The launch made a reliable way to understand what’s happening even more urgent. Today, I open my laptop each morning, check the numbers, and see whether we improved or something went wrong. I can also easily read through a couple of real conversations to get more qualitative insights on how well a given task has been accomplished by AI (completely anonymized, and our focus is task completion). Especially for this, we’re currently working on a secure, practical way for everyone at Miro to get these insights to inform their work.

Test like a user, then check how the agent talks

To set up an experiment, you usually define a dataset that contains different real use cases, like diagramming or building and editing a slide deck. It includes an example prompt a user could put into Sidekick and the context around that person, such as whether they’re a mature or first-time user and what profession they have.

From there, an agent imitates the user. A real conversation rarely ends after one question and one answer, so we run multiple prompts within one chat. If Sidekick is missing information, it can ask questions, and the user-imitating AI agent can pick the right option based on its context. Different eval runs at Miro can include around 100 evals. For example, our golden dataset, a curated reference set of test cases, has 78 evals that cover a broad range of use cases. We compare the experiment results with runs from yesterday or from a week ago, to see whether we improved.

In terms of tooling, we mainly use Braintrust as it is one of the leading products for running evals. But since its capabilities are very text-focused, we had to come up with our own custom platform on top that allows us to do way more visual evaluations.

We’re very much interested in what the agent produces on the boards as well as how the agent replies. To be able to have judgment at scale, we rely on our judges: automated reviewers that score the agent’s answers and output. They help us to better understand, for example, how the agent communicates and structures its replies. We sat down with the copy and content teams in our stream and came up with about 100 checks on tone of voice, accessibility of language, tone of language, and how we want Miro AI to sound.

To go deeper into visual evaluation, we had to build ways to compare a Miro board before Sidekick acts with the board after. We’re mainly interested in the composition, structure, layout, and also the visual appeal of what it produces on the board.

Build results for people

To make sure that we’re building with and for our internal teams, we started by interviewing 20 to 30 colleagues, including developers, designers, and product people. We asked how they would like to use evals, where they struggle today, and what they care about the most when they look at results.

Based on that research, we built a first version of tooling on top of Braintrust. Everybody at Miro can now run evals without writing code and decide where and how they are being run, for example with different product setups or large language models (LLMs).

Our custom tooling gives you three important things that were not easily possible before:

• A screenshot of the board’s end state.
• A score from 0 to 10 for overall completion, covering how the task went, how the board looks, and how the layout works. Our internal aim is to get quality to 8 out of 10.
• Access to all the boards, so you can skim through many runs and compare them.

We’ve also heavily invested in traces to better understand where real usage falls short at the moment, and this is a great source for our dataset use cases. In general, a trace is the string of one conversation, and each thing the agent does inside of it is called a span. Traces can show why an agent decided to do something and what it did, including every tool call and what the user said. Together with evals we get a complete picture of how well our AI capabilities perform.

What evals changed for Sidekick

The conversations on our teams have changed from “I hope it works” to debating which models to use for which kind of task, and then iterating to improve them. For example, we can run the same dataset four times, with two Anthropic models and two OpenAI models. Because every model has to do the same task, we can easily compare which one suits which job best.

That matters for enterprise customers because some only allow a certain LLM provider. Whatever the model, we want to make sure the quality stays the same or gets better.

To better understand the overall improvement and performance of our AI agents, we run a weekly session with the team on AI eval metrics. Over the last six months, as a rough example, the task completion rate for Sidekick rose from around 60% to more than 80%. It’s a quantitative change, but we can also see a clear qualitative difference. It tells us that what the teams do to improve the agent pays off. Today, you can use Sidekick every day for your daily work.

In early October, we ran a session with about 12 teams within Miro about how to test and then translate these tests into automated and recurring evals. Without the tooling in place and the work of the last few months, that wouldn’t have been possible.

Map your evals on a Miro board

Creating a comprehensive eval suite across multiple teams requires a lot of communication with other teams, so we started to map out the respective teams on a Miro board. We mapped out dependencies, and each team added the evals and tests it runs.

From there, we used Miro MCP to get the data from the board in no time and take all the use cases from it. With a Claude skill that I’ve created, it now takes two minutes to go from a use case to an eval in YAML, the file format our evals use. There’s also the possibility to upload the evals to Braintrust via MCP.

So what lives in Miro as use cases goes almost automatically through to Braintrust, where it populates a dataset for me.

When we have a new use case, I just tell Claude to look at the new board and run it again. Going back five years, I would have needed to type everything myself. Now I can create an overview of all the evaluations we’re running in a few minutes, so teams can see where they stand.

If your evals depend on several teams, start by mapping those teams and their test cases on a shared board. Want to try it? Open a free Miro board, add the teams you depend on, and have each one drop in the use cases they test. With Miro MCP, an AI assistant can turn that board into eval datasets. Sign up for free and start mapping your evals today.

This work is a team effort. Thanks to my team: Konstantin, Mattia, Sergey, and Cole.

Originally published on the Miro blog.