How to Build Reliable AI Agents with Datasets, Experiments, and Error Analysis

Yujohn from Mastra explains why datasets and experiments are essential for building production-grade AI agents.

If you're building an agent, you need a way to verify it's working correctly before and after you make changes. Datasets provide that baseline. You create a collection of test cases (ground truth) that represent the scenarios your agent should handle. Then you run experiments: pass each test case through your agent and measure the results. This is error analysis in practice. You start by identifying where your agent fails, then build scorers to quantify those failure modes over time. Smaller teams often ship first and add datasets later, once they have user feedback. Larger teams need them earlier. But eventually, every production agent needs this.

The demo shows how Mastra makes this accessible. You can create datasets through the UI, add items manually or import from CSV, and run experiments with a single click. The results show you exactly what went wrong: which tool calls failed, what the agent output was, and how it compared to ground truth. You can also compare experiments side by side to see if your prompt tweaks actually improved things. And because all the data lives in your own database, you can write your own agents to analyze the results, dig into traces, and iterate. The SDK makes it easy to integrate into CI/CD: run experiments on pull requests, gate deployments on eval scores, or just collect data from production and curate datasets later.

🔗 RESOURCES
Mastra Datasets docs: https://mastra.ai/docs/observability/datasets
Running Experiments: https://mastra.ai/docs/observability/datasets/running-experiments
Mastra GitHub: https://github.com/mastra-ai/mastra
Yujohn on X: https://x.com/YujohnNatt
Mastra Discord: https://discord.gg/mastra

AI Agents Hour is a weekly livestream hosted by Mastra CPO Shane Thomas and CTO Abhi Aiyer. Airing Mondays at 12PM Pacific on YouTube and X, the show covers breaking AI news, agent development techniques, and features interviews with industry experts building AI applications today.

📚 MASTRA RESOURCES
Mastra: https://mastra.ai
Learn Mastra in the world's first MCP-Based Course: https://mastra.ai/course
Principles of Building AI Agents (Book): https://mastra.ai/book
Patterns for Building AI Agents (New Book): https://mastra.ai/books/patterns-of-building-ai-agents

MASTRA?
Mastra is an open-source TypeScript framework designed for building and shipping AI-powered applications and agents with minimal friction. It supports the full lifecycle of agent development—from prototype to production. You can integrate it with frontend and backend stacks (e.g., React, Next.js, Node) or run agents as standalone services. If you’re a JavaScript or TypeScript developer looking to build an agentic or AI-powered product without starting from first principles, Mastra provides the scaffolding, tools, and integrations to accelerate that process.

00:00 – Intro
00:48 – What are Datasets and Experiments
01:55 – Error Analysis
03:35 – When to Use Datasets (Team Size Matters )
05:43 – Demo: Creating a Dataset
07:04 – Demo: Ground Truth
07:53 – Demo: Running Experiments
09:34 – Demo: Comparing Results
11:00 – Your Data, Your Database
12:24 – SDK & CI Integration
14:30 – Collecting Data from Production