The Stack Overflow Podcast How can you test your code when you don’t know what’s in it?
12 snips
Mar 31, 2026 Fitz Nowlan, VP of AI and Architecture at SmartBear and co-founder of Reflect, talks about testing in an era of agentic, non-deterministic systems. He explores why MCP-style agents break traditional testing, the rise of workflow-skeletons and LLM-driven evals, the shifting value of source code and data locality, and how on-prem and local agents reshape testing strategies.
AI Snips
Chapters
Transcript
Episode notes
Fitz Nowlan Career From Java Class To SmartBear VP
- Fitz Nowlan started programming in high school, studied CS at Georgetown, then earned a PhD at Yale focusing on low latency networking.
- He co-founded Reflect for automated cloud web testing and was acquired by SmartBear in 2024.
MCP Makes Workflows Non Deterministic
- MCP introduces non-determinism because LLMs choose tool invocations at runtime.
- Fitz Nowlan explains that leaving workflow decisions to the model enables intelligent behavior but complicates testing.
Combine Skeleton Tests With LLM Evals
- Use two testing approaches: skeleton workflow tests and open-ended evals that use LLMs to judge LLM outputs.
- Create inputs to nudge the model down expected tool sequences and run evals to capture broader behavior.