Designing a Team of Agents & Eval Metrics

What if the best way to lead product is to build it yourself first?

In this episode of Supra Insider, Marc Baselga and Ben Erez sit down with Chase Schwalbach, SVP of Product and Technology at Millie, to unpack a radically different approach to product leadership. Despite his title, Chase spent months as an IC, rolling up his sleeves to build healthcare infrastructure, teach himself AI eval systems, and ship a sophisticated patient chatbot, all before bringing his team in. He explains why shielding the team from early-stage messiness, moving at speed, and feeling the pain yourself leads to better products.

They explore how Chase built a team of AI agents (supervisor + specialized sub-agents) from scratch, why treating prompts like deterministic code requires extreme precision, and how he taught himself evals through pure iteration. Plus, the converging worlds of PM and engineering, why technical PMs and product-minded engineers are becoming the same role, why handoffs kill velocity in an AI-native world, and what “context engineering” actually means when your codebase needs to work for both humans and AI agents.

If you’re a product leader wondering whether to get more hands-on, an engineer considering the jump to PM (or vice versa), or building AI systems in regulated industries like healthcare, this episode is for you.

All episodes of the podcast are also available on Spotify, Apple and YouTube.

New to the pod? Subscribe below to get the next episode in your inbox 👇

Want company-specific PM interview intel?

If you’re a PM preparing for interviews at Uber, Figma, Stripe, DoorDash and OpenAI… we think you’re going to love Insider Loops. You can read about it in Ben’s LI post, Marc’s LI post or check out the site directly below.

In this episode, we covered the following topics:

* (02:15) The personal story that led Chase to Millie

* (06:40) The forward deployed PM story begins, spending the first 6 months building healthcare infrastructure

* (09:12) Building a dual technology layer: modern tools in front of Epic instead of integrating directly

* (16:45) The infrastructure had to exist first before experimenting, no sandbox environments in traditional healthcare

* (24:55) How Chase taught himself to run evals, via trying, retrying and getting it right

* (32:22) What surprised Chase most: prompting intent is incredibly hard, every single word matters

* (36:25) The most important moat is speed, infrastructure that allows you to move things quickly

* (44:30) The infrastructure of coding is moving from pure code to English, AI as the new interface

* (53:45) Human + AI is stronger than AI alone (the chess model)

* (57:00) What Chase optimizes for: misfits who don’t follow rules and always think “there’s gotta be a better way”

* (01:00:20) The AI rule paradox: we want people who don’t follow rules, but they need to set rules for AI

* (01:07:25) Gratitude Corner: Una Pipic, his partner at a previous startup who told him to stay a misfit

* And more!

Links:

* Chase’s post on AI Evals: https://www.linkedin.com/pulse/moving-fast-without-breaking-trust-millie-chase-schwalbach-q0tde

* Millie: https://www.millieclinic.com/

* Una Pipic: https://www.linkedin.com/in/upipic/

* Chase Schwalbach: https://www.linkedin.com/in/schwalbach/

* Marc: https://www.linkedin.com/in/marcbaselga/

* Ben: https://www.linkedin.com/in/benerez/

If you’re a new subscriber, we encourage you to check out some of the recent episodes you might have missed:

To support the podcast, please check out the links below:

* Supra has teamed up with Maven to bring you something special – courses that our own community members have personally curated. And because you’re part of the Supra family, you can get up to 35% in some of these handpicked selections with code SUPRAxMAVEN.

* Preparing for Product Sense & Analytical Thinking interviews? Check out Ben’s AI Practice Copilot and top-rated Maven course: Product Sense & Analytical Thinking Bootcamp for Meta-Style PM Interviews (selected by Lenny as a top course in product 🔥). Get 25% off the course with promo code `suprainsider` (auto-applied here).

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit suprainsider.substack.com