Managing an agentic workforce and calibration

What if the most important skill for building AI products has nothing to do with evals, technical background, or knowing how to write a prompt? What if it is the ability to design systems that can handle what you never planned for?

In this episode of Supra Insider, Marc Baselga and Ben Erez sit down with Apurva Garware, who has built and scaled products across Amazon, Microsoft, and Upwork, to make the case that systems thinking is the defining skill of the next era of product management. Apurva explains why non-determinism forces PMs to stop thinking in features and start designing the guardrails, agent contracts, and escalation points that govern how a system behaves at runtime, when no one is watching.

They explore a three-phase framework for governing AI systems across design, deployment, and production; heuristics for deciding what to hand to agents versus escalate to humans; and a sharp insight about the two products every AI-native company is actually building: the customer-facing product, and the internal operational system that drives margin and velocity. Marc and Ben also share their own experience calibrating an agentic workflow at Supra, grounding the conversation in practice.

If you are a PM trying to find your footing in the AI era without a deeply technical background, a founder wrestling with when to reach for AI versus simpler deterministic automation, or a product leader who wants to build more discipline into how your team ships AI products, this episode is for you.

All episodes of the podcast are also available on Spotify, Apple and YouTube.

New to the pod? Subscribe below to get the next episode in your inbox 👇

Want company-specific interview intel?

If you’re preparing for PM interviews at Meta, Anthropic, OpenAI, Google, Stripe, DoorDash, Uber, or Figma… we think you’re going to love Insider Loops. You can read about it in Ben’s LI post, Marc’s LI post or check out the site directly below.

In this episode, we covered the following topics:

* (02:13) Apurva’s Amazon story: the Beyoncé/Lemonade moment that exposed how a perfectly functioning algorithm failed due to missing broader context

* (03:47) Why linear thinking is not enough for AI products and why non-determinism requires a fundamentally different mindset

* (07:55) Ben’s working definition of systems thinking: never building anything in isolation, thinking in 3D maps, and setting the laws of physics that govern how a system operates

* (10:05) The NASA control room analogy: what it actually means to govern AI agents at scale with confidence

* (11:15) Three phases of AI system governance: design-time contracts, deployment benchmarks, and runtime production escalations

* (17:20) The 49/51 decision rule: a heuristic for knowing which decisions to escalate to humans and which to trust the system to handle

* (21:25) The “two products” insight: every AI-native company is simultaneously building a customer-facing product and an internal operational product, and most are treating the second one as an afterthought

* (26:26) Rule of thumb: if a deterministic automation can do the job, build that instead of reaching for an AI agent

* (32:33) How to start developing systems thinking: automate personal workflows, follow industry thought leaders, and stay curious even when your day job does not involve AI products

* (39:05) The biggest failure state: trying to build the entire system end-to-end from day one, and why scoping is more important than ever

* (41:00) The pre-mortem approach: scoping what good looks like, then mapping every way it could break before you build

* (51:00) Outsourcing thinking versus reasoning: why PMs must keep their own reasoning intact even as AI handles more of the execution

* (56:11) Gratitude corner: mentors, product communities including Supra, and past teams

* And more!

Links:

* Andrew Ng: https://www.linkedin.com/in/andrewyng/

* Dario Amodei: https://www.linkedin.com/in/dario-amodei-3934934/

* Sam Altman: https://x.com/sama?lang=en

* Aaron Levie: https://www.linkedin.com/in/boxaaron/

* Andrej Karpathy: https://www.linkedin.com/in/andrej-karpathy-9a650716/

* Ilya Sutskever: https://www.linkedin.com/in/ilya-sutskever/

* John Hegeman: https://www.linkedin.com/in/john-hegeman-02137b7/

* Supra: https://www.joinsupra.com/

* Products that count: https://productsthatcount.com/

* The Skip: https://theskip.substack.com/

* Apurva Garware: https://www.linkedin.com/in/apurvacg/

* Marc: https://www.linkedin.com/in/marcbaselga/

* Ben: https://www.linkedin.com/in/benerez/

If you’re a new subscriber, we encourage you to check out some of the recent episodes you might have missed:

To support the podcast, please check out the links below:

* Check out Ben’s AI Practice Copilot and top-rated Maven course: Self-Paced PM Interview Bootcamp: Product Sense & Analytical Thinking (selected by Lenny Rachitsky as a top course 🔥). Get 25% off the course with promo code suprainsider (auto-applied here).

* Supra has teamed up with Maven to bring you something special – courses that our own community members have personally curated. And because you’re part of the Supra family, you can get up to 35% in some of these handpicked selections with code SUPRAxMAVEN.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit suprainsider.substack.com