Vanishing Gradients Episode 58: Building GenAI Systems That Make Business Decisions with Thomas Wiecki (PyMC Labs)
17 snips
Sep 9, 2025 Thomas Wiecki, founder of PyMC Labs and co-author of PyMC, dives into how generative AI can shape business decisions. He discusses using large language models as synthetic consumers to test product ideas, revealing the efficiency of AI over traditional surveys. Thomas emphasizes Bayesian modeling's role in providing trustworthy insights and navigating complex data. His experience with Colgate highlights the iterative design of AI systems for better product and marketing strategies, urging a balance between innovative models and reliability.
AI Snips
Chapters
Transcript
Episode notes
Demographics Emerge In LLM Outputs
- Off-the-shelf LLMs (GPT-4o, Gemini) reproduced demographic effects like age and income without being trained on the survey.
- Some models performed much better than others, so model choice matters.
Design Guardrailed Agentic Loops
- Build closed-loop agentic workflows by splitting tasks into specialist agents with tight guardrails.
- Start simple and iteratively add flexibility, combining generation, critique, and data modeling agents.
Retain Human Sanity Checks
- Keep humans in the loop and set clear sanity-check thresholds for autonomous AI loops.
- Use human or AI evaluators to catch failure modes before systems act on business decisions.