Just Now Possible Debugging AI Products: From Data Leakage to Evals with Hamel Husain
43 snips
Oct 2, 2025 
Hamel Husain, a machine learning engineer with over 25 years of experience at GitHub and Airbnb, dives deep into the intricacies of debugging AI products. He shares insights from his work on forecasting Airbnb guest growth, highlighting challenges like data leakage. The conversation uncovers techniques for error analysis in machine learning, the importance of synthetic data, and the pitfalls of AI-generated outputs like hallucinations. Hamel emphasizes the need for systematic improvement and presents practical tips for enhancing AI evaluations.
AI Snips
Chapters
Transcript
Episode notes
ML Is Prediction, Not Magic
- Machine learning is fundamentally predictive: classification, recommendation, forecasting.
- The core challenge is making models generalize to unseen data, not complexity of algorithms.
Nurture Boss: An AI Leasing Assistant
- Nurture Boss built an AI leasing assistant for apartment management with tools and SMS/voice.
- The small startup launched agentic features like tour scheduling and resident communications.
Use Targeted Synthetic Data To Test Edges
- Use synthetic data targeted at observed errors to explore boundaries and frequency.
- Generate variations (formats, vagueness, edge cases) and run them as test cases against your agent.