689: Observing LLMs in Production to Automatically Catch Issues
Super Data Science: ML & AI Podcast with Jon Krohn
00:00
Exploring Phoenix: Monitoring Machine Learning Models in Production
In this chapter, the focus is on Phoenix, a tool for monitoring machine learning models in production, without the need for additional infrastructure like GPUs. Phoenix enables ad-hoc monitoring to detect model drift and issues in production data by comparing production embeddings to training embeddings. The conversation delves into the significance of comparing distributions in various scenarios and highlights the value of Phoenix in ensuring ML observability in different use cases.
Play episode from 31:01
Transcript
