Gideon Mendels

Gideon Mendels, CEO and co-founder of Comet, dives into the intricate world of testing and evaluating LLMs. He discusses the hybrid approach required for these applications, merging machine learning with software engineering best practices. Topics include innovative methods for evaluating LLMs beyond traditional metrics, the challenge of unit testing with deterministic assertions, and the importance of experiment tracking in ensuring reproducibility. Gideon also highlights the role of user interaction analysis in enhancing LLM applications' performance.

Gideon Mendels, Co-founder and CEO of Comet who built ML systems at Google, talks about building and evaluating LLM-powered agents. He covers why non-determinism breaks traditional testing. He explains evals as test suites, bootstrapping regressions from production, and treating prompts, tools, and configs as optimization/search problems.

Gideon Mendels, co-founder and CEO of Comet, dives into the fascinating world of AI with a focus on Opik, his open-source model evaluation platform. He shares how Opik's rise exceeded expectations and emphasizes the critical role of CI/CD in AI development. Gideon also discusses the alarming decline in dedicated machine learning engineers and differentiates between genuine and 'fake' open-source solutions. The conversation wraps up with insights on the evolving AI landscape and the need for organizations to adapt to new evaluation methodologies.

Gideon Mendels, Co-founder and CEO of Comet ML, discusses the intersection of machine learning and software engineering. Topics include model performance evaluation, monitoring ML models in production, basics of machine learning for software engineers, and building an effective machine learning team.

In this podcast, the CEO of Comet discusses challenges of integrating ML models into production and the importance of tracking and optimizing experiments. The discussion also explores the rising adoption of AI in various industries and the benefits of upskilling software engineers in machine learning.