The Evolution of Experiment Tracking

There is a definite need for prompt tracking, but I don't know that experiment tracking is going to 100% go away. It still feels like there's a ton of use cases for that and so how are you looking at both of those? So again, I am looking from jobs to be done perspective. But when comes to experiment tracking, let's let's discuss what it is about.

Play episode from 30:19

chevron_right

Transcript

chevron_right

Transcript

Episode notes

MLOps Coffee Sessions #168 with Piotr Niedźwiedź, Experiment Tracking in the Age of LLMs, co-hosted by Vishnu Rachakonda.

// Abstract

Piotr shares his journey as an entrepreneur and the importance of focusing on core values to achieve success. He highlights the mission of Neptune to support ML teams by providing them with control and confidence in their models. The conversation delves into the role of experiment tracking in understanding and debugging models, comparing experiments, and versioning models. Piotr introduces the concept of prompt engineering as a different approach to building models, emphasizing the need for prompt validation and testing methods.

// Bio

Piotr is the CEO of neptune.ai. Day to day, apart from running the company, he focuses on the product side of things. Strategy, planning, ideation, getting deep into user needs and use cases. He really likes it.
Piotr's path to ML started with software engineering. Always liked math and started programming when he was 7. In high school, Piotr got into algorithmics and programming competitions and loved competing with the best. That got him into the best CS and Maths program in Poland which funny enough today specializes in machine learning.
Piotr did his internships at Facebook and Google and was offered to stay in the Valley. But something about being a FAANG engineer didn’t feel right. He had this spark to do more, build something himself. So with a few of his friends from the algo days, they started Codilime, a software consultancy, and later a sister company Deepsense.ai machine learning consultancy, where he was a CTO.
When he came to the ML space from software engineering, he was surprised by the messy experimentation practices, lack of control over model building, and a missing ecosystem of tools to help people deliver models confidently.
It was a stark contrast to the software development ecosystem, where you have mature tools for DevOps, observability, or orchestration to execute efficiently in production. And then, one day, some ML engineers from Deepsense.ai came to him and showed him this tool for tracking experiments they built during a Kaggle competition (which they won btw), and he knew this could be big.
He asked around, and everyone was struggling with managing experiments. He decided to spin it off as a VC-funded product company, and the rest is history.

// MLOps Jobs board

jobs.mlops.community

// MLOps Swag/Merch

https://mlops-community.myshopify.com/

// Related Links

https://neptune.ai/blog/author/piotr-niedzwiedz

https://www.youtube.com/playlist?list=PLKePQLVx9tOfKFbg9GY2Anl41Be4T1-m5
https://thesequence.substack.com/p/-piotr-niedzwiedz-neptunes-ceo-on

https://open.spotify.com/episode/2KEqTMAHODbPKdUEtlrhm7?si=ed862b2ac7534e39
https://www.linkedin.com/in/piotrniedzwiedz/

--------------- ✌️Connect With Us ✌️ -------------

Join our Slack community: https://go.mlops.community/slack

Catch all episodes, blogs, newsletters, and more: https://mlops.community/

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/