How Do You Do Testing for M L?

MLOps Coffee Sessions #79 with Orr Shilon, Platform Thinking: A Lemonade Case Study.  

Join the Community: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTJoinIn⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠

Get the newsletter: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTNewsletter⁠⁠

// Abstract
This episode is the epitome of why people listen to our podcast. It’s a complete discussion of the technical, organizational, and cultural challenges of building a high-velocity, machine learning platform that impacts core business outcomes.   
Orr tells us about the focus on automation and platform thinking that’s uniquely allowed Lemonade’s engineers to make long-term investments that have paid off in terms of efficiency. He tells us the crazy story of how the entire data science team of 20+ people was supported by only 2 ML engineers at one point, demonstrating the leverage their technical strategy has given engineers.  

// Bio
Orr is an ML Engineering Team Lead at Lemonade, currently working on an ML Platform, empowering Data Scientists to manage the ML lifecycle from research to development and monitoring.  

Previously, Orr worked at Twiggle on semantic search, at Varonis on data governance, and at Intel. He holds a B.Sc. in Computer Science and Psychology from Tel Aviv University.  
Orr also enjoys trail running and sometimes races competitively.  

--------------- ✌️Connect With Us ✌️ -------------
Join our Slack community: https://go.mlops.community/slack
Follow us on Twitter: @mlopscommunity
Sign up for the next meetup: https://go.mlops.community/register
Catch all episodes, blogs, newsletter, and more: https://mlops.community/

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/
Connect with Vishnu on LinkedIn: https://www.linkedin.com/in/vrachakonda/
Connect with Orr on LinkedIn: https://www.linkedin.com/in/orrshilon/

Timestamps:

[00:00] Takeaways

[05:31] Introduction to Orr Shilon

[06:00] Looking for editors

[07:29] What is Lemonade and what does it do?

[08:43] Machine Learning in Lemonade

[10:06] End-to-end process of ML in Lemonade

[13:00] Recycling features

[14:52] Slack bot, Cooper

[16:15] Importance of automation in Lemonade

[18:11] Circumstances when automation is unnecessary

[19:44] Slack bot platform

[20:31] ML tools used by Lemonade

[23:45] Areas of friction

[26:15] Model serving framework

[28:30] Ownership models

[30:36] Facing challenges

[31:52] Theory about lack of talents

[34:13] Orr Shilon's team

[37:51] Continuation of the building blocks

[39:12] Testing challenges

[42:30] Clear vision of the Lemonade team

[44:46] Demetrios was ghosted by the head of data science at Lemonade

[45:46] Platform thinking

[47:27] What is a "Business point in time"?

[50:24] Wrap up