ML Tests // Svet Penkov // Coffee Sessions #61
MLOps.community
The Challenges in Machinea Testing?
The amount of data that we have, is too much for trying to go through these tests. We should be talking much more as a community and have a very grounded discussion about what it means. I think this is really thet thet the biggest changees ahead of us.
MLOps Coffee Sessions #60 with Svet Penkov, ML Tests.
Get the newsletter: https://go.mlops.community/YTNewsletter
// Abstract
How confident do you feel when you deploy a new model? Does improving an ML model feel like a game of "whack-a-mole"? ML is taking over all sorts of industries, and yet ML testing tools are virtually non-existent.
Drawing parallels from software engineering and electronic circuit board design to the aviation and semiconductor industries, the need for principled quality assurance (QA) steps in the MLOps pipeline is long overdue. Let's talk about why ML testing is hard, what we can do about it, and what place should ML QA take in the future.
// Bio
Svet has been building robots ever since he was a kid. At some point, Svet got interested in not just how to build them, but actually how to make them think, and so he did a Ph.D. in AI & Robotics at the University of Edinburgh, UK. Towards the end of Svet's Ph.D., he joined FiveAI as a Research Scientist and led the motion prediction team for 3 years.
Throughout his career, Svet spent endless hours fixing model regressions and fighting with edge cases, and so at some point, he had had enough of it and decided it was time to do something about it. That's how Svet started Efemarai, where they are building a platform for testing and improving ML continuously.
// Relevant Links
--------------- ✌️Connect With Us ✌️ -------------
Join our Slack community: https://go.mlops.community/slack
Follow us on Twitter: @mlopscommunity
Sign up for the next meetup: https://go.mlops.community/register
Catch all episodes, Feature Store, Machine Learning Monitoring, and Blogs: https://mlops.community/
Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/
Connect with Svet on LinkedIn: https://www.linkedin.com/in/svpenkov/
Timestamps:
[00:00] Introduction to Svet Penkov
[02:10] Svet's background in tech
[04:34] Testing on robotics vs areas of machine learning
[05:21] What's missing in testing right now?
[08:56] Who should test?
Step 1. Figuring out the requirements
[12:04] Edge cases
Steps 2. Access to variation
[13:29] Step 3. Validation and Verification
[16:15] New challenges that need to be addressed
[18:25] Test-driven development viability argument
[20:26] Software engineering tests vs machine learning engineering tests
[23:23] Rule of tools in MLOps
[26:15] Figuring out the difficulty in designing the API's
[27:48] Svet's vision for the future
[29:15] Moving goal post
[31:00] 10 data points being realistic
[31:27] Getting less
[32:20] Efemarai: Where did it come from and why?
[33:53] Efemarai - Functional Magnetic Resonance Imaging
[35:21] A perfect world journey
[36:22] Value of tests
[37:55] Get ready for the MLOps Community Slack testing channel!
The AI-powered Podcast Player
