How Does Testing Slash Play Into Regulations?

Machine learning and a i, as they like to call it in the e u. We are working with ames that are facing regulations and independent regulators and independent auditors into their process. And then easily after that, of your doing this, it's so easy to create specific reports for them. Beas we have already, that the results he create specific reports and custom fors for er disorganizations. The way the ft portss and open sors look at the reports. They care a lot about these, what we call specks.

Play episode from 28:20

chevron_right

Transcript

chevron_right

Transcript

Episode notes

MLOps Coffee Sessions #76 with Mohamed Elgendy, Build a Culture of ML Testing and Model Quality.

Get the newsletter: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTNewsletter⁠⁠

// Abstract
Machine learning engineers and data scientists spend most of their time testing and validating their models’ performance. But as machine learning products become more integral to our daily lives, the importance of rigorously testing model behavior will only increase.

Current ML evaluation techniques are falling short in their attempts to describe the full picture of model performance. Evaluating ML models by only using global metrics (like accuracy or F1 score) produces a low-resolution picture of a model’s performance and fails to describe the model's performance across types of cases, attributes, and scenarios.

It is rapidly becoming vital for ML teams to have a full understanding of when and how their models fail and to track these cases across different model versions to be able to identify regression. We’ve seen great results from teams implementing unit and functional testing techniques in their model testing. In this talk, we’ll cover why systematic unit testing is important and how to effectively test ML system behavior.

// Bio
Mohamed is the Co-founder & CEO of Kolena and the author of the book “Deep Learning for Vision Systems”. Previously, he built and managed AI/ML organizations at Amazon, Twilio, Rakuten, and Synapse. Mohamed regularly speaks at AI conferences like Amazon's DevCon, O'Reilly's AI conference, and Google's I/O.

--------------- ✌️Connect With Us ✌️ -------------
Join our Slack community: https://go.mlops.community/slack
Follow us on Twitter: @mlopscommunity
Sign up for the next meetup: https://go.mlops.community/register
Catch all episodes, blogs, newsletter, and more: https://mlops.community/

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/
Connect with Adam on LinkedIn: https://www.linkedin.com/in/aesroka/
Connect with Mohamed on LinkedIn: https://www.linkedin.com/in/moelgendy/

Timestamps:

[00:00] Takeways

[04:41] Why do ML Testing?

[08:41] Kolena's main goal

[09:41] Difference of ML Testing from others

[13:12] Importance of a knowledge base in the organization

[17:53] Computational cost issues from testing

[20:48] Convincing people to do more testing

[23:13] Testing resources recommendations

[25:15] How to get good at testing

[28:19] Dealing with ML regulations