MLOps.community

Coffee Sessions #54 with Niall Murphy, Machine Learning SRE.Join the Community: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTJoinIn⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Get the newsletter: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTNewsletter⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠// AbstractSRE is making its way into the machine learning world. Software engineering for machine learning requires reliability, performance, and maintainability. Site reliability engineering is the field that deals with reliability and ensuring constant, real-time performance. Niall Murphy, most recently Global Head of SRE at Microsoft Azure, helps us understand what SRE can do for modern ML products and teams.Building machine learning teams requires a diverse set of technical experiences, and Niall shares his thoughts on how to do that most effectively. Machine learning organizations need to start to take advantage of SRE best practices like SLOs, which Niall walks through. Production machine learning depends on high-quality software engineering, and we get Niall's take on how to ensure that in a machine learning context.// BioNiall Murphy has been interested in Internet infrastructure since the mid-1990s. He has worked with all of the major cloud providers from their Dublin, Ireland offices - most recently at Microsoft, where he was global head of Azure Site Reliability Engineering (SRE). His books have sold approximately a quarter of a million copies worldwide, most notably the award-winning Site Reliability Engineering, and he is probably one of the few people in the world to hold degrees in Computer Science, Mathematics, and Poetry Studies. He lives in Dublin, Ireland, with his wife and two children.--------------- ✌️Connect With Us ✌️ -------------Join our Slack community: https://go.mlops.community/slackFollow us on Twitter: @mlopscommunitySign up for the next meetup: https://go.mlops.community/registerConnect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/Connect with David on LinkedIn: https://www.linkedin.com/in/aponteanalytics/Connect with Vishnu on LinkedIn: https://www.linkedin.com/in/vrachakonda/Connect with Niall on LinkedIn: https://www.linkedin.com/in/niallm/Timestamps: [00:00] Introduction to Niall Murphy [00:36] SRE background to Machine Learning space transition [07:10] SLO's being a challenge in the ML space [09:42] SRE Hiring Investments [15:10] Behavior of teams concept [17:45] Challenges dealing with ML production [18:27] Update on Reliable Machine Learning book [22:46] Monitoring [25:05] Difference between ML and SRE [29:18] Incident response in Machine Learning [34:46] Rollbacks [35:50] Machine Learning burden over time [42:42] Niall's journey to the SRE space and focus on developing himself

Coffee Sessions #53 with David Aponte, Demetrios Brinkmann, and Vishnu Rachakonda, MLOps Insights.Join the Community: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTJoinIn⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Get the newsletter: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTNewsletter⁠⁠⁠⁠⁠⁠⁠⁠⁠// AbstractMLOps Insights from MLOps community core organizers Demetrios Brinkmann, Vishnu Rachakonda, and David Aponte. In this conversation, the guys do a deep dive on testing with respect to MLOps, talk about what they have learned recently around the ML field, and what new things are happening with the MLOps community.// BioDavid AponteDavid is one of the organizers of the MLOps Community. He is an engineer, teacher, and lifelong student. He loves to build solutions to tough problems and share his learnings with others. He works out of NYC and loves to hike and box for fun. He enjoys meeting new people, so feel free to reach out to him!Demetrios BrinkmannAt the moment, Demetrios is immersing himself in Machine Learning by interviewing experts from around the world in the weekly MLOps.community meetups. Demetrios is constantly learning and engaging in new activities to get uncomfortable and learn from his mistakes. He tries to bring creativity into every aspect of his life, whether that be analyzing the best paths forward, overcoming obstacles, or building LEGO houses with his daughter.Vishnu RachakondaVishnu is the operations lead for the MLOps Community and co-hosts the MLOps Coffee Sessions podcast. He is a machine learning engineer at Tesseract Health, a 4Catalyzer company focused on retinal imaging. In this role, he builds machine learning models for clinical workflow augmentation and diagnostics in on-device and cloud use cases. Since studying bioengineering at Penn, Vishnu has been actively working in the fields of computational biomedicine and MLOps. In his spare time, Vishnu enjoys suspending all logic to watch Indian action movies, playing chess, and writing.Other Links:Continuous Delivery for Machine Learning article by Martin Fowler: https://martinfowler.com/articles/cd4ml.htmlTo Engineer Is Human book by Henry Petroski: https://www.amazon.com/Engineer-Human-Failure-Successful-Design/dp/0679734163----------- Connect With Us ✌️-------------   Join our Slack community: https://go.mlops.community/slackFollow us on Twitter: @mlopscommunitySign up for the next meetup: https://go.mlops.community/registerConnect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/Connect with David on LinkedIn: https://www.linkedin.com/in/aponteanalytics/Connect with Vishnu on LinkedIn: https://www.linkedin.com/in/vrachakonda/Timestamps: [00:14] Tests and how to do tests in MLOps[09:10] Learning from Vishnu and David's new job[12:42] How will it change?[19:48] Forcing to do the right thing vs allowing to do the wrong thing[21:54] Dealing with Machine Learning Models and Data[25:10] Feature store and monitoring compare page

Coffee Sessions #52 with Dave Bergstein, Vector Similarity Search at Scale.Join the Community: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTJoinIn⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Get the newsletter: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTNewsletter⁠⁠⁠⁠⁠⁠⁠⁠⁠// Abstract Ever wonder how Facebook and Spotify now seem to know you better than your friends? Or why the search feature in some products really “gets” you, while in other products it feels stuck in the '90s? The difference is vector search— a method of indexing and searching through large volumes of vector embeddings to find more relevant search results and recommendations.Dave Bergstein, the Director of Product at Pinecone, joins us to describe how vector search is used by companies today, what the challenges of deploying vector search to production applications are, and how teams can overcome those challenges even without the engineering resources of Facebook or Spotify.// Bio Dave Bergstein is Director of Product at Pinecone. Dave previously held senior product roles at Tesseract Health and MathWorks, where he was deeply involved with productionizing AI. Dave holds a Ph.D. in Electrical Engineering from Boston University, studying photonics. When not helping customers solve their AI challenges, Dave enjoys walking his dog Zeus and CrossFit.--------------- ✌️Connect With Us ✌️ ------------- Join our Slack community: https://go.mlops.community/slackFollow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/registerConnect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/Connect with Vishnu on LinkedIn: https://www.linkedin.com/in/vrachakonda/Connect with Dave on LinkedIn: https://www.linkedin.com/company/pinecone-io/mycompany/Timestamps[00:00] Intro to Dave Bergstein [00:55] Dave’s tech background [04:33] Building software products [06:05] Building reliable systems [07:58] System complexity and testing [08:30] Pinecone intro [10:47] Vector Search explained [11:38] Zeus example [14:14] Vector Search use cases [16:55] Translation help [17:52] Notion on Vector Search [19:13] Common scenario [20:38] Engineering challenges [25:05] Live system updates [26:03] Compute cost challenges [26:35] Challenge comprehension [28:00] Security challenges [30:47] Importance of security [31:40] From imaging to ML [33:08] Lessons from building solo [33:38] Modern ML tooling [37:12] MLOps audience gap [39:10] Supporting diverse professionals [41:44] Openness in platforms [41:51] Benefits of in-house work [42:19] Ecosystem interoperability [43:04] Interoperability [45:10] Leveraging open ecosystem [45:40] Vector ecosystem evolution [47:40] Rise of Pinecone-like firms

Coffee Sessions #51 with Sahbi Chaieb, ML security: Why should you care?Join the Community: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTJoinIn⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Get the newsletter: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTNewsletter⁠⁠⁠⁠⁠⁠⁠⁠// AbstractSahbi, a senior data scientist at SAS, joined us to discuss the various security challenges in MLOps. We went deep into the research he found describing various threats as part of a recent paper he wrote. We also discussed tooling options for this problem that is emerging from companies like Microsoft and Google.// BioSahbi Chaieb is a Senior Data Scientist at SAS. He has been working on designing, implementing, and deploying Machine Learning solutions in various industries for the past 5 years. Sahbi graduated with an Engineering degree from Supélec, France, and holds an MS in Computer Science, specialized in Machine Learning from Georgia Tech.--------------- ✌️Connect With Us ✌️ -------------Join our Slack community: https://go.mlops.community/slackFollow us on Twitter: @mlopscommunitySign up for the next meetup: https://go.mlops.community/registerConnect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/Connect with Vishnu on LinkedIn: https://www.linkedin.com/in/vrachakonda/Connect with Sahbi on LinkedIn: https://www.linkedin.com/in/sahbichaieb/Timestamps: [00:00] Introduction to Sahbi Chaieb [01:25] Sahbi's background in tech [02:57] Inspiration for the article[09:40] Why should you care about keeping our model secure?[12:53] Model stealing [14:16] Development practices[17:24] Other tools in the toolbox covered in the article[21:29] Stories/occurrences where data was leaked[24:45] EU Regulations on robustness[26:49] Dangers of federated learning[31:50] Tooling status on model security [33:58] AI Red Teams[36:42] ML Security best practices [38:26] AI + Cyber Security [39:26] Synthetic Data [42:51] Prescription on ML Security in 5-10 years[46:37] Pain points encountered

Coffee Sessions #50 with Alex Chung and Srivathsan Canchi, Creating MLOps Standards.Join the Community: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTJoinIn⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Get the newsletter: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTNewsletter⁠⁠⁠⁠⁠⁠⁠// AbstractWith the explosion in tools and opinionated frameworks for machine learning, it's very hard to define standards and best practices for MLOps and ML platforms. Based on their building AWS SageMaker and Intuit's ML Platform, respectively, Alex Chung and Srivathsan Canchi talk with Demetrios and Vishnu about their experience navigating "tooling sprawl". They discuss their efforts to solve this problem organizationally with Social Good Technologies and technically with mlctl, the control plane for MLOps.// BioAlex ChungAlex is a former Senior Product Manager at AWS Sagemaker and an ML Data Strategy and Ops lead at Facebook. He's passionate about the interoperability of MLOps tooling for enterprises as an avenue to accelerate the industry.Srivathsan CanchiSrivathsan leads the machine learning platform engineering team at Intuit. The ML platform includes real-time distributed featurization, scoring, and feedback loops. He has a breadth of experience building high-scale mission-critical platforms. Srivathsan also has extensive experience with K8S at Intuit and previously at eBay, where his team was responsible for building a PaaS on top of K8S and OpenStack.--------------- ✌️Connect With Us ✌️ -------------Join our Slack community: https://go.mlops.community/slackFollow us on Twitter: @mlopscommunitySign up for the next meetup: https://go.mlops.community/registerConnect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/Connect with Vishnu on LinkedIn: https://www.linkedin.com/in/vrachakonda/Connect with Alex on LinkedIn: https://linkedin.com/in/alex-chung-gsdConnect with Sri on LinkedIn: https://www.linkedin.com/in/srivathsancanchi/Timestamps:[00:00] Introduction to Alex Chung and Srivathsan Canchi[01:36] Alex's background in tech[03:07] Srivathsan's background in tech[04:36] What is SGT?[05:53] 3 Categories of SGT           1. Education           2. Standardization           3. Orchestration  [07:00] Standardization is desirable[13:03] Perspective from both sides  [13:39] Profile breakdown of Standardization[17:20] Importance of Standardization in an enterprise[21:02] Tooling sprawl[24:04] Standardizing the different interfaces between MLOps tools[31:54] mlctl[33:35] mlctl's future[38:38] How MLCTL helps the workflow of Intuit[41:00] CIGS evolves the different spaces

Coffee Sessions #49 with Stefan Krawczyk, Aggressively Helpful Platform Teams.Join the Community: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTJoinIn⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Get the newsletter: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTNewsletter⁠⁠⁠⁠⁠⁠// AbstractAt Stitch Fix, there are 130+ “Full Stack Data Scientists” who, in addition to doing data science work, are also expected to engineer and own data pipelines for their production models. One data science team, the Forecasting, Estimation, and Demand team, was in a bind. Their data generation process was causing them iteration & operational frustrations in delivering time-series forecasts for the business. The solution? Hamilton, a novel Python micro-framework, solved their pain points by changing their working paradigm.Some of the main workers on Hamilton are the dedicated engineering team called the Data Platform. Data Platform builds services, tools, and abstractions to enable DS to operate in a full-stack manner, avoiding hand-off. In the beginning, this meant DS built the web apps to serve model predictions. Now, as the layers of abstractions have been built over time, they still dictate what is deployed, but write much less code.// BioStefan loves the stimulus of working at the intersection of design, engineering, and data. He grew up in New Zealand, speaks Polish, and spent formative years at Stanford, LinkedIn, Nextdoor & Idibon. Outside of work in pre-COVID times, Stefan liked to 🏊, 🌮, 🍺, and ✈.// Other Linkshttps://www.youtube.com/watch?v=B5Zp_30Knoohttps://www.slideshare.net/StefanKrawczyk/hamilton-a-micro-framework-for-creating-dataframes https://www.slideshare.net/StefanKrawczyk/deployment-for-free-removing-the-need-to-write-model-deployment-code-at-stitch-fix--------------- ✌️Connect With Us ✌️ -------------Join our Slack community: https://go.mlops.community/slackFollow us on Twitter: @mlopscommunitySign up for the next meetup: https://go.mlops.community/registerConnect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/Connect with Vishnu on LinkedIn: https://www.linkedin.com/in/vrachakonda/Connect with Stefan on LinkedIn: https://linkedin.com/in/skrawczykTimestamps: [00:00] Introduction to Stefan Krawczyk [00:37] Why Hamilton? [01:50] Stefan's background in tech [04:15] Model Life Cycle Team [06:48] Managing outcomes generated by data scientists [09:04] Teams are doing the same thing [12:41] Vision of getting code down to zero [18:40] Freedom and autonomy went wrong [21:17] Sub teams  [24:00] Create and deploy models easily [24:28] Interesting challenge to define [25:15] Stitch Fix Model productionization to be proud of [26:23] Hamilton to open-source [28:45] Model Envelope [31:45] Deployment for free [34:53] Use of Model Envelope in Model Artifact [37:16] Extending the API definition in a model envelope for the model [39:00] Dependencies [40:08] Monitoring at scale [43:43] Advice in terms of neat abstraction [46:19] Envelope vs Container [47:33] Time frame of Hamilton's development and its benefits

Coffee Sessions #48 with Julien Chaumond, Tour of Upcoming Features on the Hugging Face Model Hub.Join the Community: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTJoinIn⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Get the newsletter: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTNewsletter⁠⁠⁠⁠⁠//AbstractJulien Chaumond’s Tour of Upcoming Features on the Hugging Face Model Hub. Our MLOps community guest in this episode is Julien Chaumond, the CTO of Hugging Face - every data scientist’s favorite NLP Swiss army knife.Julien, David, and Demetrios spoke about many topics, including:Infra for hosting models/model hubsInference widgets for companies with CPUs & GPUs (for companies)Auto NLP, which trains models“Infrastructure as a service”// BioJulien Chaumond is Chief Technical Officer at Hugging Face, a Brooklyn and Paris-based startup working on Machine learning and Natural Language Processing, and is passionate about democratizing state-of-the-art AI/ML for everyone.--------------- ✌️Connect With Us ✌️ -------------Join our Slack community: https://go.mlops.community/slackFollow us on Twitter: @mlopscommunitySign up for the next meetup: https://go.mlops.community/registerConnect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/Connect with David on LinkedIn: https://www.linkedin.com/in/aponteanalytics/Connect with Julien on LinkedIn: https://www.linkedin.com/in/julienchaumond/Timestamps: [00:00] Introduction to Julien Chaumond [01:57] Julien's background in tech [04:35] "I have this vision of building a community where the greatest people in AI can come together and basically invent the future of Machine Learning together." [04:55] What is Hugging Face? [06:45] Start of open-source in Hugging Face [07:50] Chatbox experiment (reference resolution system) - linking pronouns to the subjects of sentences [10:20] From a project to a company [11:57] Importance of platform [14:25] "Transfer learning is an efficient way of Machine Learning. Providing your platform  around change that people want to start from a pre-trained model and fine-tune it into the specific use case is something that can be big, so we built some stuff to help people do that." [15:35] Narrowing down the scope of service to provide [16:27] "We have some vision of what we want to build, but a lot of it is the small incremental improvements that we bring to the platform. I think it's the natural way of building stuff nowadays because Machine Learning is moving so fast." [20:00] Model Hubs [22:37] "We're guaranteeing that we don't build anything that introduces any lag to Hugging Face because we're using GitHub. You'll have that peace of mind." [26:31] Storing model artifacts [27:00] AWS - cache - stored to an edge location all around the globe [28:39] Inference widgets powering [27:17] "For each model on the model hub, we try to ensure that we have the metadata about the model to be able to actually run it." [32:11] Deploying infra function [32:38] "Depending on the model and library, we optimize the custom containers to make sure that they run as fast as possible on the target hardware that we have."   [34:59] "Machine Learning is still pretty much hardware dependent." [36:11] Hardware usage [39:04] "CPU is super cheap. If you are able to run Berks served with a 1-millisecond on CPU because you have powerful optimizations, you don't really need GPUs anymore. It's cost-efficient and energy-efficient."   [41:10] "It may sound like a super cliche, but the team that you assembled is everything." [43:22] War stories in Hugging Face [44:12] "Our goal is more forward-looking to be helpful as much as we can to the community." [48:25] Hugging Face accessibility

Coffee Sessions #47 with Jeremy Howard, fast.ai, AutoML, Software Engineering for ML.Join the Community: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTJoinIn⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Get the newsletter: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTNewsletter⁠⁠⁠⁠// AbstractAdvancement in ML Workflows: You've been around the ML world for long enough to have seen how much workflows, tooling, frameworks, etc., have matured and allowed for greater scale and access. We'd love to reflect on your personal journey in this regard and hear about your early experiences putting models into production, as well as how you appreciate/might improve the process now.Data Professional Diversity and MLOps: Your work at fast.ai, Kaggle, and now with NBDEV has played a huge part in supercharging a diverse ecosystem of professionals that contribute to ML-like ML/data scientists, researchers, and ML engineers. As the attention turns to putting models into production, how do you think this range of professionals will evolve and work together? How will things around building models change as we build more?Turning Research into Practice: You've consistently been a leader in applying cutting-edge ideas from academia into practical code that others can use. It's one of the things I appreciate most about the fast.ai course and package. // BioJeremy Howard is a data scientist, researcher, developer, educator, and entrepreneur. Jeremy is a founding researcher at fast.ai, a research institute dedicated to making deep learning more accessible. He is also a Distinguished Research Scientist at the University of San Francisco, the chair of WAMRI, and is Chief Scientist at platform.ai.Previously, Jeremy was the founding CEO of Enlitic, which was the first company to apply deep learning to medicine, and was selected as one of the world’s top 50 smartest companies by MIT Tech Review two years running. He was the President and Chief Scientist of the data science platform Kaggle, where he was the top-ranked participant in international machine learning competitions for 2 years running. He was the founding CEO of two successful Australian startups (FastMail and Optimal Decisions Group–purchased by Lexis-Nexis). Before that, he spent 8 years in management consulting, at McKinsey & Co., and at AT Kearney. Jeremy has invested in, mentored, and advised many startups and contributed to many open-source projects.He has many media appearances, including writing for the Guardian, USA Today, and The Washington Post, appearing on ABC (Good Morning America), MSNBC (Joy Reid), CNN, Fox News, BBC, and being a regular guest on Australia’s highest-rated breakfast news program. His talk on TED.com, “The wonderful and terrifying implications of computers that can learn”, has over 2.5 million views. He is a co-founder of the global Masks4All movement.// Related Links:jhoward.fastmail.fmenlitic.comjphoward.wordpress.com/--------------- ✌️Connect With Us ✌️ -------------Join our Slack community: https://go.mlops.community/slackFollow us on Twitter: @mlopscommunitySign up for the next meetup: https://go.mlops.community/registerConnect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/Connect with Vishnu on LinkedIn: https://www.linkedin.com/in/vrachakonda/Connect with Jeremy on LinkedIn: https://www.linkedin.com/in/howardjeremy/Timestamps: [00:00] Introduction [02:11] Jeremy's background [03:10] Workflow [12:59] Platform development [19:53] Balancing API [22:57] Moment of inefficiency [27:42] Helpful tactics [29:05] University of tools evolving  [41:10] Resources to solve problems [43:30] Jupiter notebooks [47:20] Jupiter notebooks into production [48:42] MBDev [51:20] Wrap up

Coffee Sessions #46 with Pablo Estevez, What We Learned from 150 Successful ML-enabled Products at Booking.com.Join the Community: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTJoinIn⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Get the newsletter: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTNewsletter⁠⁠⁠// AbstractWhile most of the Machine Learning literature focuses on the algorithmic or mathematical aspects of the field, not much has been published about how Machine Learning can deliver meaningful impact in an industrial environment where commercial gains are paramount. We conducted an analysis on about 150 successful customer-facing applications of Machine Learning, developed by dozens of teams in Booking.com, exposed to hundreds of millions of users worldwide, and validated through rigorous Randomized Controlled Trials. Our main conclusion is that an iterative, hypothesis-driven process, integrated with other disciplines, was fundamental to building 150 successful products enabled by Machine Learning.// BioPablo Estevez is the Principal Data Scientist at Booking.com. He has worked on recommendations, personalization, and experimentation across the Booking.com website, as well as being a manager on several machine learning, data science, and product development teams.// Other LinksTalk on the topic: https://www.youtube.com/watch?v=ljhtfrtuNqw&t=4h24m30sThe paper: https://blog.kevinhu.me/2021/04/25/25-Paper-Reading-Booking.com-Experiences/bernardi2019.pdf--------------- ✌️Connect With Us ✌️ -------------Join our Slack community: https://go.mlops.community/slackFollow us on Twitter: @mlopscommunitySign up for the next meetup: https://go.mlops.community/registerConnect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/Connect with Vishnu on LinkedIn: https://www.linkedin.com/in/vrachakonda/Connect with Pablo on LinkedIn: https://www.linkedin.com/in/estevezpablo/[00:00] Introduction to Pablo Estevez[02:02] Pablo’s Background in Tech[03:43] Machine Learning at Booking.com[08:09] 150 Models: Six Key Lessons[10:20] Reflecting on Past ML Work[10:38] Pablo’s Role in Team[12:49] Broader Applications, Bigger Impact[12:55] Driving Through Business Impact[14:40] Beyond Precision: Focus on Goals[16:24] Diversity Enables Better Exploration[17:43] Three-Step Problem-Solving Framework[18:42] Framework of Problem Design[19:12] Focus on Experimentation Culture[20:46] Scaling Tooling for Experimentation[22:58] Cheap Experiments, Better Insights[28:39] Real-World Interactions and Analysis[30:15] Connecting Hypotheses to Business Value[31:04] Defining Experiments as Code[31:37] Airbnb’s Workflow Example[34:53] Decision-Making Through Experimentation Results[35:48] Building an Experimentation Platform[36:39] Investing in Better Infrastructure[36:50] Experimentation Justifies Infrastructure Investment[38:45] Monitoring Metrics for Business Value[39:40] Connecting Models to Business Value[41:35] Deployment at Booking.com[45:13] Supporting More Use Cases[46:10] Latency Challenges Business Performance[48:43] Open-Sourcing at Booking.com[49:30] Responsible Open-Source Maintenance Standards[49:45] ML Open-Source Standards[52:00] Lessons Learned Since Publication[53:30] Structuring the Exploration Phase[54:02] Maintainability Within Diversity

MLOps community meetup #70! Last Wednesday, we talked to Monika Venckauskaite, Senior Machine Learning Engineer at Vinted.Join the Community: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTJoinIn⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Get the newsletter: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTNewsletter⁠⁠// AbstractOne of the areas that has the most transformed by ML in these years is cybersecurity. Traditionally, SIEM (Security Intelligence and Event Management) is performed by human analysts. However, as the cyber powers and tools of the world are growing, we need more and more of these specialists. The entire area of cybersecurity is experiencing a shortage of talent. This is where the ML is coming in to help us. Cybersecurity ML systems require a lot of expertise from specialists as well as unique ways of handling user-sensitive data. This imposes various architectural solutions. In this talk, Monika introduces us to the ways of using ML in cybersecurity and the unique challenges we face.// BioMonika is a keen and curious ML engineer, loving to build systems. She started in machine learning as a master's student, looking for the Higgs Boson and Dark matter within the CERN data. Later on, Monika moved to the IT industry and worked on various machine learning projects, including Open Source Intelligence Tools and a distributed system for ML cybersecurity analytics.Currently, Monika works as an MLOps engineer, improving the MLOps platform that is used in production to ship models to a 45 million-user platform. Monika also works in a start-up that is innovating satellite communication. In her free time, she loves books, traveling, and playing music.// TakeawaysCyber threats are all around us. ML as technology is both a savior and a threat.GDPR and sensitive user data bring in extra challenges for cybersecurity intelligence systems, leading to more complex architectural decisions.ML helps to fight the talent shortage.Cybersecurity requires real-time ML systems and reacting ASAP.----------- Connect With Us ✌️-------------   Join our Slack community: https://go.mlops.community/slackFollow us on Twitter: @mlopscommunitySign up for the next meetup: https://go.mlops.community/registerConnect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/Connect with Monika on LinkedIn: https://www.linkedin.com/in/monika-in-space/Timestamps: [00:00] Introduction to Monika Venčkauskaitė [05:50] Monika's background in tech [08:50] Machine Learning in Cyber Security [09:37] Content [10:19] Our world is run by machines    [11:16] Cybersecurity Threats [12:44] Cybersecurity Incident Response                         Cycle:                         1. Identify           2. Protect           3. Detect           4. Respond           5. Recover[25:05] The Iceberg                          Surface Web - 4% Indexed and easily searchable            Deep Web - 90% Not Indexed, tougher to find            Dark Web - 6% Obscured, difficult to discover[47:45] Recommendation: AI Superpowers: China, Silicon Valley, And The New World Order by Kai-Fu Lee (https://www.amazon.com/AI-Superpowers-China-Silicon-Valley/dp/132854639X)[50:54] "I think we are going in the same direction, but our implementations are different."