619: Tools for Deploying Data Models into Production
Super Data Science: ML & AI Podcast with Jon Krohn
00:00
Evolution of Tools for Managing Data Pipelines and Code Repositories
This chapter explores the journey from the creation of the open-source tool Luigi at Spotify for handling complex data pipelines, to the emergence and popularity of similar tools like Airflow, Prefect, and others. The conversation also dips into the concept of code evolution within repositories, drawing parallels to the Ship of Theseus thought experiment and the importance of innovation and identity as code ages. There is a reflection on the decline in popularity of Luigi compared to its successors, along with discussions on creating open source repositories for analyzing growth over time and the challenges of making reliable conclusions with sparse data in probabilistic programming and quantifying uncertainty.
Play episode from 49:50
Transcript
