MLOps Coffee Sessions #12: Journey of Flyte at Lyft and Through Open-source // Ketan Umare

Oct 10, 2020

Ketan Umare, a Senior Staff Software Engineer at Lyft, discusses his pivotal role in the development of Flyte, a pivotal open-source project for machine learning infrastructure. He explains why Flyte was created, highlighting its capacity to handle tens of thousands of workflows and millions of tasks. The conversation delves into the complexities of mapping technology and the algorithmic challenges in ride-sharing. Ketan also shares insights on open-source community engagement and the transition to using Go for backend development.

Ask episode

AI Snips

Chapters

Transcript

Episode notes

ADVICE

Microservices For Team Scalability

Use microservices to scale engineering teams by team ownership of services and clear interface contracts.
Invest in contract evolution practices to avoid breaking changes and make onboarding easier.

INSIGHT

Flyte Powers Massive Pipeline Scale

Flyte manages hundreds of thousands of workflow executions monthly, serving all pipelines in the company.
It powers diverse workloads like Spark jobs, ETLs, analytics, and model building at large scale.

ANECDOTE

From Airflow Hack to Flyte Platform

Flyte started as an Airflow-based hack for ETA modeling that was not scalable but proved useful.
Demand from various teams pushed them to build a more robust Go-based workflow system on AWS Step Functions, evolving into Flyte.

Get the Snipd Podcast app to discover more snips from this episode

Get the app

Join the Community: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTJoinIn⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠

Get the newsletter: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTNewsletter⁠⁠

Why was Flyte built at Lyft?

What sorts of requirements does an ML infrastructure team have at Lyft?

What problems does it solve/use cases?

Where does it fit in the ML and Data ecosystem?

What is the vision?

Who should consider using it?

Learnings as the engineering team tried to bootstrap an open-source community.

Ketan Umare is a senior staff software engineer at Lyft, responsible for the technical direction of the Machine Learning Platform, and is a founder of the Flyte project. Before Flyte, he worked on ETA, routing, and mapping infrastructure at Lyft. He is also the founder of Flink Kubernetes operator and a contributor to Spark on Kubernetes. Prior to Lyft, he was a founding member of Oracle Baremetal Cloud and led teams building Elastic Block Storage. Prior to that, he started and led multiple teams in Mapping and Transportation optimization infrastructure at Amazon. He received his Master's in Computer Science from Georgia Tech, specializing in High-performance computing, and his Bachelor's in Engineering in Computer Science from VJTI Mumbai.

Besides work, he enjoys spending time with his daughter and wife. He loves the Pacific Northwest outdoors and will try anything new.

Lyft

Pricing, Locations, Estimated Time of Arrivals (ETA), Mapping, Self-Driving (L5), etc.

What sort of scale, storage, and network bandwidth are we looking at?

Tens of thousands of workflows, hundreds of thousands of executions, millions of tasks, and tens of millions of containers!

Flyte: more than 900k workflows executed a month and more than 30+ million container executions per month

Typical flow of information?

What are the user stories you’re typically dealing with at Lyft?

How do you set it up?

On-prem, cloud, etc.

Helm installable?

Why Golang?

What problems does it solve?

Complex data dependencies? Why

Orchestrated compute on demand

Reuse and sharing

Key features

Multi-tenant, hosted, serverless

Parametrized, data lineage, and caching

Additionally, if the run invokes a task that has already been computed before, regardless of who executed it, Flyte will smartly use the cached output, saving you both time and money.

Versioning, sharing

Modular, loosely coupled

Seems like you guys recognize that the best task for the job might be hosted elsewhere, so it was important to integrate other solutions into Flyte.

Flyte extensions

Backend plugins - is it true you can create and manage k8s resources like CRDs for things like Spark, Sagemaker, BigQuery?

Drop a Star

https://flyte.org

Flyte community

----------- Connect With Us ✌️-------------

Join our Slack community: https://go.mlops.community/slack

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/

Connect with David on LinkedIn: ⁠https://www.linkedin.com/in/aponteanalytics/⁠Connect with Ketan on LinkedIn: https://www.linkedin.com/in/ketanumare/