MLOps.community Evolving Workflow Orchestration // Alex Milowski // #291
30 snips
Feb 14, 2025 Alex Milowski, a serial entrepreneur and computer scientist with a PhD in Informatics, dives into the evolving landscape of workflow orchestration. He discusses the shift from traditional workflow languages to code, emphasizing its implications for complexity in orchestration. Milowski also highlights the importance of visual tools for process modeling and the unique challenges that regulated industries face in machine learning. He contrasts modern technologies with traditional automation, stressing human oversight in AI workflows and the delicate balance of documentation in agile systems.
AI Snips
Chapters
Transcript
Episode notes
Workflow Layering
- Don't think of workflows as a single entity, but rather as layered processes.
- Consider technical and business aspects, optimizing for the larger organizational context.
Stitch Fix's DSL
- Stitch Fix's 140 data scientists used a DSL that compiled into Airflow DAGs.
- This abstraction simplified workflow creation and allowed for automated task optimization.
Workflow System Maturity
- Workflow systems have matured significantly in the last decade, becoming standard practice.
- Numerous tools exist, spanning open-source projects and SaaS offerings, catering to diverse needs.