Ep193: The Conductor Behind Your Data Orchestra: Astronomer's Approach to AI Pipeline Management

Astronomer's Steven Hillion reveals how OpenAI, Anthropic, Uber, and Lyft use Apache Airflow to orchestrate AI and machine learning pipelines at scale on AWS.

Topics Include:

• Steven Hillion leads data and AI at Astronomer
• Apache Airflow surpassed Spark and Kafka in community metrics
• Astronomer coordinates data flow like conductor orchestrating instrumental platforms
• Organizations with data engineering teams use Airflow at scale
• Customers already used Airflow for ML before official promotion
• Uber and Lyft orchestrate pricing models using Airflow
• Astronomer runs on AWS with close integration partnerships
• OpenAI Anthropic and GitHub Copilot use Airflow for operations
• Internal data team uses Airflow creating feedback loops
• Evolved from constrained AI reports to agentic workflows
• Platform monitors generative AI output quality at user interactions
• Metadata and context increasingly critical for AI applications
• Learn more at Astronomer’s Data FlowCast podcast

Participants:

• Steven Hillion – SVP, Data and AI, Astronomer

See how Amazon Web Services gives you the freedom to migrate, innovate, and scale your software company at https://aws.amazon.com/isv/