Running Through Walls Empowering Data Engineers
Feb 25, 2026
Pete DeJoy, CEO and co-founder of Astronomer, who helped commercialize Apache Airflow. He recounts a high-profile public crisis and the leadership lessons that followed. He traces Astronomer’s origins with Airflow and why a commercial platform was needed. He explores Airflow’s role in AI: RAG context, batch inference, and agent orchestration, and talks about tooling that empowers modern data engineers.
AI Snips
Chapters
Transcript
Episode notes
How Astronomer Survived a Public Crisis
- Astronomer faced massive public scrutiny after the Coldplay concert incident and treated it as a team-defining crisis.
- Pete DeJoy says the team rallied, they responded swiftly and transparently, and they did not lose a single customer during the ordeal.
Spinning Out Around Airflow From A Clickstream Product
- Astronomer spun out from a Clickstream company after discovering Apache Airflow powering their batch pipelines.
- Pete recounts becoming heavy Airflow users, meeting the creators, and pivoting in 2018 to build a company around Airflow.
Why Airflow Needed An Enterprise Platform
- Astronomer layered an enterprise platform on Airflow because customers needed Airflow without the infrastructure and higher reliability for operational use cases.
- Pete explains they rewrote parts of Airflow for cloud performance and built authoring, observability, and SRE capabilities for critical workloads.