Contributor Data Processing Evolved: OpenLineage with Willy Lulciuc
13 snips
Nov 18, 2025 Willy Lulciuc, a data engineer and co-creator of OpenLineage, shares his journey in the field from his time at WeWork to founding Oleander, which focuses on AI-enabled data tooling. He discusses the importance of data lineage and the challenges of observability in data processing. Willy explains how OpenLineage emerged from previous projects, the synergy with OpenTelemetry, and how AI can enhance, but not replace, the role of data engineers. He also outlines the future of data platforms and the unique benefits of OpenLineage for engineering teams.
AI Snips
Chapters
Books
Transcript
Episode notes
Design Runtime Lineage Events
- Emit ordered runtime events (start, running, abort, complete) and include SQL and inputs/outputs.
- Build a consumer backend that handles ordering, scale, and occasional out-of-order events.
Acquisition And Astro Observe
- Datakin scaled to a small engineering team and was acquired by Astronomer within two years.
- The product became Astro Observe, integrating OpenLineage events into Airflow observability for customers.
Data Platform As A Graph
- Oleander views the data platform as a graph of nodes and edges where runs produce dataset versions.
- Combining that graph with LLM context enables automated root-cause analysis of pipeline failures and runtime drift.
