Josh Wills has spent 25 years writing data pipelines, with a career spanning Cloudera, as Director of Data Engineering at Slack, on the dbt DuckDB adapter, and now training foundation models at Datology AI. He uses coding agents every day. And he keeps running into the same wall: the agents jump to conclusions, fix the wrong thing, and ship pipelines no one understands.In this conversation, we unpack why AI agents struggle with the messiest, highest-stakes parts of data work, and what it means for the engineers managing them.We get into:- Big Data is back- Why AI agents jump to conclusions on benchmarks and complex bottlenecks- The $200K vibe-coded pipeline problem nobody wants to talk about- Why there's no training data for the gnarly enterprise pipelines that actually power businesses- "We're all managers now" - managing unreliable agents like managing unreliable people- Wicked problems and the limits of intelligence- Why politics is the last human endeavor to fall to LLMs (the data is never written down)- Whether classical ML still has a place (yes)- What Josh would tell a new grad starting in data today
