Unlimited submissions and hill-climbing problem

Bryan reflects on no-penalty scoring enabling brute-force submission strategies that undermined trust.

Play episode from 44:47

chevron_right

Transcript

chevron_right

Transcript

Episode notes

I often see what I would consider to be b******t evals, especially in data, like write this dumb SQL. Almost every one of these dumb SQL questions that I’ve seen for benchmarks are just so either obviously easy or overwhelmingly adversarial. They just, they don’t feel valuable as a data scientist, it’s something that you probably would never ask a real data scientist to do. So I went out my way to create real ones. Let me read one to you.

Bryan Bischof, Head of AI at Theory Ventures, joins Hugo to talk about what happened when 150 people spent six hours using AI agents to answer real data science questions across SQL tables, log files, and 750,000 PDFs.

They Discuss:

* Failure Funnels, pinpoint where agent reasoning breaks down using causal-chain binary evaluations instead of vague 1-5 scales;

* Median Score: 23 out of 65, what happened when world-class engineers turned agents loose on real data work, and why general-purpose coding agents with human prodding beat fancy frameworks;

* Zero-Cost Submissions Kill Trust, without a penalty for wrong answers, agents hill-climb to correct submissions through brute force instead of building confidence;

* Data Science is “Zooming”, moving beyond binary decisions to iterative problem framing, refining “does our inventory suck?” into a tractable hypothesis;

* MCP as Semantic Layer, model your organization’s proprietary knowledge once and distribute it to whatever LLM interface your team prefers;

* The Subagent vs. Tool Debate, a distinction that adds cognitive load without hiding complexity;

* Self-Orchestration Gap, agents don’t yet realize they should trigger specialized extraction frameworks like DocETL instead of reading 750K PDFs one by one;

* The Future of Evals, from vibe checks to objective functions and continuous user feedback that lets systems converge on reliability.

You can also find the full episode on Spotify, Apple Podcasts, and YouTube.

You can also interact directly with the transcript here in NotebookLM: If you do so, let us know anything you find in the comments!

👉 Want to learn more about Building AI-Powered Software? Check out our Building AI Applications course. It’s a live cohort with hands on exercises and office hours. Our final cohort has started. Registration is still open. All sessions are recorded so don’t worry about having missed any. Here is a 25% discount code for readers. 👈

LINKS

* Bryan Bischof on Twitter/X

* Bryan Bischof on LinkedIn

* Theory Ventures

* The Hunt for a Trustworthy Data Agent (blog post)

* America’s Next Top Modeler GitHub repo

* Hamel’s evals FAQ: How do I evaluate agentic workflows?

* DocETL

* LLM Judges and AI Agents at Scale (Hugo’s podcast with Shreya Shankar)

* When Your Metrics Are Lying (Cimo Labs)

* Lessons from a Year of Building with LLMs (livestream on YouTube)

* Bryan Bischof: The Map is Not the Territory (YouTube)