The Truth About AI Based Talent Assessment

“The rules haven’t changed. The technology has — but the rules haven’t.” — Nathan Mondragon

Episode Overview

In this episode, I’m joined by my old friend (and now co-worker!) Nathan Mondragon, an IO psychologist and long-time leader in creating the future at the intersection of assessment science, hiring technology, and applied AI.

Nathan and I have lived through multiple waves of “this will change everything” technology — from early online testing to video interviewing, machine learning, and now generative AI. And the beat goes on!

Nathan and I have recently joined forces at ProboTalent where we are creating defensible AI based assessment tools.

We talk about where AI has genuinely moved the field forward, where it hasn’t, and why so many of the debates we’re having today are versions of conversations we’ve been having for decades. Along the way, we unpack Nathan’s paradigm busting work at HireVue’, and why the fundamentals of good measurement haven’t changed — even as the tools have.

Topics Discussed & Key Insights

1. The Rules of Good Assessment Haven’t Changed — We Just Keep Forgetting Them

Nathan makes a point that anchors the entire episode: while technology has advanced dramatically, the core rules of good assessment — validity, relevance, interpretability, and fairness — are exactly the same.

AI doesn’t get a pass on methodology. If anything, it raises the bar for rigor, because mistakes scale faster.

2. Early Hiring Tech Was Built to Solve Operational Problems, Not Measurement Problems

We talk about the early days of online hiring and assessment, where the primary goal was digitization, not insight. Systems were designed to move paper processes online, not to improve how well we understand people.

That legacy still shapes today’s platforms — and explains why so many tools feel efficient but shallow.

3. HireVue Was a Real Paradigm Shift — and It Required Scientific Courage

Nathan reflects on the early days of HireVue and why it was genuinely revolutionary at the time. The breakthrough wasn’t just video — it was the larger shift toward digitizing and scaling structured assessment experiences in a way the field hadn’t seen before.

What made this moment interesting from an IO psychology standpoint is that it required a different mindset as a scientist: being willing to engage with a new modality, even when the measurement implications weren’t fully understood yet. Innovation in assessment has always involved tension — between rigor and experimentation, between what’s proven and what’s possible.

Nathan shares what it was like to help lead through that transition, and why thoughtful scientists have to be able to sit with uncertainty long enough to shape new approaches responsibly, rather than rejecting them outright.

4. AI Didn’t Create Bad Measurement — It Made It Easier to Scale

A recurring theme: AI doesn’t magically improve weak constructs. If you feed it noisy proxies, you just get faster, more confident noise.

We discuss why generative AI and machine learning don’t eliminate the need for careful construct definition — and why “it correlates” is not the same thing as “it measures something useful.”

5. Interactivity Matters More Than Modality

One of the most important takeaways: the future of assessment isn’t about whether something is text, video, or simulation-based — it’s about how interactive and information-rich the experience is.

Nathan explains why dynamic interaction reveals far more about decision-making, reasoning, and capability than static prompts ever will.

6. Native AI vs. Embedded AI Is a False Debate

We unpack the difference between “AI-native” products and traditional tools with AI layered on top — and why this distinction often misses the point.

What matters isn’t where AI lives in the stack, but whether it’s being used to improve interpretation, not just automate scoring or classification.

7. Skills and Knowledge Are Still Hard to Measure — and AI Has to Be Used Carefully

We close by confronting a reality the market often underestimates: skills and knowledge testing have always been difficult to do well, and scaling them without losing rigor is even harder.

We connect this directly to the work we’re doing at Probo Talent, where the focus is on a more responsible alternative: using AI to scale the parts of assessment that have historically been hardest to scale, while staying within safe, established modalities and an explainable, scientifically grounded wrapper. The goal is not novelty for its own sake, but a practical example of how AI can be used carefully to solve long-standing problems in skills-based hiring without sacrificing defensibility or trust

Final Takeaway

AI changes how we can build hiring and assessment systems — but it doesn’t change what makes them good.

If we ignore decades of psychological science in favor of speed, novelty, or convenience, AI will simply help us make the same mistakes faster. But if we use it to deepen interaction, improve interpretation, and stay disciplined about what we measure, it has the potential to finally move the field forward in meaningful ways.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit charleshandler.substack.com