Quality Research Shows the Real Impact of AI @ Work

Quote:

“If you know what you’re doing, AI makes you faster. If you don’t, it just makes you wrong faster.”

–Louis Hickman

In this episode I’m joined by esteemed Psych Tech @ Work, Alumnus and AI research machine, Louis Hickman. Our incredible conversation taps into Louis’ myriad research studies to unpack AI’s direct impact on work, domain expertise, and talent assessment.

And of course, this episode also marks the return of the now new and improved AI podcast co-host Mayda Tokens (2.0).

Besides telling dumb jokes- Mayda’s job is to remind us that AI isn’t just a tool — it’s becoming an active participant in how we think, question, and explore ideas.

In the course of our conversation Mayda and I coax some PROFOUND take aways from our friend Louis as he shares the practical outcomes of his research:

1. AI is not removing the need for expertise — it’s making it more visible.

Scaling intelligence is easy.Scaling judgment is not.

The organizations that succeed won’t be the ones that adopt AI the fastest.

They’ll be the ones that:

* Understand what they’re measuring

* Use AI to enhance — not replace — that understanding’

* Maintain control over how decisions are made

2. AI allows us to scale both good science and bad measurement

Louis pushes back on the idea that recent advances represent a fundamental shift in how we measure people. Instead, what we’re seeing is:

* Better models

* Faster processing

* More scalable systems

But none of that replaces the need for valid, reliable, and job-relevant measurement.

3. AI doesn’t level the playing field — it often rewards those who already understand the game.

One of the most interesting ideas in this episode is how AI interacts with individual differences in expertise.

At a high level:

* For simple tasks, AI helps novices perform closer to experts

* For complex tasks, AI actually widens the gap- allowing experts to perform better

Why?

Because experts know how to ask better questions, recognize when AI is wrong, and refine its outputs—while novices often lack the ability to judge quality, diagnose errors, or course-correct when things go off track.

4. Replicability in LLMs Is Possible — if you know how to set it up right

A major “wow” moment in Louis’ research:

By running the model locally on the same class of hardware, fixing the model and prompt, and turning off sampling/randomness in the settings, you can make the system produce the same output for the same input every time.

5. AI should be used to scale decisions, but those decisions still need to be grounded in clearly defined constructs

At this point, AI adoption isn’t optional—it’s expected. Organizations are being pushed to move faster and scale, while vendors are rapidly building and deploying solutions, often without deep validation.

The resolution isn’t to slow down adoption—it’s to ensure we add and maintaining rigor.

6. AI makes it easy to scale assessment, but if the underlying design is weak, we’re just scaling bad measurement faster.

The resolution is to ensure what gets scaled is built on clear constructs, strong design, and validated measurement, so speed amplifies quality—not noise.

7. Working with AI is no longer just about what you can do—it’s about how effectively you can partner to make what you do better!

The tension is clear: AI can accelerate work, but over-reliance without critical evaluation leads to lower quality, missed errors, and reduced trust.

This shows up in real ways—unchecked outputs, declining attention to detail, and growing skepticism in collaborative work.

The resolution is that AI doesn’t replace accountability—users still need to apply judgment, review outputs, and take ownership of the final result.

Tune in to get the full story on these profound revelations and hear Mayda’s stand up comedy routine.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit charleshandler.substack.com