Memorization concern and Leaky

Michael Bommarito ran GPT through the bar exam over Christmas 2022 to prove it wouldn’t pass.

He’d been working with language models for years and had never had a moment where he thought they were really useful. Dan Katz kept asking. A little eggnog was involved. So Mike did it to shut him up. By the time they published the second paper a few months later, it had passed.

That story kicked off one of the most cited moments in legal AI history. But it's just one chapter in a career that keeps moving.

In this conversation, we trace the arc from that Christmas break experiment through to OpenClaw and agentic AI, the future of the Cravath pyramid, what we should be teaching our kids, and the trillion-dollar data centre buildout that's reshaping rural communities.

Mike sees the same story playing out everywhere: a growing identity crisis at every level, from lawyers, to rural communities, to humanity itself.

Keeping up with Mike

Mike is a modern polymath and that makes him hard to keep up with. My head was spinning just trying to prepare for the interview.

He started as a classics major studying Latin and Greek at Michigan before pivoting to maths and financial engineering. A PhD in political science followed, which is where he met Dan Katz in the Center for the Study of Complex Systems. He left academia for a hedge fund, then landed in legal tech in 2013. LexPredict, the company he co-founded with Katz, was doing predictive analytics and NLP for litigation years before the rest of the industry caught up.

Today he splits his time across 273 Ventures and Kelvin on the commercial side, and the ALEA Institute, a nonprofit where he funds research, builds models and datasets, and runs projects like Leaky, an open source tool for detecting whether text was in a model’s training data.

The real story behind “GPT Takes the Bar Exam”

The first paper, GPT Takes the Bar Exam (fondly remembered as GPT Fails the Bar Exam"), showed GPT-3.5 passing some sections or coming close. But the real drama came during preparation for the second paper with OpenAI. Mike and the team discovered that the bar exam they’d tested on was in the training data. The “oh sh*t” moment wasn’t that it had passed. It was that the results might not be scientific, that they couldn’t separate memorisation from actual ability.

They had to find a new exam, transform it into a format the model could process, and run it again. Only after multiple reads with clean data did they have confidence the results held. Pablo Arredondo from CaseText was involved. The second paper, GPT Passes the Bar Exam, made it onto The Late Show, the New York Times, and into conversations around the world.

Agents aren’t new

Mike’s latest book, Agentic AI in Law and Finance, makes the case that the word “agent” didn’t appear out of nowhere. Agent-based modelling goes back to the 1970s across economics, political science, and cognitive science. Schelling’s segregation models, Monte Carlo simulations, basic behavioural heuristics programmed into interacting routines. Mike and Dan grew up intellectually in that world during their PhDs. Then the foundation model companies picked up the term and most of that history got forgotten overnight.

The book grounds the current hype in 50 years of research and asks what it means for governance in highly regulated industries. Mike’s short definition of an agent: a doer with a to-do. Beyond that, he says, it’s still a mess. And governance hasn’t kept pace. He pointed to Dario Amodei’s candid admission that nobody appointed the foundation model companies as leaders of this. The lack of governance runs all the way from the top of the AI industry down to individual firms making buying decisions, about half of which, Mike argues, are driven by FOMO rather than strategy.

The pyramid is changing

On law firm business models, Mike is direct: status quo is certainly not the right answer. He and Katz are writing a new book on transforming legal and financial organisations, and the working cover image is the Cravath pyramid being reshaped, its base becoming mechanical or cybernetic. He’s hearing anecdotally about firms slowing junior hiring and seeing large back-office reductions at large global firms.

Mike feels the question firms need to ask themselves is what their clients are actually buying. Most buyers can’t answer that consistently, in his view. Some buy big law for insurance, some for relationships, some because they believe they get the best results. Until that’s clear, nobody can say what the new model looks like. But the old one isn’t surviving this.

What should we teach our kids?

Going a bit deeper than your usual legal tech podcast, we got into the question of what we should teach our kids, in a world where AI can out-perform humans in a growing number of tasks.

Mike and his wife homeschool all three of his kids and says he doesn’t know whether they’ll go to college. He’s built custom AI tools for their education, and the gap between what you can deliver at home with today’s technology and what even the best schools offer is, in his words, huge.

The problem runs deeper than curriculum. In the US, the cost of legal education isn’t commensurate with the expected value of the degree at most institutions. Faculty are naturally resistant to redesigning programmes in ways that might not include them. And the mismatch between what law schools will try to continue doing and what firms and clients will need is only going to widen.

Mike sees some hope in states like Texas and Florida, where regulatory innovation untethered from ABA standards might allow for more practical, technical training.

But the question extends well beyond law. If productised AI tools can deliver a better education than a classroom, what does that mean for public institutions where education is one of the primary services? What happens when you don’t need a teacher for every 15 to 30 students? These aren’t hypothetical questions for Mike. He’s living the answer with his own family every day.

The real risk isn’t AGI

The thread that ran through everything was bigger than legal. Mike’s deepest concern isn’t the terminator scenario. It’s that the global middle class expanded on the back of knowledge work that can be done over the internet by someone with basic language skills and a computer. A trillion and a half dollars is now racing to convert that exact work into “pure electricity”. If the expanding middle class is what kept the world relatively peaceful, what happens when that contraction starts?

That question led us to his other new book, This is Server Country, about the physical infrastructure buildout reshaping rural communities. As we spoke, Mike was about to join a court hearing over the Oracle/OpenAI Stargate data centre project in Saline Township, Michigan, a small town of a couple thousand people where, as he put it, everything about the identity and experience they’ve had is being destroyed.

Mike doesn’t hold back at this point: “We’re replacing people’s interactions with each other with tokens and audio that’s not real. We’re replacing physical landscape with something that’s not natural. We’re replacing labour in the economy with something that’s not actually labour”.

Whether it’s a lawyer drawing a line around their profession, a community drawing a line around their town, or humans drawing a line between themselves and machines, Mike sees the same thing everywhere: an identity crisis, at every level, that will dominate politics for the foreseeable future.

Books and links

* Agentic AI in Law and Finance by Michael Bommarito and Daniel Martin Katz (2026)

* This is Server Country by Michael Bommarito (2026)

* Upcoming book on transforming legal and financial organizations (Bommarito and Katz, in progress)

* GPT Takes the Bar Exam (2022) and GPT Passes the Bar Exam (2023), research papers by Bommarito, Katz, Shang Gao and Pablo Arredondo

* 273 Ventures / Kelvin (commercial)

* ALEA Institute (nonprofit research)

Connect with Mike

Connect with Mike on LinkedIn. He maintains inbox zero (mostly).

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.agents.law