Model Risk Evaluation and Scoring

In the AI Risk Reward podcast, our host, Alec Crawford (@alec06830), Founder and CEO of Artificial Intelligence Risk, Inc. aicrisk.com , interviews guests about balancing the risk and reward of Artificial Intelligence for you, your business, and society as a whole. Podcast production and sound engineering by Troutman Street Audio. You can find them on LinkedIn.

In this episode, Alec welcomes back Michael Hind, Distinguished Research Staff Member at IBM. This episode is a special deep dive focused exclusively on the evolving field of AI governance. Michael defines AI governance from both enterprise and societal perspectives, highlighting the challenges of managing risk in rapidly evolving AI systems. He shares insights from his recent research, including the development of the AI Risk Atlas and model risk evaluation tools, and discusses the complexities of testing AI models and the importance of accurate benchmarking. The conversation covers the state of regulation, the intersection of insurance and AI risk, the role of transparency and explainability, and emerging technical solutions like entity tagging in LLMs. Alec and Michael conclude by emphasizing the need for industry-driven governance and enhanced transparency through tools such as Granite Guardian and Benchmark Cards.

Summary:

1. Defining AI Governance: Michael Hind explains the dual perspectives of AI governance—enterprise risk management and societal impact—and discusses the need for clear taxonomies.
2. Taxonomies and Risk Evaluation: IBM’s AI Risk Atlas and model risk evaluation tools help organizations identify, test, and monitor relevant AI risks for specific use cases.
3. Regulation and Industry Responsibility: With global regulation slowing, Michael argues for proactive enterprise governance, transparency, and industry benchmarks to fill the gap.
4. Testing, Explainability, and Transparency: The episode explores the limits of model evaluation, the challenge of explainability, and the need for public transparency, including the Stanford Transparency Index.
5. Insurance and Technical Advances: The dialogue addresses how insurance may eventually adapt to AI risk, and highlights new approaches like entity tagging and fault-tolerant generative computing.

Companies/Organizations:

1. IBM
2. Artificial Intelligence Risk, Inc.
3. NIST
4. MIT
5. Stanford University
6. Notre Dame
7. Drainpipe IO
8. OpenAI
9. Google

Copyright © 2025 by Artificial Intelligence Risk, Inc.