Engineering boundaries to avoid personhood

Mustafa outlines design choices to prevent models from manipulating users or claiming personal states.

Play episode from 19:56

chevron_right

Transcript

chevron_right

Transcript

Episode notes

Welcome to Exponential View, the show where I explore how exponential technologies such as AI are reshaping our future. I've been studying AI and exponential technologies at the frontier for over ten years.

Each week, I share some of my analysis or speak with an expert guest to make light of a particular topic.

To keep up with the Exponential transition, subscribe to this channel or to my newsletter: https://www.exponentialview.co/

-----

A week before OpenClaw exploded, I recorded a prescient conversation with Mustafa Suleyman, CEO of Microsoft AI and co-founder of DeepMind. We talked about what happens when AI starts to seem conscious – even if it isn’t. Today, you get to hear our conversation.

Mustafa has been sounding the alarm about what he calls “seemingly conscious AI” and the risk of collective AI psychosis for a long time. We discussed this idea of the “fourth class of being” – neither human, tool, nor nature – that AI is becoming and all it brings with it.

Skip to the best bits:

(03:38) Why consciousness means the ability to suffer

(06:52) "Your empathy circuits are being hacked"

(07:23) Consciousness as the basis of rights

(10:47) A fourth class of being

(13:41) Why market forces push toward seemingly conscious AI

(20:56) What AI should never be allowed to say

(25:06) The proliferation problem with open-source chatbots

(29:09) Why we need well-paid civil servants

(30:17) Where should we draw the line with AI?

(37:48) The counterintuitive case for going faster

(42:00) The vibe coding dopamine hit