Open-weight safety and containment limits

Episode 144

Happy New Year! This is one of my favorite episodes of the year — for the fourth time, Nathan Benaich and I did our yearly roundup of AI news and advancements, including selections from this year’s State of AI Report.

If you’ve stuck around and continue to listen, I’m really thankful you’re here. I love hearing from you.

You can find Nathan and Air Street Press here on Substack and on Twitter, LinkedIn, and his personal site. Check out his writing at press.airstreet.com.

Find me on Twitter (or LinkedIn if you want…) for updates on new episodes, and reach me at editor@thegradient.pub for feedback, ideas, guest suggestions.

Outline

* (00:00) Intro

* (00:44) Air Street Capital and Nathan world

* Nathan’s path from cancer research and bioinformatics to AI investing

* The “evergreen thesis” of AI from niche to ubiquitous

* Portfolio highlights: Eleven Labs, Synthesia, Crusoe

* (03:44) Geographic flexibility: Europe vs. the US

* Why SF isn’t always the best place for original decisions

* Industry diversity in New York vs. San Francisco

* The Munich Security Conference and Europe’s defense pivot

* Playing macro games from a European vantage point

* (07:55) VC investment styles and the “solo GP” approach

* Taste as the determinant of investments

* SF as a momentum game with small information asymmetry

* Portfolio diversity: defense (Delian), embodied AI (Syriact), protein engineering

* Finding entrepreneurs who “can’t do anything else”

* (10:44) State of AI progress in 2025

* Momentous progress in writing, research, computer use, image, and video

* We’re in the “instruction manual” phase

* The scale of investment: private markets, public markets, and nation states

* (13:21) Range of outcomes and what “going bad” looks like

* Today’s systems are genuinely useful—worst case is a valuation problem

* Financialization of AI buildouts and GPUs

* (14:55) DeepSeek and China closing the capability gap

* Seven-month lag analysis (Epoch AI)

* Benchmark skepticism and consumer preferences (”Coca-Cola vs. Pepsi”)

* Hedonic adaptation: humans reset expectations extremely quickly

* Bifurcation of model companies toward specific product bets

* (18:29) Export controls and the “evolutionary pressure” argument

* Selective pressure breeds innovation

* Chinese companies rushing to public markets (Minimax, ZAI)

* (21:30) Reasoning models and test-time compute

* Chain of thought faithfulness questions

* Monitorability tax: does observability reduce quality?

* User confusion about when models should “think”

* AI for science: literature agents, hypothesis generation

* (23:53) Chain of thought interpretability and safety

* Anthropomorphization concerns

* Alignment faking and self-preservation behaviors

* Cybersecurity as a bigger risk than existential risk

* Models as payloads injected into critical systems

* (27:26) Commercial traction and AI adoption data

* Ramp data: 44% of US businesses paying for AI (up from 5% in early 2023)

* Average contract values up to $530K from $39K

* State of AI survey: 92% report productivity gains

* The “slow takeoff” consensus and human inertia

* Use cases: meeting notes, content generation, brainstorming, coding, financial analysis

* (32:53) The industrial era of AI

* Stargate and XAI data centers

* Energy infrastructure: gas turbines and grid investment

* Labs need to own models, data, compute, and power

* Poolside’s approach to owning infrastructure

* (35:40) Venture capital in the age of massive GPU capex

* The GP lives in the present, the entrepreneur in the future, the LP in the past

* Generality vs. specialism narratives

* “Two or 20”: management fees vs. carried interest

* Scaling funds to match entrepreneur ambitions

* (40:10) NVIDIA challengers and returns analysis

* Chinese challengers: 6x return vs. 26x on NVIDIA

* US challengers: 2x return vs. 12x on NVIDIA

* Grok acquired for $20B; Samba Nova markdown to $1.6B

* “The tide is lifting all boats”—demand exceeds supply

* (44:06) The hardware lottery and architecture convergence

* Transformer dominance and custom ASICs making a comeback

* NVIDIA still 90–95% of published AI research

* (45:49) AI regulation: Trump agenda and the EU AI Act

* Domain-specific regulators vs. blanket AI policy

* State-level experimentation creates stochasticity

* EU AI Act: “born before GPT-4, takes effect in a world shaped by GPT-7”

* Only three EU member states compliant by late 2025

* (50:14) Sovereign AI: what it really means

* True sovereignty requires energy, compute, data, talent, chip design, and manufacturing

* The US is sovereign; the UK by itself is not

* Form alliances or become world-class at one level of the stack

* ASML and the Netherlands as an example

* (52:33) Open weight safety and containment

* Three paths: model-based safeguards, scaffolding/ecosystem, procedural/governance

* “Pandora’s box is open”—containment on distribution, not weights

* Leak risk: the most vulnerable link is often human

* Developer–policymaker communication and regulator upskilling

* (55:43) China’s AI safety approach

* Matt Sheehan’s work on Chinese AI regulation

* Safety summits and China’s participation

* New Chinese policies: minor modes, mental health intervention, data governance

* UK’s rebrand from “safety” to “security” institutes

* (58:34) Prior predictions and patterns

* Hits on regulatory/political areas; misses on semiconductor consolidation, AI video games

* (59:43) 2026 Predictions

* A Chinese lab overtaking US on frontier (likely ZAI or DeepSeek, on scientific reasoning)

* Data center NIMBYism influencing midterm politics

* (01:01:01) Closing

Links and Resources

Nathan / Air Street Capital

* Air Street Capital

* State of AI Report 2025

* Air Street Press — essays, analysis, and the Guide to AI newsletter

* Nathan on Substack

* Nathan on Twitter/X

* Nathan on LinkedIn

From Air Street Press (mentioned in episode)

* Is the EU AI Act Actually Useful? — by Max Cutler and Nathan Benaich

* China Has No Place at the UK AI Safety Summit (2023) — by Alex Chalmers and Nathan Benaich

Research & Analysis

* Epoch AI: Chinese AI Models Lag US by 7 Months — the analysis referenced on the US-China capability gap

* Sara Hooker: The Hardware Lottery — the essay on how hardware determines which research ideas succeed

* Matt Sheehan: China’s AI Regulations and How They Get Made — Carnegie Endowment

Companies Mentioned

* Eleven Labs — AI voice synthesis (Air Street portfolio)

* Synthesia — AI video generation (Air Street portfolio)

* Crusoe — clean compute infrastructure (Air Street portfolio)

* Poolside — AI for code (Air Street portfolio)

* DeepSeek — Chinese AI lab

* Minimax — Chinese AI company

* ASML — semiconductor equipment

Other Resources

* Search Engine Podcast: Data Centers (Part 1 & 2) — PJ Vogt’s two-part series on XAI data centers and the AI financing boom

* RAAIS Foundation — Nathan’s AI research and education charity