What Comes After AI Transformers? (Ep. 531)

The discussion sets the stage for exploring what comes after transformers.

Key Points Discussed

Transformers show limits in reasoning, instruction following, and real-world grounding.

The AI field is moving from scaling to exploring new architectures.

Smarter transformers can be enhanced with test-time compute, neurosymbolic logic, and mixture-of-experts.

Revolutionary alternatives like Mamba, Retinette, and world models introduce different approaches.

Emerging ideas such as spiking neural networks, Kolmogorov Arnold networks, and temporal graph networks may reduce energy costs and improve reasoning.

Neurosymbolic hybrids are highlighted as a promising path for logical reasoning.

The challenge of commercializing research and balancing innovation with environmental costs.

Hybrid futures likely combine multiple architectures into a layered system for AGI.

The concept of swarm intelligence and agent collaboration as another route toward advanced AI.

Timestamps & Topics

00:00:00 💡 Introduction and GPT 5 disappointment

00:02:00 🔍 The shift from scaling to new paradigms

00:04:00 ⚙️ Smarter transformers and test-time compute

00:05:20 🚀 Revolutionary alternatives including Mamba and Retinette

00:06:20 🌍 World models and embodied AI

00:06:58 🧠 Spiking neural networks and novel approaches

00:11:00 ⛵ Exploration analogies and transformer context challenges

00:12:20 🎮 Applications of world models in 3D spaces and XR

00:16:45 🔗 Neurosymbolic hybrids for reasoning

00:19:00 ⚡ Energy efficiency and productization challenges

00:24:00 🌱 Balancing research speed with environmental costs

00:31:00 📉 Four structural limits of transformers

00:35:00 📚 RKV and new memory-efficient mechanisms

00:37:00 📝 Analogies for architectures: note taker, stenographer, librarian, consultant

00:41:00 🕵️ Transformer reasoning illusions and dangers

00:44:00 🔬 Outlier experiments: physical neural nets, temporal graph networks, recurrent GANs

00:49:00 🧩 Hybrid architecture visions for AGI

00:53:30 🐝 Swarm agents and collaborative intelligence

00:55:00 📢 Closing announcements and upcoming shows

The Daily AI Show Co-Hosts: Andy Halliday, Beth Lyons, Brian Maucere, Eran Malloch, Jyunmi Hatcher, and Karl Yeh