Hanselminutes with Scott Hanselman The next supercomputer with NVIDIA's Wen-Mei Hwu
Sep 26, 2024
Wen-Mei Hwu, Senior Distinguished Research Scientist at NVIDIA and Professor Emeritus at the University of Illinois, shares his groundbreaking work on processor architectures. He dives into the evolution of Moore's Law and Dennard Scaling, shedding light on how these concepts have propelled advancements in computing. The conversation covers the rise of specialized processors and forecasts the future of AI and personal computing. Hwu emphasizes the need for innovative software solutions to keep pace with technology and discusses the importance of a developer-centric approach.
AI Snips
Chapters
Transcript
Episode notes
Why Long-Term Hardware Forecasting Matters
- Dennard scaling and Moore's Law enabled decades where code sped up without change, letting architects plan hardware leaps a decade ahead.
- Wen-Mei Hwu recounts Berkeley research in the 1980s that predicted exploiting parallelism would be feasible once transistor scaling reached Intel's P6 era.
Design Research For A Decade Ahead
- When doing long-term research, anticipate technology states 10 years out because hardware adoption lags by 5–10 years.
- Hwu credits Berkeley work in the 1980s that only became practical when Intel's P6 realized those capabilities a decade later.
APIs Cause Hidden Parallelism Bottlenecks
- High-level languages and APIs (e.g., Python/PyTorch) create separated GPU activities that introduce runtime inefficiencies and bottlenecks.
- Hwu predicts on-the-fly fusion and runtime optimizations will be deployed to remove isolated API-call overheads within the next decade.