d-Matrix - Ultra-low Latency Batched Inference for Gen AI

5 snips

Mar 7, 2026

Satyam Srivastava, electrical engineer and d-Matrix co-founder who built energy-efficient ML systems after stints at NVIDIA and Intel. He discusses why inference—not training—is becoming the real scaling challenge. He explains memory, data-movement, and power bottlenecks, why general-purpose GPUs struggle, and how purpose-built, efficiency-first hardware and 3D memory designs can change data center economics.

Ask episode

AI Snips

Chapters

Transcript

Episode notes

INSIGHT

Inference Is The Real Productization Bottleneck

Inference, not training, will define AI's real-world success because models must serve millions of daily interactions.
Satyam explains that serving those interactions strains memory, energy, and data movement far more than model research did.

ADVICE

Plan Data Centers For Efficiency Not Raw Power

Plan capacity around efficiency instead of brute-force scaling to avoid impossible power and cooling demands.
Satyam urges custom designs that deliver the work of 10 GPUs within existing power and cooling limits.

INSIGHT

Low GPU Utilization Reveals Hidden Cost Problem

Low utilization of expensive general-purpose hardware signals a fundamental ROI problem for inference workloads.
Satyam notes top-end hardware often runs at low single-digit utilization, wasting capital and infrastructure.

Get the Snipd Podcast app to discover more snips from this episode

Get the app

What happens when the real bottleneck in artificial intelligence is no longer training models, but actually running them at scale?

In this episode of Tech Talks Daily, I sit down with Satyam Srivastava from d-Matrix to explore a shift that is quietly reshaping the entire AI infrastructure landscape. While much of the early AI race focused on training ever larger models, the next phase of AI adoption is increasingly defined by inference. That is the moment when trained models are deployed and used to generate real-world results millions of times a day.

Satyam brings a unique perspective shaped by years of experience in signal processing, machine learning, and hardware architecture, including time spent at NVIDIA and Intel working on graphics, media technologies, and AI systems. Now at d-Matrix, he is helping design next-generation computing architectures focused on one of the biggest challenges facing the AI industry today: efficiently running large language models without overwhelming data centers with unsustainable power and infrastructure demands.

During our conversation, we explored why the industry underestimated the infrastructure implications of inference at scale. While training large models grabs headlines, the real operational pressure often comes later when those models must serve millions of queries in real time. That shift places enormous strain on memory bandwidth, energy consumption, and data movement inside modern data centers.

Satyam explains how d-Matrix identified this challenge years before generative AI exploded into the mainstream. Instead of focusing on training hardware like many AI startups at the time, the company concentrated on inference efficiency. That decision is becoming increasingly relevant as organizations begin to realize that simply adding more GPUs to data centers is not a sustainable long-term strategy.

We also discuss the growing power constraints surrounding AI infrastructure, and why efficiency-driven design may be the only realistic path forward. With electricity supply, cooling capacity, and semiconductor availability all becoming limiting factors, the industry is being forced to rethink how AI systems are architected. Custom silicon, purpose-built accelerators, and heterogeneous computing environments are now emerging as key pieces of the puzzle.

The conversation also touches on the geopolitical and economic importance of AI semiconductor leadership, and why the relationship between frontier AI labs, infrastructure providers, and chip designers is becoming increasingly strategic. As governments and companies compete to maintain technological leadership, the question of who controls the hardware powering AI may prove just as important as the models themselves.

Looking ahead, Satyam shares his perspective on how the role of engineers will evolve as AI infrastructure becomes more specialized and energy-aware. Foundational engineering skills remain essential, but the next generation of engineers will also need to think in terms of entire systems, combining software, hardware, and AI tools to build more efficient computing environments.

As AI continues to move from research labs into everyday products and services, are organizations prepared for the infrastructure shift that comes with an inference-driven future? And could efficiency, rather than raw computing power, become the defining metric of the next phase of the AI race?