We Cut LLM Latency by 70% in Production

26 snips

Apr 10, 2026

Maher Hanafi, SVP of Engineering who led self-hosting LLMs at enterprise scale and optimized GPU inference, shares practical production stories. He describes cutting latency 50–70% with TensorRT LLM. He explains cold-start fixes, KV-cache and in-flight batching, scaling strategies that lower GPU spend, and how vertical features evolve into a reusable AI platform.

Ask episode

AI Snips

Chapters

Transcript

Episode notes

INSIGHT

TensorRT LLM Cut Latency And Changed GPU Tradeoffs

TensorRT LLM rewires models to GPU architecture and delivered 50–70% latency reductions for Maher’s team.
Combined with in-flight batching and larger KV cache, one model per GPU boosted throughput more than packing multiple models did.

INSIGHT

Leave Memory For KV Cache To Maximize Throughput

Counterintuitively, leaving VRAM for KV cache and running one model per GPU yields higher throughput than packing multiple models.
Maher used KV-heavy deployments plus TensorRT in-flight batching to keep batches filling and decoding continuously.

ADVICE

Buy Bigger GPUs If They Cut Total Run Hours

Try upgrading to larger, costlier GPUs when higher throughput lets you run them far fewer hours — total cost can drop.
Maher calculated a 30% pricier GPU used 50% less time, yielding overall savings while improving latency.

Get the Snipd Podcast app to discover more snips from this episode

Get the app

Maher Hanafi is an engineering leader who went from zero AI experience to self-hosting LLMs at enterprise scale — managing GPU costs, optimizing inference with TensorRT LLM, and building an AI platform for HR tech. In this conversation, he breaks down exactly how his team cut latency by 70%, reduced GPU spend through counterintuitive scaling strategies, and navigated the messy reality of taking AI from proof-of-concept to production.

How We Cut LLM Latency 70% With TensorRT in Production // MLOps Podcast #369 with Maher Hanafi, SVP of Engineering at Betterworks

Key topics covered:

The AI Iceberg — Why the invisible work behind AI (performance, latency, throughput, cost, accuracy) is harder than building the features themselves

GPU Cost Optimization — How upgrading to more expensive GPUs actually saved money by reducing total runtime hours

TensorRT LLM Deep Dive — Rewiring neural networks to match GPU architecture for 50-70% latency reduction

Cold Start Solutions — Using AWS FSx, baking models into container images, and cutting minutes off spin-up times

KV Cache & In-Flight Batching — Why using one model per GPU with maximum KV cache beats cramming multiple models together

Scheduled & Dynamic Scaling — Pattern-based scaling for HR tech workloads (nights, weekends, end-of-quarter spikes)

Verticalized AI Platform — Building horizontal AI infrastructure that serves multiple HR product verticals

AI Engineering Lab — How junior vs. senior engineers adopted AI coding tools differently, and the cultural shift that followed

Agentic Coding in Practice — Navigating AI coding agent costs, quality control, and redefining the SDLC

Chinese Models & Compliance — Why enterprise customers block DeepSeek/Qwen and the geopolitics of model training data

This episode is for engineering leaders building AI in production, MLOps engineers optimizing GPU infrastructure, and anyone navigating the gap between AI demos and enterprise-scale deployment.

Links & Resources:

TensorRT LLM: https://github.com/NVIDIA/TensorRT-LLM

NVIDIA Run: ai Model Streamer (cold start optimization): https://developer.nvidia.com/blog/reducing-cold-start-latency-for-llm-inference-with-nvidia-runai-model-streamer/

vLLM vs TensorRT-LLM comparison: https://northflank.com/blog/vllm-vs-tensorrt-llm-and-how-to-run-them

Timestamps:

[00:00] Optimizing GPU Usage and Latency

[00:21] Learning AI as Leadership

[04:34] AI Cost Centers

[13:56] Throughput and Infrastructure Efficiency

[18:10] Scaling and Unit Economics

[24:14] Championing AI ROI

[36:11] Queue to Value Engine

[41:30] Failed Product Features

[46:12] Agentic Engineering Costs

[58:49] AI Self-Hosting in Engineering

[1:04:40] Wrap up