Operational Challenges with Inference on Kubernetes

#344: Kubernetes is boring now. That's the whole point. KubeCon EU 2026 in Amsterdam -- likely the biggest KubeCon ever at more than 13,000 attendees -- made one thing extremely clear: the container orchestrator is done being interesting on its own. Every keynote, every new sandbox project, every vendor announcement pointed the same direction. AI. Inference. Agents.

NVIDIA donated a DRA driver for GPUs to CNCF. Google open-sourced their cluster autoscaler and shipped a DRA driver for TPUs. Red Hat brought LLM-D for disaggregated inference. NVIDIA contributed the KAI Scheduler for AI workloads. The Gateway API now has an inference extension in beta -- model routing baked directly into the Kubernetes networking layer. And here's the thing Whitney pointed out that should make everyone pause: you can't even run inference workloads in containers. They can escape. You need micro VMs. So the container orchestrator is orchestrating things that aren't containers.

The platform engineering conversation shifted too. The bottleneck isn't technology anymore -- it's culture. Getting teams to work together differently. And if your company can't trust its own employees to make decisions, good luck trusting agents. Viktor's take on the determinism objection was blunt: agents aren't deterministic, but neither are you. You just think you are.

One thread that kept surfacing: agents as first-class platform users. Not agents doing agent things -- agents as the users your platform serves. Viktor sees it in real time -- pull requests created by agents, reviewed by his Claude, responses written by the submitter's agent. Humans aren't even in the conversation anymore.

The new CNCF sandbox projects tell the story too. LLM-D, KAI Scheduler, Higress (AI-native gateway). And then Velero -- the Kubernetes backup tool that everyone assumed was already CNCF -- finally donated by Broadcom. Which raises a fair question: is CNCF becoming a dumping ground for projects companies don't want to maintain? Probably some of both.

Viktor compared the current state to the first five years of Kubernetes -- everyone focused on low-level components, trying to figure out how to combine 57 different tools. The next wave will be higher-level platforms that bundle all of it. And somewhere underneath it all, the mainframe keeps running. Viktor's bet: it'll outlive AI.

YouTube channel:

https://youtube.com/devopsparadox

Review the podcast on Apple Podcasts:

https://www.devopsparadox.com/review-podcast/

Slack:

https://www.devopsparadox.com/slack/

Connect with us at:

https://www.devopsparadox.com/contact