KubeFM The Data Engineer's guide to optimizing Kubernetes, with Niels Claeys
Oct 14, 2025
Niels Claeys, a lead engineer at Dataminded and expert in Kubernetes optimization, shares insights on building Conveyor, a data platform processing over 1.5 million core hours monthly. He reveals how switching scheduler strategies can cut costs significantly while enhancing resource use. Niels also discusses achieving 97% spot instance utilization and the importance of multi-type diversification. He emphasizes the need for simplicity in coding and effective communication in tech, alongside practical tips for scaling and optimizing workloads.
AI Snips
Chapters
Transcript
Episode notes
Bin Pack By Preferring Most-Allocated
- Switch scheduler strategy from least-allocated to most-allocated to consolidate pods and enable faster scale-down.
- This change typically yields immediate 10–15% cost savings without user changes.
Exploit Spot Instances Smartly
- Use spot instances for batch workloads to cut compute costs 70–90%.
- Reduce interruptions via instance-type diversification, region choice, and running drivers on on-demand while executors use spot.
Reduce Spot Risk And Mitigate Interruptions
- Minimize spot-term risk with AWS Spot Placement API and broad instance-type support.
- Mitigate impact by running critical roles on on-demand and leveraging Spark decommissioning to avoid full job failure.