Screaming in the Cloud

When AWS has a major outage, what actually happens behind the scenes? Ben Hartshorne, a principal engineer at Honeycomb, joins Corey Quinn to discuss a recent AWS outage and how they kept customer data safe even when their systems couldn't fully work. Ben explains why building services that expect things to break is the only way to survive these outages. Ben also shares how Honeycomb used its own tools to cut their AWS Lambda costs in half by tracking five different things in a spreadsheet and making small changes to all of them.About Ben Hartshorne: Ben has spent much of his career setting up monitoring systems for startups and now is thrilled to help the industry see a better way. He is always eager to find the right graph to understand a service and will look for every excuse to include a whiteboard in the discussion.Show highlights: (02:41)Two Stories About Cost Optimization(04:20) Cutting Lambda Costs by 50%(08:01) Surviving the AWS Outage(09:20) Preserving Customer Data During the Outage(13:08) Should You Leave AWS After an Outage?(15:09) Multi-Region Costs 10x More(18:10) Vendor Dependencies(22:06) How LaunchDarkly's SDK Handles Outages(24:40) Rate Limiting Yourself(29:00) How Much Instrumentation Is Too Much?(34:28) Where to Find BenLinks: Linkedin: https://www.linkedin.com/in/benhartshorne/GitHub: https://github.com/maplebedSponsored by: duckbillhq.com

R. Tyler Croy, an infrastructure architect at Scribd and veteran open-source developer, discusses the staggering costs associated with managing billions of S3 objects. He reveals how normal assumptions break down under extreme scale and why engineering solutions are essential. Tyler emphasizes innovative data strategies, like packing files into Parquet, to minimize object counts and reduce expenses. He also explores how AI is transforming old documents into valuable assets, driving new storage priorities in a rapidly evolving tech landscape.

Avery Pennarun, co-founder and CEO of Tailscale, is a veteran software engineer revolutionizing secure networking. He shares how Tailscale transforms VPNs into user-friendly tools and tackles AI security with zero-click authentication. Avery discusses the chaos of running multiple tailnets and the challenges of scaling during rapid growth. He introduces TSIDP for effortless OAuth and talks about bridging the gap between personal and corporate networks. Expect insights sprinkled with humor on making security both powerful and approachable.

Jeremy Price, VP of Technology at Grokability and key player behind the Snipe-IT open source project, shares insights on building a sustainable business model without VC pressure. He discusses how Grokability prioritizes product quality over explosive growth and the importance of customer relationships when they pay for software. Jeremy highlights the success of running thousands of separate installations and the joy of creating 'boring' yet profitable tools that meet real needs without succumbing to market hype.

In this engaging discussion, Keith Townsend, founder of The CTO Advisor and an expert in cloud and AI, reveals the stark contrast between AI hype and its real-world application. He shares a cautionary tale about a biopharma company's rejection of Microsoft Copilot, highlighting enterprise fear of reputational risk. Keith also explores how AI has boosted his personal productivity tenfold, while cautioning that enterprises treat powerful tools like 'radioactive material.' The conversation touches on AI’s strengths in productivity but warns of its limitations in judgment, underscoring the challenges enterprises face in adoption.

Dev Rishi, GM of AI at Rubrik and a former machine learning CEO, shares insights on enterprise AI adoption and the evolution of agentic systems. He discusses the challenges enterprises face with AI, emphasizing the gap between consumer excitement and organizational risk aversion. Dev introduces Rubrik's innovative Agent Rewind, a safety net for AI-driven actions, helping prevent costly data loss. The conversation also covers trends in AI support, the importance of observability, and the role of governance in ensuring resilience in this rapidly changing landscape.

André Arko, CEO of Spinel Cooperative and longtime Bundler maintainer, discusses RV, a groundbreaking Ruby tool that installs Ruby in one second using precompiled binaries. He dives into the challenges of Ruby dependency management, contrasting RV with legacy tools like RVM and rbenv. André addresses misconceptions about Ruby's relevance, critiques Apple's outdated Ruby shipping in macOS, and shares his journey from nonprofit struggles to building a sustainable cooperative model that charges companies for expertise in open source.

Anneka Gupta, Chief Product Officer at Rubrik, shares her expertise in data security and cyber resilience. She discusses the evolution of backup into modern cyber recovery, emphasizing the need for organizations to prioritize recovery simulations amidst increasing ransomware threats. Gupta highlights the importance of the 'assume breach' strategy and how attacking backups has become a common tactic in cyber warfare. With AI's role in enhancing recovery operations, she illustrates how a unified platform can mitigate risks in multi-cloud environments.

Deana Solis, a seasoned FinOps engineer and 2022 FinOps Foundation Evangelist of the Year, shares her fascinating journey from electrical engineering to healthcare IT. She discusses why cloud certifications are often performative and the dramatic impact AI can have on AWS billing. Delving into cloud repatriation, she humorously critiques conspiracy theories surrounding it. Deana emphasizes the importance of communication between engineering and finance teams, making FinOps both relatable and engaging.

In this conversation with Chris Weichel, CTO of Ona, they delve into the exciting future of software development with AI agents at the helm. Chris shares insights on the rebranding from Gitpod to Ona, emphasizing the shift towards agent-driven coding. He discusses the essential role of safe, isolated environments for these agents and the fascinating 'five slot machines' effect of multi-agent workflows. With his expertise, he highlights the potential of machine-readable interfaces and how AI is redefining developer tools and productivity.