Why Caches Can Cause Metastable Failures

In this episode, I talked to Marc Brooker, a distinguished engineer at AWS who started there as a new grad and rose through the ranks. We discussed technical learnings from 3,000+ cloud system postmortems, how software engineering is changing with AI, how to find impactful problems and much more.

🔶 My keyboard Kickstarter: https://www.kickstarter.com/projects/ryanlpeterman/compose-simple-ergonomics-beautifully-done

𝗣𝗼𝗱𝗰𝗮𝘀𝘁 𝗹𝗶𝗻𝗸𝘀:

• YouTube: https://youtu.be/u3GjIXP9N0s

• Spotify: https://open.spotify.com/episode/1qX2GfpbzxzGpGvDZVINdO?si=wsDGZo9PTbCNalKVybFVnA

• Apple: https://podcasts.apple.com/us/podcast/the-peterman-pod/id1777363835

• Transcript: https://www.developing.dev/p/aws-distinguished-eng-learnings-from

𝗘𝗽𝗶𝘀𝗼𝗱𝗲 𝗹𝗶𝗻𝗸𝘀:

• Post we discussed on hobbies and apparent expertise: https://brooker.co.za/blog/2023/04/20/hobbies.html

• Post on software engineering changing: https://brooker.co.za/blog/2026/02/07/you-are-here.html

• Post about Senior engineers and AI: https://brooker.co.za/blog/2026/03/20/ic-leadership.html

• Post on Junior engineers and AI: https://brooker.co.za/blog/2026/03/25/ic-junior.html

𝗧𝗶𝗺𝗲𝘀𝘁𝗮𝗺𝗽𝘀:

0:00 - Intro

1:27 - Finding problems that matter

11:42 - Learnings from 3000 postmortems

23:58 - Why caches are bad

29:37 - How AI will change software engineering

36:49 - Advice for junior engineers given AI

44:02 - Thoughts for senior engineers

49:59 - Why engineers should write

57:51 - Visibility and apparent expertise

1:04:23 - AWS engineers he admires

1:06:53 - Technical book recommendations

1:09:06 - Advice for his younger self

1:10:37 - Outro

𝗪𝗵𝗲𝗿𝗲 𝘁𝗼 𝗳𝗶𝗻𝗱 𝗠𝗮𝗿𝗰:

• LinkedIn: https://www.linkedin.com/in/marc-brooker-b431772b/

• Twitter/X: https://x.com/MarcJBrooker

• Personal Blog: https://brooker.co.za/blog/

𝗪𝗵𝗲𝗿𝗲 𝘁𝗼 𝗳𝗶𝗻𝗱 𝗥𝘆𝗮𝗻:

• Newsletter: https://www.developing.dev/

• X/Twitter: https://x.com/ryanlpeterman

• LinkedIn: https://www.linkedin.com/in/ryanlpeterman/

• Threads: https://www.threads.com/@ryanlpeterman

• Instagram: https://www.instagram.com/ryanlpeterman

• TikTok: https://www.tiktok.com/@ryanlpeterman