The Changelog: Software Development, Open Source

Learning from incidents (Interview)

Feb 4, 2022

This week, Nora Jones, Founder and CEO of Jeli, shares her insights from chaos engineering at Netflix and incident analysis at Slack. She emphasizes the importance of learning from incidents to improve team resilience. Nora discusses creating developer-centric tools and the emotional complexities of incident reviews. She explores knowledge silos, the balance between quantitative and qualitative insights, and the evolving role of incident analysts. Additionally, she reflects on how studying real-world incidents can enhance software practices and decision-making.

Ask episode

AI Snips

Chapters

Books

Transcript

Episode notes

ADVICE

Third-Party Incident Reviews

Have a third party, uninvolved in the incident, conduct the review for an objective perspective.
This approach reduces emotional bias and promotes a more thorough analysis.

ANECDOTE

Sully vs. Costa Concordia

Captain Sully's deviation from the runbook saved lives, but a similar situation with the Costa Concordia led to jail time.
This highlights the conflict between expertise and strict adherence to procedures.

ADVICE

The Role of Runbooks

Use runbooks for learning, but recognize that expertise surpasses them.
Documenting expertise reduces the reliance on constantly updating runbooks.

Get the Snipd Podcast app to discover more snips from this episode

Get the app

This week we’re joined by Nora Jones, founder and CEO at Jeli where they help teams gain insight and learnings from incidents. Back in December Nora shared here thoughts in a Changelog post titled “Incident” shouldn’t be a four-letter word - which got a lot of attention from our readers. Today we’re talking with Nora about all things incidents — the learning and growth they represent for teams, why teams should focus on learning from incidents in the first place, their Howie guide to post‑incident investigations, why the next emerging role is an Incident Analyst, and she also shares a few book recommendations which we’ve linked up in the show notes.

Join the discussion

Changelog++ members get a bonus 10 minutes at the end of this episode and zero ads. Join today!

Sponsors:

InfluxData – InfluxDB empowers developers to build IoT, analytics, and monitoring software. It’s purpose-built to handle massive volumes and countless sources of time-stamped data produced by sensors, applications, and infrastructure. Learn about the wide range of use cases of InfluxDB at influxdata.com/changelog
Sentry – Working code means happy customers. That’s exactly why teams choose Sentry. From error tracking to performance monitoring, Sentry helps teams see what actually matters, resolve problems quicker, and learn continuously about their applications - from the frontend to the backend. Use the code CHANGELOG and get the team plan free for three months.
WorkOS – A platform that gives developers a set of building blocks for quickly adding enterprise-ready features to their application. Add Single Sign-On (Okta, Azure, Google, Microsoft OAuth), sync users from any SCIM directory, HRIS integration, audit trails (SIEM), free magic link sign-in. WorkOS is designed for developers and offers a single, elegant interface that abstracts dozens of enterprise integrations. Learn more and get started at WorkOS.com
Subspace – Network-as-a-Service that helps developers accelerate real-time applications for hundreds of millions of users worldwide. When every millisecond counts, Subspace gives you the fastest, most reliable network to route your traffic through. And it all works via a global IP proxy that sets up using a simple API. Learn more and get started for free at subspace.com/changelog

Featuring:

Nora Jones – LinkedIn, X
Adam Stacoviak – Website, GitHub, LinkedIn, Mastodon, X
Jerod Santo – Website, GitHub, LinkedIn, Mastodon, X

Show Notes:

Something missing or broken? PRs welcome!