How to conduct a blameless post-mortem

Blameless post-mortems are critical to maintaining a positive culture of continuous improvement while also acknowledging room for growth.

The CrowdStrike incident affected millions globally, costing an estimated $1-2 billion in fixes. It underscores the critical need for learning from failures in our interconnected tech world.

What's a blameless post-mortem?

• A structured analysis of an incident that focuses on systemic issues, not individual blame.
• Aims to identify what happened, why, and how to prevent future occurrences. You stay focused on the actions, not the people.
• Emphasizes learning and improvement over punishment.

Why it's crucial:

• You’re intentional about maintaining open, honest communication about failures.
• You foster a culture of continuous improvement. (Missed last week’s newsletter? Here’s your chance!)
• You’re addressing root causes, not just symptoms. Root causes → longevity for overall system resilience.

Large-scale incidents rarely result from one person's mistake, if ever. These incidents are a sign of a process issue.

Here's how to conduct effective blameless post-mortems:

1. Understand what happened: Gather all relevant data about the incident. Focus on facts, not blame. List out action by action, along with a timeline of how each action played out, all the way to full recovery.
2. Analyze the impact: Consider immediate and long-term consequences. Look at effects across various stakeholders if applicable.
3. Identify root causes: Dig deep beyond surface-level issues. Look for systemic problems.
4. Generate action items: Be specific and actionable. Assign clear owners and deadlines, along with a plan to follow up.
5. Share learnings: Communicate these insights widely. Use learnings to inform future practices. We learn best from mistakes, and when appropriate, we learn from others when they share their mistakes openly. Bottom line: Blameless post-mortems transform crises into opportunities for systemic improvement. They're not about avoiding accountability, but creating an environment where honest analysis leads to real solutions.

Diving deeper

To maximize learning from incidents like the CrowdStrike-Microsoft outage, here's a comprehensive structure for conducting blameless post-mortems. Want a Notion template to use? Here you go.

Preparation (Before the meeting)

• Assign a neutral facilitator
• Gather all relevant data, logs, and timeline information
• Invite key stakeholders from all affected areas
• Ask key stakeholders to contribute their thoughts to the shared template above so they come prepared

Meeting structure

• Introduction and ground rules
  • Emphasize the blameless nature of the discussion
  • Encourage open and honest communication
• Incident overview
  • Review summary of what happened and its impact. All details should be included in your shared document
• Timeline reconstruction
  • Walk through the incident chronologically
  • Identify key decision points and actions taken
• Impact analysis
  • Discuss effects on various stakeholders (customers, employees, partners)
  • Quantify losses if possible (financial, reputational, operational)
• Root cause analysis
  • This is the bulk of your time. Consider technical, process, and cultural factors
  • Identify what went well too - not just what went wrong
  • Brainstorm preventive measures and improvements
  • Assign owners and deadlines for each item
• Wrap-up and next steps (5 minutes)
  • Summarize key takeaways and action items
  • Schedule follow-up meetings as needed

Some prodding questions

• What exactly happened? When did we first detect the issue?
• How did our response evolve over time?
• What went well in our response? What could have gone better?
• Were there any early warning signs we missed?
• How could we have detected the issue earlier?
• What dependencies or interconnections contributed to the spread of the problem?
• How can we improve our testing and rollout procedures?
• What changes do we need to make to our incident response process?

Applying this structure to the CrowdStrike-Microsoft incident

Here's how this template might be applied to the recent outage:

1. Incident Overview: On July 19, 2024, a CrowdStrike Falcon sensor update caused widespread "Blue Screen of Death" issues for Microsoft systems globally.
2. Timeline (it would be more granular with timestamps if I had them):
   • Update pushed to production
   • First reports of system failures
   • Issue identified and update halted
   • Emergency response team assembled
   • Mitigation steps communicated to customers
   • Systems gradually restored
3. Impact analysis:
   • Affected industries: healthcare, finance, manufacturing, retail, education
   • Estimated cost of fixes: $1-2 billion globally
   • (Stock analysis, if finance team is involved)
   • Reputational impact on both CrowdStrike and Microsoft
4. Root Cause Analysis:
   • Primary cause: Logic error in Falcon sensor configuration update
   • Contributing factors:
     • Inadequate testing scenarios for diverse environments
     • Lack of proper sandboxing for critical updates
     • Rapid rollout without phased approach
     • Microsoft OS vulnerability allowing repeated failures on reboot
5. Action Items:
   • Implement more comprehensive testing procedures
   • Develop sandbox environment mirroring diverse customer setups
   • Design phased rollout strategy for critical updates
   • Enhance monitoring for early detection of widespread issues
   • Collaborate with Microsoft on improving OS resilience to third-party update failures

Remember: the goal is not to assign blame, but to uncover systemic weaknesses and drive meaningful improvements in processes, technologies, and organizational culture.