Overview

Managing incidents and conducting effective postmortems.

Incident Severity Levels

P0 - Critical

• Impact: Service completely down or major functionality unavailable
• Response: Immediate, all-hands
• Communication: Every 30 minutes
• Examples: Complete outage, data loss, security breach

P1 - High

• Impact: Significant degradation affecting many users
• Response: Immediate, primary on-call
• Communication: Every hour
• Examples: Elevated error rates, slow response times

P2 - Medium

• Impact: Minor degradation or single component affected
• Response: Next business day
• Communication: Daily updates
• Examples: Single region issue, non-critical feature down

P3 - Low

• Impact: No user impact yet, potential future issue
• Response: Track in backlog
• Communication: Async
• Examples: Monitoring gaps, capacity warnings

Incident Response Process

1. Detection

Alert fires → On-call acknowledges → Initial assessment

2. Triage

- Assess severity
- Page additional responders if needed
- Establish incident channel
- Assign incident commander

3. Mitigation

- Identify mitigation options
- Execute fastest safe mitigation
- Monitor for improvement
- Escalate if not improving

4. Resolution

- Verify service health
- Communicate resolution
- Document actions taken
- Schedule postmortem

5. Follow-up

- Conduct postmortem
- Identify action items
- Track completion
- Update runbooks

Incident Roles

Incident Commander (IC)

• Owns incident response
• Makes decisions
• Coordinates responders
• Manages communication
• Declares incident resolved

Operations Lead

• Executes technical remediation
• Proposes mitigation strategies
• Implements fixes
• Tests changes

Communications Lead

• Updates status page
• Posts to incident channel
• Notifies stakeholders
• Prepares external messaging

Planning Lead

• Tracks action items
• Takes detailed notes
• Monitors responder fatigue
• Coordinates shift changes

Communication Templates

Initial Notification

🚨 INCIDENT DECLARED - P0

Service: API Gateway
Impact: All API requests failing
Started: 2024-01-15 14:23 UTC
IC: @alice
Status Channel: #incident-001

Current Status: Investigating
Next Update: 30 minutes

Status Update

📊 INCIDENT UPDATE #2 - P0

Service: API Gateway
Elapsed: 45 minutes

Progress: Identified root cause as database connection pool exhaustion.
Mitigation: Increasing pool size and restarting services.

ETA to Resolution: 15 minutes
Next Update: 15 minutes or when resolved

Resolution Notice

✅ INCIDENT RESOLVED - P0

Service: API Gateway
Duration: 1h 12m
Impact: 100% of API requests failed

Resolution: Increased database connection pool and restarted services.

Next Steps:
- Postmortem scheduled for tomorrow 10am
- Monitoring for recurrence
- Action items being tracked in #incident-001

Blameless Postmortem

Template

# Incident Postmortem: API Outage 2024-01-15

## Summary

On January 15th, our API was completely unavailable for 72 minutes due to
database connection pool exhaustion.

## Impact

- Duration: 72 minutes (14:23 - 15:35 UTC)
- Severity: P0
- Users Affected: 100% of API users (~50,000 requests failed)
- Revenue Impact: ~$5,000 in SLA credits

## Timeline

**14:23** - Alerts fire for elevated error rate
**14:25** - IC paged, incident channel created
**14:30** - Identified all database connections exhausted
**14:45** - Decided to increase pool size
**15:00** - Configuration deployed
**15:15** - Services restarted
**15:35** - Error rate returned to normal, incident resolved

## Root Cause

Database connection pool was sized for normal load (100 connections).
Traffic spike from new feature launch (3x normal) exhausted connections.
No alerting existed for connection pool utilization.

## What Went Well

- Detection was quick (2 minutes from issue start)
- Team assembled rapidly
- Clear communication maintained

## What Didn't Go Well

- No capacity testing before feature launch
- Connection pool metrics not monitored
- No automated rollback capability

## Action Items

1. [P0] Add connection pool utilization monitoring (@bob, 1/17)
2. [P0] Implement automated rollback for deploys (@charlie, 1/20)
3. [P1] Establish capacity testing process (@diana, 1/25)
4. [P1] Increase connection pool to 300 (@bob, 1/16)
5. [P2] Update deployment runbook with load testing (@eve, 1/30)

## Lessons Learned

- Always load test before launching features
- Monitor resource utilization at all layers
- Have rollback mechanisms ready

Runbooks

Example Runbook

# Runbook: High Database Latency

## Symptoms

- Database query times > 500ms
- Elevated API latency
- Alert: DatabaseLatencyHigh

## Impact

Users experience slow page loads. P1 severity if p95 > 1s.

## Investigation

1. Check database metrics in Grafana
   https://grafana.example.com/d/db-overview

2. Identify slow queries:
   ```sql
   SELECT * FROM pg_stat_statements 
   ORDER BY total_time DESC LIMIT 10;

1. Check for locks:

   SELECT * FROM pg_stat_activity 
   WHERE state = 'active';
   

Mitigation

Quick fixes:

• Kill long-running queries if safe
• Add missing indexes if identified
• Scale up read replicas if read-heavy

Escalation: If latency > 2s for > 15 minutes, page DBA team.

Prevention

• Regular query performance reviews
• Automated index recommendations
• Capacity planning for growth

## Best Practices

### Blameless Culture

- Focus on systems, not individuals
- Assume good intentions
- Learn from mistakes
- Reward transparency

### Clear Severity Definitions

- Severity should be based on user impact
- Document response time expectations
- Update definitions based on learnings

### Practice Incident Response

- Run "game days" quarterly
- Practice different scenarios
- Test on-call handoffs
- Review and improve runbooks

### Track Action Items

- Assign owners and due dates
- Review in team meetings
- Close loop on completion
- Measure time to completion