What are the responsibilities of an incident commander, and how do you structure an effective incident response process?

Question

Accepted Answer

The incident commander role is the cornerstone of effective incident response, borrowed from the Incident Command System used by emergency services and adapted for technology organizations. As a senior SRE, I have served as incident commander for hundreds of incidents and have designed incident management processes for multiple organizations.

The incident commander's primary responsibility is coordination, not technical problem-solving. This is the most important and most frequently misunderstood aspect of the role. The IC manages the incident process: declaring severity, assembling the response team, directing investigation efforts, making mitigation decisions, and communicating status. The IC should not be debugging code or running queries because doing so compromises their ability to maintain situational awareness across the entire incident.

I structure the incident response process around five phases. Detection and declaration begins when monitoring or user reports indicate a problem. The on-call responder performs initial triage and, if the issue meets severity criteria, declares an incident and pages the incident commander. Clear severity definitions prevent both under-escalation where serious issues languish and over-escalation where minor issues trigger expensive response.

Mobilization is where the IC assembles the response team. I maintain an on-call roster that maps incident types to the relevant experts: database incidents page the DBA on-call, network issues page the network engineer, and application issues page the service owner. The IC creates a dedicated communication channel, typically a Slack channel or bridge call, and establishes a shared incident document for real-time note-taking.

Investigation and mitigation is where the technical work happens. The IC delegates specific investigation tasks to responders, tracks progress, and makes decisions about mitigation approaches. A critical IC skill is time-boxing investigations: if the root cause is not identified within 15 minutes, shift focus to mitigation through techniques like rollback, traffic shifting, or feature disabling that restore service before root cause is found. Faster mitigation means shorter user impact even if the root cause investigation continues afterward.

Communication is an ongoing responsibility throughout the incident. The IC provides regular status updates to stakeholders through predefined channels. I recommend updates every 15 to 30 minutes for high-severity incidents, even if the update is simply that investigation continues. These updates should include what is known, what is being done, the estimated time to resolution, and the current user impact. Stakeholder communication prevents the IC from being interrupted by executives and customer support requesting updates.

Resolution and handoff completes the active incident. Once the service is restored, the IC confirms with monitoring and affected users, documents the timeline and resolution, and schedules the postmortem. For extended incidents, IC handoff procedures ensure continuity when the original IC needs rest. I require a structured handoff that includes current status, actions in progress, decisions made, and outstanding concerns.

I define three severity levels with clear criteria. SEV1 indicates significant user impact or revenue loss requiring immediate response and executive notification. SEV2 indicates partial impact with workarounds available requiring prompt response. SEV3 indicates minor impact requiring scheduled response. Each severity has defined response time expectations, communication cadences, and escalation paths.

The IC rotation ensures the burden is shared and skills are developed across the team. I train new ICs through shadowing, where they observe experienced ICs during incidents, then reverse-shadowing where they lead with an experienced IC observing and providing feedback. Regular incident response drills using realistic scenarios, sometimes called game days, build IC skills without waiting for real incidents.