What is a production incident, and how do SRE teams typically classify incident severity levels?

Question

Accepted Answer

A production incident is any unplanned event that disrupts or degrades a service for users. This includes complete outages, performance degradation, data inconsistencies, security breaches, and any situation where the service is not meeting its reliability targets. Incidents are a normal part of operating complex systems, and having a clear framework for handling them is what separates reactive firefighting from professional incident management.

Severity classification helps teams prioritize response effort and communication. While exact definitions vary by organization, a common four-level system works like this.

SEV1 or Critical means a complete outage or severe degradation affecting all or most users. Revenue-impacting features are completely unavailable. Examples include the entire website being down, payment processing failing for all customers, or a data breach in progress. Response is immediate with all-hands engagement, executive communication, and status page updates.

SEV2 or High means significant degradation or partial outage affecting a substantial subset of users. Core functionality is impaired but not completely unavailable. Examples include slow response times causing timeouts for 30% of users, a specific region being unable to access the service, or a secondary feature like search being completely broken. Response is urgent with the on-call engineer plus additional support as needed.

SEV3 or Medium means minor degradation that affects a limited number of users or a non-critical feature. The core service works but something is noticeably wrong. Examples include elevated error rates on a non-critical API endpoint, a monitoring dashboard being unavailable, or an internal tool being slow. Response is during business hours with normal prioritization.

SEV4 or Low means a cosmetic issue, minor bug, or potential problem that has not yet impacted users. Examples include a typo on a low-traffic page, a warning in logs that has not caused errors yet, or a non-critical dependency showing intermittent issues. Response is scheduled into normal sprint work.

The classification determines who gets notified, how quickly the team must respond, what communication channels are used, and whether a formal postmortem is required. Most organizations require postmortems for SEV1 and SEV2 incidents.

Clear severity definitions prevent two common problems: under-escalation where a serious issue is treated as minor because nobody wants to raise alarm, and over-escalation where minor issues trigger maximum response and exhaust the team. When in doubt, it is better to escalate and downgrade than to under-escalate and discover the problem is worse than expected.