How do you define toil in an SRE context, and what methods do you use to measure it accurately across teams?

Question

Accepted Answer

Toil in SRE has a specific definition that goes beyond simply meaning work that is unpleasant. Understanding this definition precisely is essential for a senior SRE because mislabeling work as toil leads to misallocated automation efforts, while failing to recognize toil allows it to silently consume engineering capacity.

Google's SRE book defines toil as work that is manual, repetitive, automatable, tactical, devoid of enduring value, and scales linearly with service growth. All six characteristics must apply. Overhead tasks like meetings and planning are not toil even if tedious. Building a new monitoring dashboard is not toil because it provides enduring value. Investigating a novel outage is not toil because it is not repetitive. However, manually restarting a service when it runs out of memory every week, running certificate rotation scripts monthly, or manually provisioning resources for each new team are classic examples of toil.

For measurement, I use three complementary approaches. The first is time tracking through categorized work logs. I ask SREs to categorize their time into buckets: toil, engineering projects, overhead, and learning. This can be as simple as a weekly survey or as structured as time codes in ticket systems. The key is making it low-friction so the measurement itself does not become toil. I track the percentage of time spent on toil per engineer and per team over time, aiming for Google's benchmark of no more than 50% toil.

The second approach is ticket analysis. I examine operational tickets over the past quarter, categorizing them by type and looking for patterns. Tickets that recur with similar root causes and similar resolution steps indicate toil. I track metrics like ticket volume per service, mean time to resolution, and the ratio of automated versus manual resolution. Services generating a disproportionate number of manual tickets are prime candidates for automation investment.

The third approach is interrupt-driven work analysis. I instrument on-call systems to capture every alert and page, categorize them, and measure the response. Alerts that require the same manual intervention each time are toil. I track the alert-to-toil ratio: what percentage of alerts result in manual work versus being informational or auto-resolved? This metric directly connects observability investment to toil reduction.

Beyond measuring volume, I measure the impact of toil on team health and productivity. Surveys capturing SRE satisfaction, burnout indicators, and perceived time for engineering work provide qualitative signals that complement quantitative metrics. High toil often correlates with lower job satisfaction, higher attrition, and reduced innovation.

I present toil measurements to leadership using a cost model. If an SRE's time costs the organization a certain amount annually and 60% of their time is toil, that quantifies the waste. Comparing the cost of continued toil against the investment required for automation makes the business case concrete. I maintain a toil backlog ranked by time saved per engineering hour invested, which I call the automation ROI, to prioritize elimination efforts systematically.

Regular toil reviews, held monthly or quarterly, keep measurement current and prevent toil from creeping back after being eliminated. These reviews celebrate automation wins, identify new toil sources, and adjust priorities based on changing service landscapes.