What is chaos engineering, and how does it differ from traditional testing approaches like load testing or fault injection?

Question

Accepted Answer

Chaos engineering is the discipline of experimenting on a system to build confidence in its ability to withstand turbulent conditions in production. It was pioneered by Netflix and formalized in the Principles of Chaos Engineering. While it shares goals with other testing approaches, its methodology and philosophy are distinct.

Traditional testing verifies that a system behaves correctly under known conditions. Unit tests check individual functions, integration tests check component interactions, and load tests check behavior under expected traffic volumes. These tests operate on the assumption that you can enumerate the scenarios worth testing. They answer the question "does the system do what we designed it to do?"

Chaos engineering operates on a different assumption: in complex distributed systems, failures emerge from unexpected interactions that no one anticipated. Instead of testing for known scenarios, chaos experiments explore the unknown by introducing real-world turbulence and observing whether the system maintains its expected behavior. Chaos engineering answers the question "what happens to user experience when things go wrong in ways we did not predict?"

The scientific method is central to chaos engineering. Every experiment follows a structured process. First, you define the steady state, which is the normal behavior of the system expressed in measurable terms like request success rate, latency percentiles, or business metrics like orders per minute. Second, you form a hypothesis: "we believe that when we introduce this failure, the system will maintain its steady state because of our resilience mechanisms." Third, you introduce the failure. Fourth, you observe whether the hypothesis holds or is disproven.

Fault injection is a technique used within chaos engineering, but it is not the same thing. Fault injection deliberately introduces specific failures such as network errors, process crashes, or disk failures. Chaos engineering uses fault injection as one of its tools but also encompasses broader experiments like introducing latency, throttling resources, or simulating entire availability zone failures. The key difference is that chaos engineering is an experimental discipline with hypotheses and observations, while fault injection is a technique for creating failure conditions.

Load testing focuses on capacity and performance under increasing demand. It answers "how much traffic can we handle?" Chaos engineering focuses on resilience under failure conditions at any traffic level. It answers "does our system degrade gracefully when components fail?" The two are complementary: a system might handle high load perfectly but fall apart when a single database replica goes offline.

A critical principle is that chaos experiments should be run in production or in an environment as close to production as possible. This is because the emergent behaviors that chaos engineering seeks to discover arise from the specific configuration, traffic patterns, and component interactions of the real system. Running chaos experiments only in staging misses many of the conditions that cause real outages.

The ultimate goal of chaos engineering is not to find bugs but to build confidence. Each experiment that validates a hypothesis increases the team's confidence that their resilience mechanisms work. Each experiment that disproves a hypothesis reveals a gap that can be fixed before it causes a real outage.