What is the difference between latency and throughput, and how do they relate to system performance?

Question

Accepted Answer

Latency and throughput are two fundamental metrics for measuring system performance, but they measure different things.

Latency is the time it takes for a single request to complete, from when it is sent until the response is received. It is measured in time units like milliseconds. Low latency means fast individual responses. For example, if a database query takes fifty milliseconds, that is the query latency.

Throughput is the number of requests a system can handle per unit of time. It is measured in requests per second, transactions per second, or similar units. High throughput means the system can process many requests. For example, a server handling one thousand requests per second has that throughput.

These metrics are related but distinct. You can have low latency with low throughput, like a single fast worker handling one request at a time. You can have high throughput with higher latency, like a batch processing system that handles many requests but each takes longer.

The relationship between them is often inversely affected by load. As a system approaches its throughput limit, latency typically increases. Imagine a highway. When traffic is light, cars move fast with low latency. As more cars enter and throughput increases, speeds decrease and latency rises. Eventually, adding more cars causes congestion where both latency suffers and effective throughput drops.

Different applications prioritize differently. Real-time applications like gaming or video calls prioritize low latency. Batch processing systems like report generation prioritize high throughput. Most web applications need a balance, wanting acceptable latency while maximizing the number of users they can serve.

When optimizing, understand which metric matters most for your use case. Optimizing for one often involves trade-offs with the other. Adding caching might reduce latency but not necessarily increase throughput if the bottleneck is elsewhere.