What are the four core metric types in Prometheus, and when would you use each one?

Question

Accepted Answer

Prometheus defines four core metric types, each designed for different measurement scenarios. Understanding when to use each one is essential for building meaningful observability into your services.

A Counter is a cumulative metric that only ever increases or resets to zero on restart. It is perfect for tracking things that accumulate over time, such as the total number of HTTP requests served, errors encountered, or bytes processed. You never use a counter for a value that can decrease. When querying counters, you typically use the rate() or increase() functions in PromQL to calculate how fast the counter is growing over a time window, which gives you requests per second or errors per minute.

A Gauge represents a single numerical value that can go up and down. Think of it like a thermometer or a fuel gauge. Common use cases include current CPU usage, memory consumption, the number of active connections, or queue depth. Gauges are ideal for snapshots of current state. You can directly query a gauge's value without needing rate functions, and you often use functions like avg_over_time() or max_over_time() to understand trends.

A Histogram samples observations and counts them in configurable buckets. This is the go-to choice for measuring request latencies or response sizes when you want to understand the distribution. The histogram automatically creates multiple time series: cumulative bucket counters with a le label, a sum of all observed values, and a total count. You can then use histogram_quantile() in PromQL to calculate percentiles like p50, p95, or p99. A key decision is choosing your bucket boundaries, which should align with your SLO thresholds.

A Summary is similar to a histogram but calculates quantiles on the client side. It provides a count, a sum, and pre-calculated quantiles like the median or 95th percentile. Summaries are useful when you need accurate quantiles for a single instance, but they have a major trade-off: you cannot aggregate quantiles across multiple instances. If you have ten pods and each reports its own p99, you cannot combine those into a meaningful overall p99. For this reason, histograms are generally preferred in distributed systems.

In practice, most mid-level SRE work revolves around counters for throughput and error tracking, gauges for resource utilization, and histograms for latency measurement. The choice between histogram and summary comes down to whether you need aggregation across instances, which in a Kubernetes environment you almost always do. A good rule of thumb is to default to histograms unless you have a specific reason to use summaries.