What is Prometheus, and how does it collect and store metrics? Explain the pull-based model.

Question

Accepted Answer

Prometheus is an open-source monitoring and alerting toolkit that has become the standard for metrics collection in cloud-native environments. Originally built at SoundCloud and now a graduated CNCF project, it is designed for reliability and simplicity, which makes it a natural fit for SRE work.

Prometheus uses a pull-based model, which means it actively reaches out to your services to collect metrics rather than waiting for services to push data to it. Each service exposes a metrics endpoint, typically at /metrics, in a simple text format. Prometheus is configured with a list of targets (services to monitor) and scrape intervals (how often to collect), and it periodically fetches metrics from each target.

The pull model has several advantages. Prometheus knows immediately if a target is down because the scrape fails, which is itself useful information. You can easily run Prometheus on your laptop and point it at targets for testing. The configuration is centralized in Prometheus rather than distributed across every service. And you avoid overwhelming the monitoring system during traffic spikes because the collection rate is controlled by Prometheus, not the services.

Prometheus stores data as time series, where each series is identified by a metric name and a set of key-value labels. For example, http_requests_total{method="GET", status="200", handler="/api/users"} is a specific time series tracking successful GET requests to the users endpoint. This labeling system is incredibly flexible and allows powerful querying.

The four core metric types are counters (only go up, like total requests), gauges (can go up and down, like current temperature or active connections), histograms (sample observations into configurable buckets, good for latency), and summaries (similar to histograms but calculate quantiles on the client side).

PromQL is the query language used to select and aggregate time series data. For example, rate(http_requests_total[5m]) calculates the per-second request rate averaged over five minutes. You can filter by labels, aggregate across dimensions, and perform mathematical operations.

Service discovery mechanisms allow Prometheus to automatically find targets in dynamic environments. Instead of manually listing every server, Prometheus can discover targets from Kubernetes, Consul, AWS EC2, DNS, and many other sources. This is essential for cloud environments where servers come and go.

Prometheus is designed for reliability over absolute accuracy. It stores data locally and operates independently, so even if the network is partitioned, your monitoring keeps working on each Prometheus instance.