Explain the architecture of Apache Kafka. What are topics, partitions, brokers, producers, and consumers?

Question

Accepted Answer

Apache Kafka is a distributed event streaming platform designed for high-throughput, fault-tolerant, real-time data pipelines. Understanding its architecture is fundamental for any data engineer working with streaming data.

Topics are logical channels for organizing data. Each topic represents a category of events, such as user-clicks, order-events, or sensor-readings. Producers write messages to topics, and consumers read from them. Topics decouple producers from consumers, meaning producers do not need to know which consumers will read their data.

Partitions are the unit of parallelism within a topic. Each topic is divided into one or more partitions, and each partition is an ordered, immutable sequence of records. Messages within a partition have a sequential offset that uniquely identifies them. Partitions are distributed across brokers in the cluster, enabling horizontal scaling. More partitions allow more consumers to read in parallel.

Brokers are the servers that form the Kafka cluster. Each broker stores a subset of partitions and handles read and write requests. One broker acts as the controller, managing partition assignments and leader elections. For fault tolerance, each partition has one leader and multiple replicas on different brokers. The leader handles all reads and writes, while replicas stay synchronized. If a leader fails, a replica is promoted.

Producers write records to topics. They determine which partition receives each message, either through a specified key, which is hashed to a partition, or through round-robin distribution when no key is provided. Using keys ensures that all messages with the same key go to the same partition, preserving order for that key. Producers can choose acknowledgment levels: fire-and-forget for maximum speed, leader acknowledgment for a balance, or all-replica acknowledgment for maximum durability.

Consumers read records from topics. They track their position using offsets, allowing them to resume from where they left off after a restart. Consumer groups enable parallel consumption: each partition is assigned to exactly one consumer in the group, so the group processes all partitions collectively. Adding consumers to a group increases throughput up to the number of partitions.

ZooKeeper or KRaft manages cluster metadata. Traditional Kafka deployments use ZooKeeper for broker coordination, leader election, and configuration. Newer versions use KRaft, an internal consensus protocol that eliminates the ZooKeeper dependency and simplifies operations.

Kafka retains messages for a configurable period regardless of whether they have been consumed. This retention enables consumers to reprocess historical data, supporting use cases like backfilling or replaying events after a bug fix.