Managing Kafka

Kafka Raft (KRaft) protocol

Zookeeper (deprecated)

• It manages broker i.e keeps a list of them.

• It helps in performing leader election for partitions.

• It sends notifications to Kafka in case of changes i.e., broker changes like new broker added, broker dies, delete topics etc.

• Upto Kafka 2.x cannot work without Zookeeper.

• From 3.x can work without Zookeeper using (KIP-500), using Kafka Raft instead.

• From 4.x will not have Zookeeper.

• Works with odd number of servers.

• They also have leaders and followers concept.

• Zookeeper does not store consumer offset with Kafka > v0.10. Now it is stored in internal Kafka topic.

Since Kafka 2.2 the kafka-topics.sh CLI command references Kafka brokers and not Zookeeper for topic management and Zookeeper CLI argument is deprecated.

• The reason Zookeeper is deprecated is because it is less secure than Kafka, and therefore Zookeeper ports should only be opened to allow traffic from Kafka brokers, and not Kafka clients. More about it can be read here and here.

Zookeeper Cluster

Zookeeper Cluster