Kafka Commands
Topic management
kafka-topics.sh --bootstrap-server localhost:9092 --list
List topics on broker.
kafka-topics.sh --bootstrap-server localhost:9092 --create --topic first_topic
Create a topic named first_topic in broker.
kafka-topics.sh --bootstrap-server localhost:9092 --create --topic second_topic --partitions 3
Create a topic named first_topic in the broker with 3 partitions.
kafka-topics.sh --bootstrap-server localhost:9092 --create --topic third_topic --partitions 3 --replication-factor 2
Create a topic named first_topic in the broker with 3 partition and 2 replication factor.
Producer Command
kafka-console-producer.sh --bootstrap-server localhost:9092 --topic first_topic <message>
Produce messages to topic named first_topic.
kafka-console-producer.sh --bootstrap-server localhost:9092 --topic first_topic --producer-property acks=all <message>
Produce messages to topic named first_topic with acknowledgement set to all.
kafka-console-producer.sh --bootstrap-server localhost:9092 --topic first_topic --property parse.key=true --property key.separator=: <message>
Produce message to topic named first_topic and with key specified in the message of the form
kafka-console-producer.sh --bootstrap-server localhost:9092 --producer-property partitioner.class=org.apache.kafka.clients.producer.RoundRobinPartitioner --topic first_topic
Produce message to the topic first_topic, this will produce to one partitioner at a time. Don't use this partitioner in production.
Consumer Command
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic first-topic
Consume from the topic named first-topic
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic first-topic --from-beginning
Consume from the topic named first-topic from the beginning the message may come out of order if they are being read from topic with multiple partition. Note the consumer offset must be 0 to be read from the beginning, if already some read happened then it will start from last committed offset.
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic first-topic --formatter org.apache.kafka.tools.consumer.DefaultMessageFormatter --property print.timestamp=true --property print.key=true --property print.value=true --property print.partition=true --from-beginning
Prints message from given topic in a format specified by formatter.
Consumer group commands
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic first-topic --group <group Id>
Consume from the topic named first-topic and consumer belongs to a consumer group. Please not sticky partition causes messages to routed to one partition so use keys in messages.
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list
List all groups in the broker.
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group <group id>
Describe a group with given group id.
Reset Consumer Offset
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group my-first-application --reset-offsets --to-earliest --topic first_topic --dry-run
Allows dry run to reset offsets for a topic.
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group my-first-application --reset-offsets --to-earliest --topic first_topic --execute
Execute reset offsets for a topic. Reset cannot be done when consumer is running, consumer must be stopped.
Advanced Configurations
kafka-configs --bootstrap-server localhost:9092 --entity-type topics --entity-name first_topic --describe
To return a dynamic configuration of topic named first_topic
kafka-configs --bootstrap-server localhost:9092 --entity-type topics --entity-name first_topic --alter --add-config min.insync.replicas=2
Alters topic configuration to add minimum insync replicas
kafka-configs --bootstrap-server localhost:9092 --entity-type topics --entity-name first_topic --alter --delete-config min.insync.replicas=2
Alters topic configuration to delete minimum insync replicas
Last updated