Apache kafka in rehat 7 -part 2

Lets see how to set up kafka with multiple brokers and single partition topic with a replication factor 2

Setting up multiple brokers cluster in a single machine is relatively simple. We basically have to create separate server.properties file for each broker and edit the broker propeties to make sure that it is running on different ports whereas in production environment usually all the brokers will run on different machine in the same port

The broker id is a unique identifier of each kafka broker. Then edit the port to 9093 and the log directory

To create a topic with replication factor 2

bin/kafka-topics.sh --create --topic my_topic_replica --zookeeper localhost:2181  --replication-factor 2 --partitions 1

bin/kafka-topics.sh --describe --topic my_topic_replica --zookeeper localhost:2181 

Output

Topic:my_topic_replica PartitionCount:1 ReplicationFactor:2 Configs:
Topic: my_topic_replica Partition: 0 Leader: 0 Replicas: 0,1 Isr: 0,1

If the topic is created with mutiple partition you will see multiple lines here.

Leader host for this topic is 0

In sync replicas is 1, since the number of replicas is equal to the Isr we can say that the partition is in healthy state.

When one of the broker is down, where you run the describe command the output will be the following. Notice that the number of Isr is not equal to the number of partition which means the quoram is not in healthy state. If there is another server available zookeeper will add this to the quoram automatically.

Topic:my_topic_replica PartitionCount:1 ReplicationFactor:2 Configs:
Topic: my_topic_replica Partition: 0 Leader: 1 Replicas: 0,1 Isr: 1

To test the producer and consumer

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic my_topic_replica

bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic my_topic_replica --from-beginning

Command to list the number of brokers registered to zookeper

bin/zookeeper-shell.sh localhost:2181 <<< "ls /brokers/ids"