This tutorial will provide you with the instructions to add partitions to an existing Topic in Apache Kafka.

Abstract

Apache Kafka provides the concept of Partitions in a Topic. While Topic is mainly used to categorize stream of messages, Partitions enable parallel processing of a Topic stream at consumer side. In case of multiple partitions, a consumer in a group pulls the messages from one of the Topic partitions.

However, while paritiions speed up the processing at consumer side, it violates message ordering guarantees. Hence partitions should only be used when there is no requirement of processing Topic messages in the order that these were received in. Having said that messages from a particular partition will still be in order.

Ideally, we will specify number of partitions at the time of creation of Topic. E.g. below command can be executed from Kafka home directory to create a topic 'my-topic' with 2 partitions among other things -

./bin/kafka-topics.sh --create --zookeeper localhost:2181 --topic my-topic --replication-factor 1 --partitions 2

However, there may be cases where you need to add partitions to an existing Topic. We will cover how to add more partitions to a Topic, in next section.

Adding Partitions to a Topic

Apache Kafka provides us with alter command to change Topic behaviour and add/modify configurations. We will be using alter command to add more partitions to an existing Topic.

Here is the command to increase the partitions count from 2 to 3 for topic 'my-topic' -

./bin/kafka-topics.sh --alter --zookeeper localhost:2181 --topic my-topic --partitions 3

You can verify whether partitions have been increased by using describe command as follows - 

./bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-topic

# Command output
Topic:my-topic	PartitionCount:3		ReplicationFactor:1	Configs:
	Topic: my-topic	Partition: 0		Leader: 0	Replicas: 0	Isr: 0
	Topic: my-topic	Partition: 1		Leader: 0	Replicas: 0	Isr: 0
	Topic: my-topic	Partition: 2		Leader: 0	Replicas: 0	Isr: 0

Note: While Kafka allows us to add more partitions, it is NOT possible to decrease number of partitions of a Topic. In order to achieve this, you need to delete and re-create your Topic.

Thank you for reading through the tutorial. In case of any feedback/questions/concerns, you can communicate same to us through your comments and we shall get back to you as soon as possible.