Listing Kafka Topics

1. Overview

In this quick tutorial, we're going to see how we can list all topics in an Apache Kafka cluster.

First, we'll set up a single-node Apache Kafka and Zookeeper cluster. Then, we'll ask that cluster about its topics.

2. Setting Up Kafka

Before listing all the topics in a Kafka cluster, let's set up a test single-node Kafka cluster in three steps:

• Downloading Kafka and Zookeeper
• Starting Zookeeper Service
• Starting Kafka Service

First, we should make sure to download the right Kafka version from the Apache site. Once the download finishes, we should extract the downloaded archive:

$ tar xvf kafka_2.13-2.6.0.tgz

Kafka is using Apache Zookeeper to manage its cluster metadata, so we need a running Zookeeper cluster.

For test purposes, we can run a single-node Zookeeper instance using the zookeeper-server-start.sh script in the bin directory:

$ cd kafka_2.13-2.6.0 # extracted directory
$ ./bin/zookeeper-server-start.sh config/zookeeper.properties

This will start a Zookeeper service listening on port 2181. After this, we can use another script to run the Kafka server:

$ ./bin/kafka-server-start.sh config/server.properties

After a while, a Kafka broker will start. Let's add a few topics to this simple cluster:

$ bin/kafka-topics.sh --create --topic users.registrations --replication-factor 1 \
  --partitions 2  --zookeeper localhost:2181
$ bin/kafka-topics.sh --create --topic users.verfications --replication-factor 1 \
  --partitions 2  --zookeeper localhost:2181

Now that everything is ready, let's see how we can list Kafka topics.

3. Listing Topics

To list all Kafka topics in a cluster, we can use the bin/kafka-topics.sh shell script bundled in the downloaded Kafka distribution. All we have to do is to pass the –list option along with the information about the cluster. For instance, we can pass the Zookeeper service address:

$ bin/kafka-topics.sh --list --zookeeper localhost:2181
users.registrations
users.verfications

As shown above, the –list option tells the kafka-topics.sh shell script to list all the topics. In this case, we have two topics to store user-related events. If there is no topic in the cluster, then the command will return silently without any result.

Also, in order to talk to the Kafka cluster, we need to pass the Zookeeper service URL using the –zookeeper option.

It's even possible to pass the Kafka cluster address directly using the –bootstrap-server option:

$ ./bin/kafka-topics.sh --bootstrap-server=localhost:9092 --list
users.registrations
users.verfications

Our single-instance Kafka cluster listens to the 9092 port, so we specified “localhost:9092” as the bootstrap server. Put simply, bootstrap servers are Kafka brokers.

If we don't pass the information necessary to talk to a Kafka cluster, the kafka-topics.sh shell script will complain with an error:

$ ./bin/kafka-topics.sh --list
Exception in thread "main" java.lang.IllegalArgumentException: Only one of --bootstrap-server or --zookeeper must be specified
        at kafka.admin.TopicCommand$TopicCommandOptions.checkArgs(TopicCommand.scala:721)
        at kafka.admin.TopicCommand$.main(TopicCommand.scala:52)
        at kafka.admin.TopicCommand.main(TopicCommand.scala)

As shown above, the shell scripts require us to pass either the –bootstrap-server or –zookeeper option.

4. Topic Details

Once we've found a list of topics, we can take a peek at the details of one specific topic. To do that, we can use the “–describe –topic <topic name>” combination of options:

$ ./bin/kafka-topics.sh --bootstrap-server=localhost:9092 --describe --topic users.registrations
Topic: users.registrations      PartitionCount: 2       ReplicationFactor: 1    Configs: segment.bytes=1073741824
        Topic: users.registrations      Partition: 0    Leader: 0       Replicas: 0     Isr: 0
        Topic: users.registrations      Partition: 1    Leader: 0       Replicas: 0     Isr: 0

These details include information about the specified topic such as the number of partitions and replicas, among others. Similar to other commands, we must pass the cluster information or Zookeeper address. Otherwise, we won't be able to talk to the cluster.

5. Conclusion

In this short tutorial, we learned how to list all topics in a Kafka cluster. Along the way, we saw how to set up a simple, single-node Kafka cluster.

Currently, Apache Kafka is using Zookeeper to manage its cluster metadata. This, however, will change shortly as part of KIP-500, as Kafka is going to have its own metadata quorum.