Something that doesn’t get talked about much when you’re starting out with Kafka is monitoring. Most initial thoughts go into topic log timeouts and maximum message sizes.

JMX is your friend

Kafka emits a lot of JMX information, this gives us a very good indication of the internal measurements that are happening. To tap in to this gold mine of information we need to do some very basic settings (well, it’s one setting) to get this information.

Ladies and Gentlemen, start your brokers.

Let’s get this show on the road, it’s very easy.

First of all let’s startup Zookeeper.

$ $KAFKA_HOME/bin/ config/

Open another terminal window tab. Before you start your server, export the JMX_PORTenvironment setting and use a free port, I’m going to use 9999.

$ export JMX_PORT=9999

Once this is set you can start the Kafka server.

$ bin/ config/

Last thing to do is test that our JMX setup is working properly. I’m just using the plain old, but handy all the same, jconsolefrom the Java Development Kit.

$ jconsole localhost:9999

You will see the JConsole window appear, now if you click on the MBeans tab you’ll see all the Kafka JMX metrics. One you should really be keeping an eye on is kafka.server:type=ReplicaManager,name=UnderReplicatedPartitionsbut I’ll talk more on that another time.

More Kafka Posts

Topic level settings you can’t ignore.

Monitoring Consumer Offsets.

Quick Recipe for Kafka Streams in Clojure.