Continuing on from my post about installing kafka on Ec2, Kafka doesn't have (yet) any really good UI's for monitoring or maintaining your cluster, so I decided to have a hunt around for some open source goodies. I checked out a couple and landed on evaluating Yahoo's Kafka Manager, I chose this over kafka-web-console only because it was made by Yahoo and I'm making the assumption that it's well maintained (I could pay for this later on) and considering the last commit to web console was 9 months ago I think I made the right choice.
Update - As I started playing around with Kafka Manager it became apparent to me that this tool was only intended to be used as a UI for administration tasks, not for monitoring health (such as lag) of existing production topics. So soon enough I will also be looking at web-console. Previously we had built our own monitoring solution with bastardized angular and bootstrap, but it was painfully slow and took many developer hours to maintain it (the slowness could be down to the JMX reporters and thousands of topics). Step 1 - Download and extract the package If you followed my previous post, I set up a node for zookeeper and I'm going to install Kafka manager on here also. So, you will need to first SSH onto the Zookeeper Node and download the package from GitHub, unzip and then rename the dir. _ wget https://github.com/yahoo/kafka-manager/archive/master.zip
Step 2 - Install sbt
If you're like me and you don't have scala installed you will need to grab it (it doesn't seem to be as easy as "apt-get install sbt"). Below are the commands to run to install scala manually. wget -q apt.typesafe.com/repo-deb-build-0002.deb
Step 3 - Build your kafka-manager scripts
Cd into your new kafka-manager directory then run the build command. cd kafka-manager
As you can see there are a few warnings around existing library conflicts, but all is good. It has generated a zip file, see the line "your package is ready in ...".
Step 4 - Copy that Zip file to a suitable location and unzip sudo mv target/universal/kafka-manager-1.1.zip ~/
Step 5 - Some weirdness and a hacky work around..
Yes you would expect a conf file to point the manager to a Zookeeper host(s) of which there is a such a file, but for me at least it didn't work, I tried the dns, the ip and since I'm running this app on the same host as the Zookeeper node I also tried localhost...none of them worked. In case you have better luck than me below are the entries that need to be made: The conf file is located at: "conf/application.conf" and alter the zkhosts entry Below is the command I ran to over-ride the conf and it worked just fine. bin/kafka-manager -Dkafka-manager.zkhosts="localhost:2181"
Once that's started, you should be able to open it up in a browser using the standard port 9000.
Step 6: Initial Setup and adding a cluster
Go to Cluster -> Setup a new cluster.
For this evaluation I'm running my Kafka Manager from my zookeeper host, I only have one host for my tests but if you have multiple hosts, just add them in as "host1:2181,host2:2181" etc. I'm using Kafka 0.8.2.x. Hit the save button and some magic should happen.
You can now see it's picked up my cluster and the existing 2 topics I created. If you have an existing topic in your cluster, you can click on it and explore from there.
Creating A Topic through Kafka Manager.
In the top menu, click on topic and then create. You are presented with a bunch of parameters that you might be accustomed to from using the stand CLI. Once done there click on topic view.
Topic View
We created our topic, now we can see it in all it's glory! If you're old hat to Kafka... you'll be pleased to know that the topic "delete" function.. works!.
Brokers
Clicking on the broker menu item, provides you with a summary of your brokers.
Clicking on one of the Brokers will provide you with a summary of the topics and partitions.
Re-assigning Partitions
This bit was a little buggy for me, I created a test4 topic and deleted it (out of this example) and it's still showing up as an option to reassign. When I selected test4 it then presented an error that it didn't exist.
Conclusion
This is a very very simple application, for some reason I assumed because it was made by Yahoo it would provide more features. When you should use it: I think it would be useful on an enterprise level where system engineers don't have to always go through the CLI to complete their work, where engineers complete 5,000 tasks a day and this is just one of them. It enables you to digest the information pretty quickly. Cons I guess I expected more? Maybe I'm being greedy? I would love to see some metrics of cluster health around topic consumer lag etc. I'm assuming that since Yahoo is large scale they use standardized health metrics systems across their applications and they have something already feeding into that. Who knows? Kafka is pretty young in the terms of things and we could see this project grow. (Also it screams bootstrap).. confirmed src it's bootstrapped. Keep posted as I dive into web-console.
13 Comments
Michael U.
9/10/2015 01:20:50 am
Thank you for this guide!
Reply
Kamal Singh Negi
10/28/2015 11:10:52 pm
My kafka and zookeeper are dockerized. Can i monitor them using yahoo-kafka-manager?
Reply
Ed
10/28/2015 11:48:19 pm
You can as long as you have configured your docker container to be accessible from outside the container (in the configuration)
Reply
Ramone Hamilton
11/22/2015 06:20:24 pm
Just had to drop in and say that this guide was spot on, I was stuck on not being able to get Kafka Manager to recognize my zookeeper instance and your comment about running kafa-manager with the zookepper config file was correct.
Reply
Suhas
1/8/2016 10:49:53 am
Mate,
Reply
Thirumal
6/16/2016 11:56:59 am
Hi, thanks for the post, it was very helpful. I am using kafka version 0.10. When I start kafka manager, cluster summary shows correct number of topics, but doesn't show the number of brokers. Is this a known issue or am I missing something ?
Reply
Bhavesh
9/15/2016 11:53:41 pm
wget http://repo.scala-sbt.org/scalasbt/sbt-native-packages/org/scala-sbt/sbt/0.13.1/sbt.deb
Reply
Philippe Back
1/11/2018 01:49:15 am
Do you have any steps? Did you use Software Collections Scala?
Reply
Shriyog
11/8/2016 10:09:16 pm
Hey! Thanks a lot for such detailed guide. Got my kafka-manager up and running fine.
Reply
Sanjeev
2/26/2017 10:42:01 pm
Seems this project needs active contributors, to move to next level.. Hope it becomes Apache project.
Reply
Mike M
9/25/2017 05:24:33 pm
Does this Kafka Manager has https ?
Reply
Leave a Reply. |
AuthorNew Zealand big data nerd, facial hair sculptor and classic car fanatic. Owner of needles.io, freelance big data consultant, ex Activision. Archives
April 2016
Categories |