Details

    • Type: New Feature New Feature
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: contrib
    • Labels:

      Description

      This issue is created to track a community-contributed Kafka Web UI.

      Here is an initial list of goals:

      • Be able to easily see which brokers are up
      • Be able to see lists of topics, connected producers, consumer groups, connected consumers
      • Be able to see, for each consumer/partition, its offset, and more importantly, # of bytes unconsumed (== largest offset for partition - current offset)
      • (Wish list) have a graphical view of the offsets
      • (Wish list) be able to clean up consumer state, such as stale claimed partitions

      List of challenges/questions:

      • Which framework? Play! for Scala?
      • Is all the data available from JMX and ZK? Hopefully, watching the files on the filesystem can be avoided....
      • How to handle large numbers of topics, partitions, consumers, etc. efficiently

        Activity

        Hide
        Jay Kreps added a comment -

        That is correct. If JMX would be a workable approach, adding JMX for producer connections should be reasonably straight-forward to do...

        Show
        Jay Kreps added a comment - That is correct. If JMX would be a workable approach, adding JMX for producer connections should be reasonably straight-forward to do...
        Hide
        Claude Mamo added a comment -

        Definitely Jay Kreps, docs are in my TODO list before releasing the first version of the project. I noticed in the JIRA description that a goal for the console is to get a list of connected producers. I checked on ZK as well as on JMX and there is no info on producers, unless I missed it. Is this possible to achieve with the current version of Kafka?

        Show
        Claude Mamo added a comment - Definitely Jay Kreps , docs are in my TODO list before releasing the first version of the project. I noticed in the JIRA description that a goal for the console is to get a list of connected producers. I checked on ZK as well as on JMX and there is no info on producers, unless I missed it. Is this possible to achieve with the current version of Kafka?
        Hide
        Jay Kreps added a comment -

        Claude Mamo Very cool. I added it to the Ecosystem page. It would be great if you could add a README and a couple screen shots to the project so people would know what they get with it.

        James Miller Let me know if you get this up on github and I'll add it to the Ecosystem page too.

        It would also be good to just ping the users list and let people know that this exists. A LOT of people have been asking for something like this so I suspect there will be a lot of interest.

        Show
        Jay Kreps added a comment - Claude Mamo Very cool. I added it to the Ecosystem page. It would be great if you could add a README and a couple screen shots to the project so people would know what they get with it. James Miller Let me know if you get this up on github and I'll add it to the Ecosystem page too. It would also be good to just ping the users list and let people know that this exists. A LOT of people have been asking for something like this so I suspect there will be a lot of interest.
        Hide
        Claude Mamo added a comment -

        Adding another one to the list: https://github.com/claudemamo/kafka-web-console.

        The project is as well built on top of Play and uses AngularJS in addition to jQuery. It's using Twitter's Zookeeper client to make non-blocking queries to Zookeeper for brokers, consumers groups, partitions, etc... I forked the high level consumer and replaced its blocking queues with callbacks so that it would be possible to implement topic feeds without being too much of a drain on resources. I'm going to bump the version no. to 1.0 soon but in the meantime please feel free to take the console for a spin and let me know your thoughts. The current list of implemented features is:

        • Topic feeds via Web Sockets
        • Which brokers are up
        • Lists of topics & no. partitions, and connected consumer groups
        • For each consumer/partition, its current offset

        Finally the console provides a JSON interface.

        Show
        Claude Mamo added a comment - Adding another one to the list: https://github.com/claudemamo/kafka-web-console . The project is as well built on top of Play and uses AngularJS in addition to jQuery. It's using Twitter's Zookeeper client to make non-blocking queries to Zookeeper for brokers, consumers groups, partitions, etc... I forked the high level consumer and replaced its blocking queues with callbacks so that it would be possible to implement topic feeds without being too much of a drain on resources. I'm going to bump the version no. to 1.0 soon but in the meantime please feel free to take the console for a spin and let me know your thoughts. The current list of implemented features is: Topic feeds via Web Sockets Which brokers are up Lists of topics & no. partitions, and connected consumer groups For each consumer/partition, its current offset Finally the console provides a JSON interface.
        Hide
        Jay Kreps added a comment -

        Hey James,

        That's awesome! Yeah I recommend github and adding it to our Ecosystem page. Maintaining outside tools with the main code base turned out to be a real drag for the people working on the tool. The apache process works better (we think) for a focused code base, github works well for a large diverse ecosystem as each person can maintain their piece without blocking code reviews or the need to release in sync with Kafka. So we aren't trying to suck in everything that integrates with Kafka into the main project.

        Show
        Jay Kreps added a comment - Hey James, That's awesome! Yeah I recommend github and adding it to our Ecosystem page. Maintaining outside tools with the main code base turned out to be a real drag for the people working on the tool. The apache process works better (we think) for a focused code base, github works well for a large diverse ecosystem as each person can maintain their piece without blocking code reviews or the need to release in sync with Kafka. So we aren't trying to suck in everything that integrates with Kafka into the main project.
        Hide
        Jun Rao added a comment -

        James,

        Thanks for your interest. Perhaps you can start with a wiki in Kafka that describes your current design and how it looks? As for the code, we can put it in a separate dir and build a separate jar. So, the additional dependency won't affect other existing jars.

        Show
        Jun Rao added a comment - James, Thanks for your interest. Perhaps you can start with a wiki in Kafka that describes your current design and how it looks? As for the code, we can put it in a separate dir and build a separate jar. So, the additional dependency won't affect other existing jars.
        Hide
        James Miller added a comment -

        Hi Folks,
        I have a admin web app that I built for this purpose, it connects to Zk and displays information about brokers, topics, and consumers.

        It really useful, especially when just getting started with Kafka, I would like to contribute it to the this project.

        It is a Play App, written in Scala and depends on Playframework, Kafka, and jQuery. Any concerns about these dependencies?

        Also, what are the preferred next steps, should I attach source or put in public github, not sure how you would like to move forward?

        Show
        James Miller added a comment - Hi Folks, I have a admin web app that I built for this purpose, it connects to Zk and displays information about brokers, topics, and consumers. It really useful, especially when just getting started with Kafka, I would like to contribute it to the this project. It is a Play App, written in Scala and depends on Playframework, Kafka, and jQuery. Any concerns about these dependencies? Also, what are the preferred next steps, should I attach source or put in public github, not sure how you would like to move forward?
        Hide
        Swapnil Ghike added a comment -

        Hey Andrea, Guy, did you make any head way on this?

        Show
        Swapnil Ghike added a comment - Hey Andrea, Guy, did you make any head way on this?
        Hide
        Jun Rao added a comment -

        Andrea, thanks for your interest. The tool can be written in java, if that's more convenient.

        Show
        Jun Rao added a comment - Andrea, thanks for your interest. The tool can be written in java, if that's more convenient.
        Hide
        Andrea Gazzarini added a comment -

        Any problem if it will be in Java? I have to explore better Kafka and I think It would be a nice opportunity....unfortunately I'm not so skilled in Scala

        Show
        Andrea Gazzarini added a comment - Any problem if it will be in Java? I have to explore better Kafka and I think It would be a nice opportunity....unfortunately I'm not so skilled in Scala
        Hide
        Samir Madhavan added a comment -

        This would be absolutely great to have. Wanted to know, how it has progressed?

        Show
        Samir Madhavan added a comment - This would be absolutely great to have. Wanted to know, how it has progressed?
        Hide
        Guy Doulberg added a comment -

        I am very happy to hear,

        I will share you with my code, as soon as I get approval for doing that from my company legal department, I don't think it should be a problem .

        Show
        Guy Doulberg added a comment - I am very happy to hear, I will share you with my code, as soon as I get approval for doing that from my company legal department, I don't think it should be a problem .
        Hide
        Neha Narkhede added a comment -

        Absolutely, it seems like a good starting point. Would you mind uploading it as a patch ?

        Show
        Neha Narkhede added a comment - Absolutely, it seems like a good starting point. Would you mind uploading it as a patch ?
        Hide
        Guy Doulberg added a comment -

        Hi
        I started building an admin web app, from which I let the users of the kafka cluster:
        1. Watch how much their consumer group is behind a certain offset,
        2. Update their topic to latest offset

        I developed it using play!, it connects the ZK and the Brokers using the RPC,

        Will interest anyone?

        Show
        Guy Doulberg added a comment - Hi I started building an admin web app, from which I let the users of the kafka cluster: 1. Watch how much their consumer group is behind a certain offset, 2. Update their topic to latest offset I developed it using play!, it connects the ZK and the Brokers using the RPC, Will interest anyone?
        Hide
        Evan Chan added a comment -

        BTW, just some thoughts. I'm thinking of not so much a web console for individual Kafka brokers, but a web console for an entire Kafka cluster, targetting mostly smaller clusters. This would grab information from ZK and from JMX (or JMX-over-HTTP) for individual brokers to put together reports like:

        • Status of different Kafka brokers in the cluster, including overall read and write rates, etc.
        • Information on consumers (at least the high level consumers registered in ZK), outstanding messages or how far behind they are, etc.
        Show
        Evan Chan added a comment - BTW, just some thoughts. I'm thinking of not so much a web console for individual Kafka brokers, but a web console for an entire Kafka cluster, targetting mostly smaller clusters. This would grab information from ZK and from JMX (or JMX-over-HTTP) for individual brokers to put together reports like: Status of different Kafka brokers in the cluster, including overall read and write rates, etc. Information on consumers (at least the high level consumers registered in ZK), outstanding messages or how far behind they are, etc.
        Hide
        Evan Chan added a comment -

        The web console would not primarily be for monitoring – for us anyways – and I agree it should not replace Ganglia etc. (Although at some shops - most notably Google - having web routes for every single service and app is a policy). I think the web console would be invaluable for debugging and status though. For looking at a snapshot of the system easily.

        Also, standard JMX just doesnt work in EC2, you don't know what ports need to be opened up.

        I think the metric Im most interested in – number of messages or MB outstanding – is not available directly from Kafka server itself anyways.

        What is the timeframe for KAFKA-203?

        Show
        Evan Chan added a comment - The web console would not primarily be for monitoring – for us anyways – and I agree it should not replace Ganglia etc. (Although at some shops - most notably Google - having web routes for every single service and app is a policy). I think the web console would be invaluable for debugging and status though. For looking at a snapshot of the system easily. Also, standard JMX just doesnt work in EC2, you don't know what ports need to be opened up. I think the metric Im most interested in – number of messages or MB outstanding – is not available directly from Kafka server itself anyways. What is the timeframe for KAFKA-203 ?
        Hide
        Evan Chan added a comment -

        The web console would not primarily be for monitoring – for us anyways – and I agree it should not replace Ganglia etc. (Although at some shops - most notably Google - having web routes for every single service and app is a policy). I think the web console would be invaluable for debugging and status though. For looking at a snapshot of the system easily.

        Also, standard JMX just doesnt work in EC2, you don't know what ports need to be opened up.

        I think the metric Im most interested in – number of messages or MB outstanding – is not available directly from Kafka server itself anyways.

        What is the timeframe for KAFKA-203?

        Show
        Evan Chan added a comment - The web console would not primarily be for monitoring – for us anyways – and I agree it should not replace Ganglia etc. (Although at some shops - most notably Google - having web routes for every single service and app is a policy). I think the web console would be invaluable for debugging and status though. For looking at a snapshot of the system easily. Also, standard JMX just doesnt work in EC2, you don't know what ports need to be opened up. I think the metric Im most interested in – number of messages or MB outstanding – is not available directly from Kafka server itself anyways. What is the timeframe for KAFKA-203 ?
        Hide
        Jay Kreps added a comment -

        My take is that, as Joe says, most serious folks already have some system they use for monitoring across their stack (ganglia or whatever). To really work operationally you can't try to replace this or add a new system. For this reason, I kind of prefer the web app to be an optional stand-alone thing since it may not be of use to everyone, though i think that complicates its design (potentially). I think the advantage of the web app is custom display of very kafka-specific things (the audit, cluster status, etc).

        Show
        Jay Kreps added a comment - My take is that, as Joe says, most serious folks already have some system they use for monitoring across their stack (ganglia or whatever). To really work operationally you can't try to replace this or add a new system. For this reason, I kind of prefer the web app to be an optional stand-alone thing since it may not be of use to everyone, though i think that complicates its design (potentially). I think the advantage of the web app is custom display of very kafka-specific things (the audit, cluster status, etc).
        Hide
        Joe Stein added a comment -

        JMX, HTTP are both fine but also hooking into Graphite, Ganglia etc make for a more ubuitious friendly system (for the ops folk) I think KAFKA-203 really address this and a web console that is going to be for monitoring only I think should just pull the JSON from the http connector of the coda hale metrics implementation IMHO

        Show
        Joe Stein added a comment - JMX, HTTP are both fine but also hooking into Graphite, Ganglia etc make for a more ubuitious friendly system (for the ops folk) I think KAFKA-203 really address this and a web console that is going to be for monitoring only I think should just pull the JSON from the http connector of the coda hale metrics implementation IMHO
        Hide
        Eric Hauser added a comment -

        I would like to see a simple monitoring console as something that is embedded within the brokers themselves (ActiveMQ does something similar). It should have minimal dependencies (jetty, servlets, and Javascript sound appropriate). The simple HTTP interface should allow for quick checking to see if:

        1) Brokers are up
        2) Topic are being written to
        3) Topics are being consumed

        While most of this data is available via JMX already, HTTP is a preferable alternative. An external application that provides comprehensive management, audit solution seems to make sense as well.

        Show
        Eric Hauser added a comment - I would like to see a simple monitoring console as something that is embedded within the brokers themselves (ActiveMQ does something similar). It should have minimal dependencies (jetty, servlets, and Javascript sound appropriate). The simple HTTP interface should allow for quick checking to see if: 1) Brokers are up 2) Topic are being written to 3) Topics are being consumed While most of this data is available via JMX already, HTTP is a preferable alternative. An external application that provides comprehensive management, audit solution seems to make sense as well.
        Hide
        Jay Kreps added a comment -

        yeah i will post a patch.

        Show
        Jay Kreps added a comment - yeah i will post a patch.
        Hide
        Evan Chan added a comment -

        Jay,

        Would it be possible to look at the source of the java app you guys have, or open source it somehow? It would be great to see how you guys are pulling out the stats, and might save lots of work in doing the web console.

        thanks.

        Show
        Evan Chan added a comment - Jay, Would it be possible to look at the source of the java app you guys have, or open source it somehow? It would be great to see how you guys are pulling out the stats, and might save lots of work in doing the web console. thanks.
        Hide
        Jay Kreps added a comment -

        I think this would be fantastic to have. We don't have such a thing at LinkedIn, we just have hooks into our monitoring system, but I think this would be better.

        We do have an application that does a data audit. This basically proves that all data that is sent is received by all interested consumers. See KAFKA-260 for a detailed description and screenshot. We had planned to fold this functionality into the open source project. Since both of these are kind of monitoring/health functionality are pretty closely related, maybe it would make sense to combine them.

        The existing app is a java app that implements runs a few jetty servlets to implement a JSON api. The UI just runs off this api and is implemented in hmtl/css/js. The charts are dygraph and it uses jquery for javascript helpers. I am not married to this setup (the original code was written by someone else and just refactored by me), so I would be okay moving it into another mvc framework if there is something much better out there as long as it isn't too complex.

        Show
        Jay Kreps added a comment - I think this would be fantastic to have. We don't have such a thing at LinkedIn, we just have hooks into our monitoring system, but I think this would be better. We do have an application that does a data audit. This basically proves that all data that is sent is received by all interested consumers. See KAFKA-260 for a detailed description and screenshot. We had planned to fold this functionality into the open source project. Since both of these are kind of monitoring/health functionality are pretty closely related, maybe it would make sense to combine them. The existing app is a java app that implements runs a few jetty servlets to implement a JSON api. The UI just runs off this api and is implemented in hmtl/css/js. The charts are dygraph and it uses jquery for javascript helpers. I am not married to this setup (the original code was written by someone else and just refactored by me), so I would be okay moving it into another mvc framework if there is something much better out there as long as it isn't too complex.

          People

          • Assignee:
            Unassigned
            Reporter:
            Evan Chan
          • Votes:
            6 Vote for this issue
            Watchers:
            26 Start watching this issue

            Dates

            • Created:
              Updated:

              Time Tracking

              Estimated:
              Original Estimate - 672h
              672h
              Remaining:
              Remaining Estimate - 672h
              672h
              Logged:
              Time Spent - Not Specified
              Not Specified

                Development