Details

    • Type: Sub-task Sub-task
    • Status: Open
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 1.8.0
    • Component/s: monitor
    • Labels:

      Description

      The monitor is awesome. It should die.

      I'm going to move other monitor tickets under this one (if I can), and create some requirement tickets.

      We would be better off putting our weight behind an existing monitoring program which can scale, if one exists.

      Hopefully we can combine tracing efforts and have a nicer distributed trace-based tool, too.

      For display functionality, lots of possibilities: Graphite, Cubism.js, D3.js (really, any number of really slick Javascript graphing libraries). For log collection, any number of distributed log management services out there too can serve as inspiration for functionality: statsd, logstash, cacti/rrdtool.

      Currently all of Accumulo monitoring information is exposed via JMX; a nice balance could be found leveraging the existing monitoring capabilities with JMXTrans (or equivalent) and applying a new GUI.

      Familiarity with Java and JMX would be ideal.

        Issue Links

          Activity

          Eric Newton created issue -
          Josh Elser made changes -
          Field Original Value New Value
          Labels gsoc2013
          Josh Elser made changes -
          Description The monitor is awesome. It should die.

          I'm going to move other monitor tickets under this one (if I can), and create some requirement tickets.

          We would be better off putting our weight behind an existing monitoring program which can scale, if one exists.

          Hopefully we can combine tracing efforts and have a nicer distributed trace-based tool, too.

          The monitor is awesome. It should die.

          I'm going to move other monitor tickets under this one (if I can), and create some requirement tickets.

          We would be better off putting our weight behind an existing monitoring program which can scale, if one exists.

          Hopefully we can combine tracing efforts and have a nicer distributed trace-based tool, too.

          For display functionality, lots of possibilities: Graphite, Cubism.js, D3.js (really, any number of really slick Javascript graphing libraries). For log collection, any number of distributed log management services out there too can serve as inspiration for functionality: statsd, logstash, cacti/rrdtool.

          Currently all of Accumulo monitoring information is exposed via JMX; a nice balance could be found leveraging the existing monitoring capabilities with JMXTrans (or equivalent) and applying a new GUI.

          Familiarity with Java and JMX would be ideal.
          Josh Elser made changes -
          Labels gsoc2013 gsoc2013 mentor
          Hide
          Andres Danter added a comment -

          Hi Eric,

          I'm hoping to get involved in the Summer of Code event and I would like to help enhance Accumulo. I noticed that several of the issues flagged as SoC ideas mostly revolve around shortcomings of Accumulo's monitor application. I think that the introduction of a new monitor to Accumulo is the way to go, and it would be a perfect project for SoC. I do not have an existing monitoring framework in mind for this, but that research on my part can be performed before I submit my proposal.

          Let me know what I need to do to start discussing my ideas and possible directions for this project with the Accumulo community.

          Thank you,

          Andres

          Show
          Andres Danter added a comment - Hi Eric, I'm hoping to get involved in the Summer of Code event and I would like to help enhance Accumulo. I noticed that several of the issues flagged as SoC ideas mostly revolve around shortcomings of Accumulo's monitor application. I think that the introduction of a new monitor to Accumulo is the way to go, and it would be a perfect project for SoC. I do not have an existing monitoring framework in mind for this, but that research on my part can be performed before I submit my proposal. Let me know what I need to do to start discussing my ideas and possible directions for this project with the Accumulo community. Thank you, Andres
          Eric Newton made changes -
          Assignee Eric Newton [ ecn ]
          Hide
          John Vines added a comment -

          This is a lofty goal we continuously have. But there are bigger fish so it keeps getting pushed (BTW, pushing it to 1.7). I'm wondering if we should delist a fix version for it.

          Show
          John Vines added a comment - This is a lofty goal we continuously have. But there are bigger fish so it keeps getting pushed (BTW, pushing it to 1.7). I'm wondering if we should delist a fix version for it.
          John Vines made changes -
          Fix Version/s 1.7.0 [ 12324607 ]
          Fix Version/s 1.6.0 [ 12322468 ]
          David Medinets made changes -
          Priority Major [ 3 ] Minor [ 4 ]
          Hide
          Josh Elser added a comment -

          The more I've been thinking about this, the more I'm leaning towards trying to migrate the "monitor" into some REST service and rehash the GUI to use that service instead of making direct calls to Accumulo. This would provide an interface to monitoring consumers that isn't reliant on JMX (or some process that can convert the JMX for you like jmxtrans). It would also make it infinitely easier to write tests to ensure that we're returning correct metrics. Something that packages into a WAR would also be nice as it could be deployed into an existing application server (not sure if we had a ticket specifically for that elsewhere).

          Lots of alternatives for writing a webservice too: dropwizard, jersey, jax-rs initially come to mind.

          Show
          Josh Elser added a comment - The more I've been thinking about this, the more I'm leaning towards trying to migrate the "monitor" into some REST service and rehash the GUI to use that service instead of making direct calls to Accumulo. This would provide an interface to monitoring consumers that isn't reliant on JMX (or some process that can convert the JMX for you like jmxtrans). It would also make it infinitely easier to write tests to ensure that we're returning correct metrics. Something that packages into a WAR would also be nice as it could be deployed into an existing application server (not sure if we had a ticket specifically for that elsewhere). Lots of alternatives for writing a webservice too: dropwizard, jersey, jax-rs initially come to mind.
          Hide
          David Medinets added a comment -

          Is there any way to quantify the number of hours this kind of change might take? If the change were broken into phases, what might those phases look like?

          Show
          David Medinets added a comment - Is there any way to quantify the number of hours this kind of change might take? If the change were broken into phases, what might those phases look like?
          David Medinets made changes -
          Summary integrate with a scalable monitoring tool Integrate with a scalable monitoring tool
          Hide
          Josh Elser added a comment -

          re: number of hours, it's obviously going to be relative to the amount of experience with the tool being used. I could see the work (that I proposed) being broken into the following:

          1. Identify metrics/API that we want to create from existing monitor (potentially what extra we also want to add)
          2. Stub out interfaces using library of choice.
          3. Begin writing implementations + tests for each interface

          I also forgot to mention that, with a good REST API, we could implement some version notion to provide stability as to the API we expose.

          Show
          Josh Elser added a comment - re: number of hours, it's obviously going to be relative to the amount of experience with the tool being used. I could see the work (that I proposed) being broken into the following: 1. Identify metrics/API that we want to create from existing monitor (potentially what extra we also want to add) 2. Stub out interfaces using library of choice. 3. Begin writing implementations + tests for each interface I also forgot to mention that, with a good REST API, we could implement some version notion to provide stability as to the API we expose.
          Josh Elser made changes -
          Link This issue relates to ACCUMULO-1926 [ ACCUMULO-1926 ]
          Josh Elser made changes -
          Link This issue relates to ACCUMULO-302 [ ACCUMULO-302 ]
          Josh Elser made changes -
          Parent ACCUMULO-3034 [ 12731106 ]
          Issue Type Improvement [ 4 ] Sub-task [ 7 ]
          Josh Elser made changes -
          Fix Version/s 1.8.0 [ 12329879 ]
          Fix Version/s 1.7.0 [ 12324607 ]
          Hide
          Dave Marion added a comment - - edited

          Personally I would like to see some API where I can provide an implementation to get the metrics off of each tserver and into some type of external monitoring tool. I'm thinking of something like the StatsD protocol.

          Show
          Dave Marion added a comment - - edited Personally I would like to see some API where I can provide an implementation to get the metrics off of each tserver and into some type of external monitoring tool. I'm thinking of something like the StatsD protocol.
          Hide
          Josh Elser added a comment -

          Personally I would like to see some API where I can provide an implementation to get the metrics off of each tserver and into some type of external monitoring tool

          If you haven't noticed yet, the Hadoop Metrics2 support in 1.7.0 (I think...) does allow you to register custom sinks (aka a collector) that all of the existing metrics will be sent to automatically. For example, this is how Ambari gets metrics and draws pretty graphs about what Accumulo is doing.

          Show
          Josh Elser added a comment - Personally I would like to see some API where I can provide an implementation to get the metrics off of each tserver and into some type of external monitoring tool If you haven't noticed yet, the Hadoop Metrics2 support in 1.7.0 (I think...) does allow you to register custom sinks (aka a collector) that all of the existing metrics will be sent to automatically. For example, this is how Ambari gets metrics and draws pretty graphs about what Accumulo is doing.
          Hide
          Dave Marion added a comment -

          Thanks. I did see the ticket for the Metrics2 integration, but have not looked yet at the implementation. Do you know if it is just exposing the metrics collected by the JMX MBeans, or a different set of metrics?

          Show
          Dave Marion added a comment - Thanks. I did see the ticket for the Metrics2 integration, but have not looked yet at the implementation. Do you know if it is just exposing the metrics collected by the JMX MBeans, or a different set of metrics?
          Hide
          Josh Elser added a comment -

          Do you know if it is just exposing the metrics collected by the JMX MBeans, or a different set of metrics?

          Yeah, same metrics we have exposed now. No new metrics were added, just the ability to use metrics2.

          Show
          Josh Elser added a comment - Do you know if it is just exposing the metrics collected by the JMX MBeans, or a different set of metrics? Yeah, same metrics we have exposed now. No new metrics were added, just the ability to use metrics2.
          Hide
          Dave Marion added a comment -

          Cool, thx.

          Show
          Dave Marion added a comment - Cool, thx.

            People

            • Assignee:
              Unassigned
              Reporter:
              Eric Newton
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:

                Development