Uploaded image for project: 'Chukwa'
  1. Chukwa
  2. CHUKWA-441

Real time hadoop activity monitoring capabilities via Client Trace Logs

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.4.0
    • Labels:
      None
    • Environment:

      Redhat EL 5.1, Java 6

      Description

      Many moons ago, in one of the HUG talk for Chukwa, one user ask if it is possible to create a splitter interface for Chukwa Collector for monitoring data at near real time. The answer was possible, but there was no code to show this capabilities. Over the Xmas holidays, I decided to take on this task and write something simple to show how this is done.

      The idea is to reuse existing Chukwa Collector with PipelineStageWriter coupled with SocketTeeWriter for data stream split. By adding a SocketDataLoader inside HICC to act as data stream subscriber and extract client trace information to render on the browser. Hence, the additional pipeline looks like this:

      Adaptor > Agent -> Collector -> SocketTeeWriter < Socket Data Loader <- HICC

      1. Picture 1.png
        265 kB
        Eric Yang
      2. CHUKWA-441-1.patch
        0.6 kB
        Eric Yang
      3. CHUKWA-441.patch
        72 kB
        Eric Yang

        Activity

        Hide
        hudson Hudson added a comment -

        Integrated in Chukwa-trunk #247 (See http://hudson.zones.apache.org/hudson/job/Chukwa-trunk/247/)
        . Fixed compile error.

        Show
        hudson Hudson added a comment - Integrated in Chukwa-trunk #247 (See http://hudson.zones.apache.org/hudson/job/Chukwa-trunk/247/ ) . Fixed compile error.
        Hide
        eyang Eric Yang added a comment -

        Resolved compile error.

        Show
        eyang Eric Yang added a comment - Resolved compile error.
        Hide
        eyang Eric Yang added a comment -

        Clean up compile error.

        Show
        eyang Eric Yang added a comment - Clean up compile error.
        Hide
        eyang Eric Yang added a comment -

        Remove unused EventBean.

        Show
        eyang Eric Yang added a comment - Remove unused EventBean.
        Hide
        eyang Eric Yang added a comment -

        etting this error message trying to compile trunk (r894877):

        [javac] /home/jiaqi/code/chukwa/latest3/trunk/src/java/org/apache/hadoop/chukwa/rest/resource/ClientTrace.java:39:
        cannot find symbol
        [javac] symbol : class EventBean
        [javac] location: package org.apache.hadoop.chukwa.rest.bean
        [javac] import org.apache.hadoop.chukwa.rest.bean.EventBean;

        Unused import statement, should get clean up.

        Show
        eyang Eric Yang added a comment - etting this error message trying to compile trunk (r894877): [javac] /home/jiaqi/code/chukwa/latest3/trunk/src/java/org/apache/hadoop/chukwa/rest/resource/ClientTrace.java:39: cannot find symbol [javac] symbol : class EventBean [javac] location: package org.apache.hadoop.chukwa.rest.bean [javac] import org.apache.hadoop.chukwa.rest.bean.EventBean; Unused import statement, should get clean up.
        Hide
        eyang Eric Yang added a comment -

        I just committed this, thanks Ari.

        Show
        eyang Eric Yang added a comment - I just committed this, thanks Ari.
        Hide
        asrabkin Ari Rabkin added a comment -

        +1

        Show
        asrabkin Ari Rabkin added a comment - +1
        Hide
        eyang Eric Yang added a comment -

        Real time hadoop activity monitor. This feature only works for single collector, to make this work on large scale cluster, the data needs to be channeled to a single collector.

        Show
        eyang Eric Yang added a comment - Real time hadoop activity monitor. This feature only works for single collector, to make this work on large scale cluster, the data needs to be channeled to a single collector.
        Hide
        eyang Eric Yang added a comment -

        Screenshot of the activity monitor.

        Show
        eyang Eric Yang added a comment - Screenshot of the activity monitor.
        Hide
        eyang Eric Yang added a comment -
        • Implemented Socket Data Loader, ClientTrace REST API, and Hadoop Activity widget.
        • Test cases included.
        Show
        eyang Eric Yang added a comment - Implemented Socket Data Loader, ClientTrace REST API, and Hadoop Activity widget. Test cases included.

          People

          • Assignee:
            eyang Eric Yang
            Reporter:
            eyang Eric Yang
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development