Uploaded image for project: 'Chukwa'
  1. Chukwa
  2. CHUKWA-441

Real time hadoop activity monitoring capabilities via Client Trace Logs

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.4.0
    • Labels:
      None
    • Environment:

      Redhat EL 5.1, Java 6

      Description

      Many moons ago, in one of the HUG talk for Chukwa, one user ask if it is possible to create a splitter interface for Chukwa Collector for monitoring data at near real time. The answer was possible, but there was no code to show this capabilities. Over the Xmas holidays, I decided to take on this task and write something simple to show how this is done.

      The idea is to reuse existing Chukwa Collector with PipelineStageWriter coupled with SocketTeeWriter for data stream split. By adding a SocketDataLoader inside HICC to act as data stream subscriber and extract client trace information to render on the browser. Hence, the additional pipeline looks like this:

      Adaptor > Agent -> Collector -> SocketTeeWriter < Socket Data Loader <- HICC

        Attachments

        1. CHUKWA-441-1.patch
          0.6 kB
          Eric Yang
        2. Picture 1.png
          265 kB
          Eric Yang
        3. CHUKWA-441.patch
          72 kB
          Eric Yang

          Activity

            People

            • Assignee:
              eyang Eric Yang
              Reporter:
              eyang Eric Yang
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: