Uploaded image for project: 'Chukwa (retired)'
  1. Chukwa (retired)
  2. CHUKWA-441

Real time hadoop activity monitoring capabilities via Client Trace Logs

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 0.4.0
    • None
    • Redhat EL 5.1, Java 6

    Description

      Many moons ago, in one of the HUG talk for Chukwa, one user ask if it is possible to create a splitter interface for Chukwa Collector for monitoring data at near real time. The answer was possible, but there was no code to show this capabilities. Over the Xmas holidays, I decided to take on this task and write something simple to show how this is done.

      The idea is to reuse existing Chukwa Collector with PipelineStageWriter coupled with SocketTeeWriter for data stream split. By adding a SocketDataLoader inside HICC to act as data stream subscriber and extract client trace information to render on the browser. Hence, the additional pipeline looks like this:

      Adaptor > Agent -> Collector -> SocketTeeWriter < Socket Data Loader <- HICC

      Attachments

        1. Picture 1.png
          265 kB
          Eric Yang
        2. CHUKWA-441-1.patch
          0.6 kB
          Eric Yang
        3. CHUKWA-441.patch
          72 kB
          Eric Yang

        Activity

          People

            eyang Eric Yang
            eyang Eric Yang
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: