Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4435

Expose JobTracker metrics for number of reducers in shuffle vs. sort vs. reduce phase

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: jobtracker, tasktracker
    • Labels:
      None

      Description

      We'd like to be able to show our Cloudera Manager users some more detailed metrics about the number of reducers running at any given time--specifically, how many reducers are running in each of the three possible phases (shuffle, sort, and reduce). This would require the addition of some new overridable methods to the JobTrackerInstrumentation API, plus a little bit of code to actually call them from the JobTracker class. The necessary information seems to already be available in the TaskStatus object. The attached patch (which I've tested on hadoop-common/branch-1.0) shows one way to do it.

        Attachments

        1. mapreduce-4435-branch-1.0_with_test.patch
          7 kB
          Eirik Bakke
        2. mapreduce.patch
          4 kB
          Eirik Bakke

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              eirik.bakke@cloudera.com Eirik Bakke
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated: