Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4435

Expose JobTracker metrics for number of reducers in shuffle vs. sort vs. reduce phase

Add voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • jobtracker, tasktracker
    • None

    Description

      We'd like to be able to show our Cloudera Manager users some more detailed metrics about the number of reducers running at any given time--specifically, how many reducers are running in each of the three possible phases (shuffle, sort, and reduce). This would require the addition of some new overridable methods to the JobTrackerInstrumentation API, plus a little bit of code to actually call them from the JobTracker class. The necessary information seems to already be available in the TaskStatus object. The attached patch (which I've tested on hadoop-common/branch-1.0) shows one way to do it.

      Attachments

        1. mapreduce-4435-branch-1.0_with_test.patch
          7 kB
          Eirik Bakke
        2. mapreduce.patch
          4 kB
          Eirik Bakke

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            eirik.bakke@cloudera.com Eirik Bakke

            Dates

              Created:
              Updated:

              Slack

                Issue deployment