Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2124

Add job counters for measuring time spent in three different phases in reducers

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: 0.22.0
    • Fix Version/s: None
    • Component/s: jobtracker
    • Labels:
      None

      Description

      We currently have SLOTS_MILLIS_REDUCES which measures the total slot time of reducer.
      It will be useful if we have

      SLOTS_MILLIS_REDUCES_COPY
      SLOTS_MILLIS_REDUCES_SORT
      SLOTS_MILLIS_REDUCES_REDUCE
      

      which measures three different phases of a reducer.
      This will help us identify the bottleneck of the reducers.

      1. MAPREDUCE-2124.3.txt
        4 kB
        Scott Chen
      2. test-result.txt
        54 kB
        Scott Chen
      3. MAPREDUCE-2124-v2.txt
        4 kB
        Scott Chen
      4. MAPREDUCE-2124.txt
        2 kB
        Scott Chen

        Activity

        Scott Chen created issue -
        Scott Chen made changes -
        Field Original Value New Value
        Summary Add counters for measuring time spent in three different phases in reducers Add job counters for measuring time spent in three different phases in reducers
        Scott Chen made changes -
        Attachment MAPREDUCE-2124.txt [ 12456903 ]
        Scott Chen made changes -
        Attachment MAPREDUCE-2124-v2.txt [ 12458431 ]
        Scott Chen made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Scott Chen made changes -
        Attachment test-result.txt [ 12458569 ]
        Nigel Daley made changes -
        Fix Version/s 0.22.0 [ 12314184 ]
        Tom White made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Scott Chen made changes -
        Attachment MAPREDUCE-2124.3.txt [ 12472483 ]
        Scott Chen made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Arun C Murthy made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]

          People

          • Assignee:
            Scott Chen
            Reporter:
            Scott Chen
          • Votes:
            0 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

            • Created:
              Updated:

              Development