Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-901

Move Framework Counters into a TaskMetric structure

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.23.0
    • Component/s: task
    • Labels:
      None
    • Release Note:
      Efficient implementation of MapReduce framework counters.

      Description

      I think we should move all of the Counters that the framework updates into a single class called TaskMetrics. TaskMetrics would have specific fields for each of the metrics like input records, input bytes, output records, etc.

      It would both reduce the serialized size of the heartbeats (by shrinking the Counters down to just the user's counters) and decrease the latency for updates to the JobTracker (since Counters are sent at most 1/minute instead of 1/heartbeat).

      1. MAPREDUCE-901.patch_2
        292 kB
        Arun C Murthy
      2. MAPREDUCE-901.patch
        156 kB
        Arun C Murthy
      3. mr-901-trunk-v1.patch
        104 kB
        Luke Lu
      4. FrameworkCounterGroup.java
        4 kB
        Luke Lu
      5. MAPREDUCE-901.patch
        60 kB
        Arun C Murthy
      6. MAPREDUCE-901.patch
        24 kB
        Arun C Murthy
      7. 901_1.patch
        54 kB
        Devaraj Das
      8. 901_1.patch
        49 kB
        Devaraj Das

        Issue Links

          Activity

            People

            • Assignee:
              Luke Lu
              Reporter:
              Owen O'Malley
            • Votes:
              1 Vote for this issue
              Watchers:
              20 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development