Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-901

Move Framework Counters into a TaskMetric structure

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.23.0
    • Component/s: task
    • Labels:
      None
    • Release Note:
      Efficient implementation of MapReduce framework counters.

      Description

      I think we should move all of the Counters that the framework updates into a single class called TaskMetrics. TaskMetrics would have specific fields for each of the metrics like input records, input bytes, output records, etc.

      It would both reduce the serialized size of the heartbeats (by shrinking the Counters down to just the user's counters) and decrease the latency for updates to the JobTracker (since Counters are sent at most 1/minute instead of 1/heartbeat).

        Attachments

        1. mr-901-trunk-v1.patch
          104 kB
          Luke Lu
        2. MAPREDUCE-901.patch_2
          292 kB
          Arun C Murthy
        3. MAPREDUCE-901.patch
          24 kB
          Arun C Murthy
        4. MAPREDUCE-901.patch
          60 kB
          Arun C Murthy
        5. MAPREDUCE-901.patch
          156 kB
          Arun C Murthy
        6. FrameworkCounterGroup.java
          4 kB
          Luke Lu
        7. 901_1.patch
          49 kB
          Devaraj Das
        8. 901_1.patch
          54 kB
          Devaraj Das

          Issue Links

            Activity

              People

              • Assignee:
                vicaya Luke Lu
                Reporter:
                owen.omalley Owen O'Malley
              • Votes:
                1 Vote for this issue
                Watchers:
                20 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: