Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-901

Move Framework Counters into a TaskMetric structure

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.21.0
    • 0.23.0
    • task
    • None
    • Efficient implementation of MapReduce framework counters.

    Description

      I think we should move all of the Counters that the framework updates into a single class called TaskMetrics. TaskMetrics would have specific fields for each of the metrics like input records, input bytes, output records, etc.

      It would both reduce the serialized size of the heartbeats (by shrinking the Counters down to just the user's counters) and decrease the latency for updates to the JobTracker (since Counters are sent at most 1/minute instead of 1/heartbeat).

      Attachments

        1. 901_1.patch
          54 kB
          Devaraj Das
        2. 901_1.patch
          49 kB
          Devaraj Das
        3. FrameworkCounterGroup.java
          4 kB
          Luke Lu
        4. MAPREDUCE-901.patch
          156 kB
          Arun Murthy
        5. MAPREDUCE-901.patch
          60 kB
          Arun Murthy
        6. MAPREDUCE-901.patch
          24 kB
          Arun Murthy
        7. MAPREDUCE-901.patch_2
          292 kB
          Arun Murthy
        8. mr-901-trunk-v1.patch
          104 kB
          Luke Lu

        Issue Links

          Activity

            People

              vicaya Luke Lu
              omalley Owen O'Malley
              Votes:
              1 Vote for this issue
              Watchers:
              20 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: