Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-901

Move Framework Counters into a TaskMetric structure

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.21.0
    • 0.23.0
    • task
    • None
    • Efficient implementation of MapReduce framework counters.

    Description

      I think we should move all of the Counters that the framework updates into a single class called TaskMetrics. TaskMetrics would have specific fields for each of the metrics like input records, input bytes, output records, etc.

      It would both reduce the serialized size of the heartbeats (by shrinking the Counters down to just the user's counters) and decrease the latency for updates to the JobTracker (since Counters are sent at most 1/minute instead of 1/heartbeat).

      Attachments

        1. 901_1.patch
          49 kB
          Devaraj Das
        2. 901_1.patch
          54 kB
          Devaraj Das
        3. MAPREDUCE-901.patch
          24 kB
          Arun Murthy
        4. MAPREDUCE-901.patch
          60 kB
          Arun Murthy
        5. FrameworkCounterGroup.java
          4 kB
          Luke Lu
        6. mr-901-trunk-v1.patch
          104 kB
          Luke Lu
        7. MAPREDUCE-901.patch
          156 kB
          Arun Murthy
        8. MAPREDUCE-901.patch_2
          292 kB
          Arun Murthy

        Issue Links

          blocks

          New Feature - A new feature of the product, which has yet to be developed. MAPREDUCE-2102 Collect iops per task

          • Major - Major loss of function.
          • Open

          New Feature - A new feature of the product, which has yet to be developed. MAPREDUCE-220 Collecting cpu and memory usage for MapReduce tasks

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-718 Support for per-phase speculative execution

          • Major - Major loss of function.
          • Open
          is blocked by

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-917 Remove getInputCounter and getOutputCounter from Contexts

          • Blocker - Blocks development and/or testing work, production could not run
          • Closed
          is related to

          New Feature - A new feature of the product, which has yet to be developed. MAPREDUCE-1304 Add counters for task time spent in GC

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-2835 Make per-job counter limits configurable

          • Major - Major loss of function.
          • Closed
          is required by

          New Feature - A new feature of the product, which has yet to be developed. MAPREDUCE-2037 Capturing interim progress times, CPU usage, and memory usage, when tasks reach certain progress thresholds

          • Major - Major loss of function.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-1500 Improve the way Counters are managed by JobTracker

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-1173 Documenting MapReduce metrics

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-2025 Improve job counter wire protocol with incremental schema and delta values

          • Major - Major loss of function.
          • Open

          Activity

            People

              vicaya Luke Lu
              omalley Owen O'Malley
              Votes:
              1 Vote for this issue
              Watchers:
              20 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: