Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-901

Move Framework Counters into a TaskMetric structure

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.23.0
    • Component/s: task
    • Labels:
      None
    • Release Note:
      Efficient implementation of MapReduce framework counters.

      Description

      I think we should move all of the Counters that the framework updates into a single class called TaskMetrics. TaskMetrics would have specific fields for each of the metrics like input records, input bytes, output records, etc.

      It would both reduce the serialized size of the heartbeats (by shrinking the Counters down to just the user's counters) and decrease the latency for updates to the JobTracker (since Counters are sent at most 1/minute instead of 1/heartbeat).

      1. MAPREDUCE-901.patch_2
        292 kB
        Arun C Murthy
      2. MAPREDUCE-901.patch
        156 kB
        Arun C Murthy
      3. mr-901-trunk-v1.patch
        104 kB
        Luke Lu
      4. FrameworkCounterGroup.java
        4 kB
        Luke Lu
      5. MAPREDUCE-901.patch
        60 kB
        Arun C Murthy
      6. MAPREDUCE-901.patch
        24 kB
        Arun C Murthy
      7. 901_1.patch
        54 kB
        Devaraj Das
      8. 901_1.patch
        49 kB
        Devaraj Das

        Issue Links

          Activity

          Owen O'Malley created issue -
          Owen O'Malley made changes -
          Field Original Value New Value
          Link This issue blocks MAPREDUCE-717 [ MAPREDUCE-717 ]
          Owen O'Malley made changes -
          Link This issue blocks MAPREDUCE-717 [ MAPREDUCE-717 ]
          Owen O'Malley made changes -
          Link This issue blocks MAPREDUCE-718 [ MAPREDUCE-718 ]
          Arun C Murthy made changes -
          Link This issue blocks MAPREDUCE-220 [ MAPREDUCE-220 ]
          Devaraj Das made changes -
          Attachment 901_1.patch [ 12417510 ]
          Devaraj Das made changes -
          Attachment 901_1.patch [ 12417555 ]
          Arun C Murthy made changes -
          Link This issue is blocked by MAPREDUCE-917 [ MAPREDUCE-917 ]
          Arun C Murthy made changes -
          Assignee Devaraj Das [ devaraj ] Arun C Murthy [ acmurthy ]
          Arun C Murthy made changes -
          Attachment MAPREDUCE-901.patch [ 12418454 ]
          Arun C Murthy made changes -
          Attachment MAPREDUCE-901.patch [ 12418610 ]
          Hong Tang made changes -
          Link This issue relates to MAPREDUCE-1173 [ MAPREDUCE-1173 ]
          Arun C Murthy made changes -
          Link This issue is related to MAPREDUCE-1304 [ MAPREDUCE-1304 ]
          Jeff Hammerbacher made changes -
          Link This issue relates to MAPREDUCE-1500 [ MAPREDUCE-1500 ]
          Tom White made changes -
          Fix Version/s 0.21.0 [ 12314045 ]
          Luke Lu made changes -
          Assignee Arun C Murthy [ acmurthy ] Luke Lu [ vicaya ]
          Jeff Hammerbacher made changes -
          Link This issue relates to MAPREDUCE-2025 [ MAPREDUCE-2025 ]
          Luke Lu made changes -
          Attachment FrameworkCounterGroup.java [ 12455883 ]
          Luke Lu made changes -
          Attachment mr-901-trunk-v1.patch [ 12456155 ]
          Luke Lu made changes -
          Link This issue blocks MAPREDUCE-2102 [ MAPREDUCE-2102 ]
          Dick King made changes -
          Link This issue is required by MAPREDUCE-2037 [ MAPREDUCE-2037 ]
          Arun C Murthy made changes -
          Attachment MAPREDUCE-901.patch [ 12490192 ]
          Arun C Murthy made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Fix Version/s 0.23.0 [ 12315570 ]
          Tom White made changes -
          Link This issue is related to MAPREDUCE-2835 [ MAPREDUCE-2835 ]
          Arun C Murthy made changes -
          Attachment MAPREDUCE-901.patch_2 [ 12490298 ]
          Arun C Murthy made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Arun C Murthy made changes -
          Release Note Efficient implementation of MapReduce framework counters.
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Luke Lu
              Reporter:
              Owen O'Malley
            • Votes:
              1 Vote for this issue
              Watchers:
              20 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development