Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6647

MR usage counters use the resources requested instead of the resources allocated

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      As can be seen in the following snippet, the MR counters for usage use the resources requested instead of the resources allocated. The scheduler increment-allocation-mb configs could lead to these values not being the same. We could change the counters to use the allocated resources in order to account for this.

        private static void updateMillisCounters(JobCounterUpdateEvent jce,
            TaskAttemptImpl taskAttempt) {
           /***omitted**/
          long duration = (taskAttempt.getFinishTime() - taskAttempt.getLaunchTime());
          int mbRequired =
              taskAttempt.getMemoryRequired(taskAttempt.conf, taskType);
          int vcoresRequired = taskAttempt.getCpuRequired(taskAttempt.conf, taskType);
      
          int minSlotMemSize = taskAttempt.conf.getInt(
            YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_MB,
            YarnConfiguration.DEFAULT_RM_SCHEDULER_MINIMUM_ALLOCATION_MB);
      
          int simSlotsRequired =
              minSlotMemSize == 0 ? 0 : (int) Math.ceil((float) mbRequired
                  / minSlotMemSize);
      
          if (taskType == TaskType.MAP) {
            jce.addCounterUpdate(JobCounter.SLOTS_MILLIS_MAPS, simSlotsRequired * duration);
            jce.addCounterUpdate(JobCounter.MB_MILLIS_MAPS, duration * mbRequired);
            jce.addCounterUpdate(JobCounter.VCORES_MILLIS_MAPS, duration * vcoresRequired);
            jce.addCounterUpdate(JobCounter.MILLIS_MAPS, duration);
          } else {
            jce.addCounterUpdate(JobCounter.SLOTS_MILLIS_REDUCES, simSlotsRequired * duration);
            jce.addCounterUpdate(JobCounter.MB_MILLIS_REDUCES, duration * mbRequired);
            jce.addCounterUpdate(JobCounter.VCORES_MILLIS_REDUCES, duration * vcoresRequired);
            jce.addCounterUpdate(JobCounter.MILLIS_REDUCES, duration);
          }
      

        Attachments

        1. mapreduce6647.001.patch
          12 kB
          Haibo Chen
        2. mapreduce6647.002.patch
          15 kB
          Haibo Chen
        3. mapreduce6647.003.patch
          15 kB
          Haibo Chen
        4. mapreduce6647.004.patch
          13 kB
          Haibo Chen

          Issue Links

            Activity

              People

              • Assignee:
                haibochen Haibo Chen
                Reporter:
                haibochen Haibo Chen
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: