[MAPREDUCE-6647] MR usage counters use the resources requested instead of the resources allocated

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.9.0, 3.0.0-alpha1
• Hadoop Flags: Reviewed
• Assignee: Haibo Chen
• Reporter: Haibo Chen
• Created: 02/Mar/16 23:11
• Updated: 29/Jun/17 19:59
• Resolved: 07/Apr/16 00:16

Description

As can be seen in the following snippet, the MR counters for usage use the resources requested instead of the resources allocated. The scheduler increment-allocation-mb configs could lead to these values not being the same. We could change the counters to use the allocated resources in order to account for this.

  private static void updateMillisCounters(JobCounterUpdateEvent jce,
      TaskAttemptImpl taskAttempt) {
     /***omitted**/
    long duration = (taskAttempt.getFinishTime() - taskAttempt.getLaunchTime());
    int mbRequired =
        taskAttempt.getMemoryRequired(taskAttempt.conf, taskType);
    int vcoresRequired = taskAttempt.getCpuRequired(taskAttempt.conf, taskType);

    int minSlotMemSize = taskAttempt.conf.getInt(
      YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_MB,
      YarnConfiguration.DEFAULT_RM_SCHEDULER_MINIMUM_ALLOCATION_MB);

    int simSlotsRequired =
        minSlotMemSize == 0 ? 0 : (int) Math.ceil((float) mbRequired
            / minSlotMemSize);

    if (taskType == TaskType.MAP) {
      jce.addCounterUpdate(JobCounter.SLOTS_MILLIS_MAPS, simSlotsRequired * duration);
      jce.addCounterUpdate(JobCounter.MB_MILLIS_MAPS, duration * mbRequired);
      jce.addCounterUpdate(JobCounter.VCORES_MILLIS_MAPS, duration * vcoresRequired);
      jce.addCounterUpdate(JobCounter.MILLIS_MAPS, duration);
    } else {
      jce.addCounterUpdate(JobCounter.SLOTS_MILLIS_REDUCES, simSlotsRequired * duration);
      jce.addCounterUpdate(JobCounter.MB_MILLIS_REDUCES, duration * mbRequired);
      jce.addCounterUpdate(JobCounter.VCORES_MILLIS_REDUCES, duration * vcoresRequired);
      jce.addCounterUpdate(JobCounter.MILLIS_REDUCES, duration);
    }

Issue Links

• is related to: MAPREDUCE-6677 LocalContainerAllocator doesn't specify resource of the containers allocated.