[YARN-4499] Bad config values of "yarn.scheduler.maximum-allocation-vcores"

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.7.1, 2.6.2
• Component/s: scheduler
• Assignee: Unassigned
• Reporter: Tianyin Xu
• Created: 23/Dec/15 07:49
• Updated: 25/Jan/16 09:17
• Resolved: 25/Jan/16 08:24

Description

Currently, the default value of yarn.scheduler.maximum-allocation-vcores is 32, according to yarn-default.xml.

However, in YarnConfiguration.java, we specify the default to be 4.

  public static final String RM_SCHEDULER_MAXIMUM_ALLOCATION_VCORES =
      YARN_PREFIX + "scheduler.maximum-allocation-vcores";
  public static final int DEFAULT_RM_SCHEDULER_MAXIMUM_ALLOCATION_VCORES = 4;

The default in the code looks correct to me. Actually I feel that the default value should be the same as yarn.nodemanager.resource.cpu-vcores (whose default is 8) ---if we have 8 cores for scheduling, there's few reason we only allow the maximum of 4...

The Cloudera's article on Tuning the Cluster for MapReduce v2 (YARN) also suggests that "the maximum value ( yarn.scheduler.maximum-allocation-vcores) is usually equal to yarn.nodemanager.resource.cpu-vcores..."

At the very least, we should fix yarn-default.xml. The error is pretty bad. A simple search on the Internet shows some ppl are confused by this error, for example,
https://community.cloudera.com/t5/Cloudera-Manager-Installation/yarn-nodemanager-resource-cpu-vcores-and-yarn-scheduler-maximum/td-p/31098

(but seriously, I think we should have an automatic defaults with the min as 1 and the max equal to the number of cores on the machine...