[SLIDER-1201] Slider should make resource normalization configurable for app-owners

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Slider 0.91
• Fix Version/s: Slider 0.92
• Component/s: appmaster
• Assignee: Gour Saha
• Reporter: Gour Saha
• Created: 02/Feb/17 19:33
• Updated: 06/Mar/17 21:23
• Resolved: 02/Feb/17 22:13

Description

Slider currently by default normalizes all resource requests which exceeds max allowed in the cluster.

2016-11-02 07:24:47,722 [main] WARN  state.AppState - Resource requirements of COMMAND_LOGGER normalized from <memory:186368, vCores:1> to <memory:143616, vCores:1>

This behavior should be made configurable. It will allow app owners to choose if they want normalization and the app to continue running with reduced resources (as default), OR if they do not want normalization and want the app to fail right away, rather than several days later when it actually tries to use as much memory it had requested, but never got. This causes a debugging nightmare as well, trying to figure out why an app was even killed by YARN (under false presumption that the app did not exceed its requested resource).

Solution:

A new property yarn.resource.normalization.enabled is being introduced which needs to be set in the app's resources definition. By default it is "true" (which was the behavior all along, to keep backward compatibility for all existing apps). To disable normalization, set it to "false" for the desired component of the app. An example is below -

{
  "schema": "http://example.org/specification/v2.0.0",
  "metadata": {
  },
  "global": {
  },
  "components": {
    "COMMAND_LOGGER": {
      "yarn.role.priority": "1",
      "yarn.component.instances": "1",
      "yarn.resource.normalization.enabled": "false",
      "yarn.memory": "186368",
      "yarn.vcores": "1"
    },
    "slider-appmaster": {
    }
  }
}

Issue Links

• relates to: SLIDER-1203 Document SLIDER-1201 - new feature for configurable resource normalization