Uploaded image for project: 'Slider'
  1. Slider
  2. SLIDER-1201

Slider should make resource normalization configurable for app-owners

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: Slider 0.91
    • Fix Version/s: Slider 0.92
    • Component/s: appmaster
    • Labels:
      None

      Description

      Slider currently by default normalizes all resource requests which exceeds max allowed in the cluster.

      2016-11-02 07:24:47,722 [main] WARN  state.AppState - Resource requirements of COMMAND_LOGGER normalized from <memory:186368, vCores:1> to <memory:143616, vCores:1>
      

      This behavior should be made configurable. It will allow app owners to choose if they want normalization and the app to continue running with reduced resources (as default), OR if they do not want normalization and want the app to fail right away, rather than several days later when it actually tries to use as much memory it had requested, but never got. This causes a debugging nightmare as well, trying to figure out why an app was even killed by YARN (under false presumption that the app did not exceed its requested resource).

      Solution:

      A new property yarn.resource.normalization.enabled is being introduced which needs to be set in the app's resources definition. By default it is "true" (which was the behavior all along, to keep backward compatibility for all existing apps). To disable normalization, set it to "false" for the desired component of the app. An example is below -

      {
        "schema": "http://example.org/specification/v2.0.0",
        "metadata": {
        },
        "global": {
        },
        "components": {
          "COMMAND_LOGGER": {
            "yarn.role.priority": "1",
            "yarn.component.instances": "1",
            "yarn.resource.normalization.enabled": "false",
            "yarn.memory": "186368",
            "yarn.vcores": "1"
          },
          "slider-appmaster": {
          }
        }
      }
      

        Attachments

        1. SLIDER-1201.01.patch
          11 kB
          Gour Saha

          Issue Links

            Activity

              People

              • Assignee:
                gsaha Gour Saha
                Reporter:
                gsaha Gour Saha
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: