Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-2230

Fix description of yarn.scheduler.maximum-allocation-vcores in yarn-default.xml (or code)

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • 2.4.0
    • 2.7.0
    • client, documentation, scheduler
    • None

    Description

      When a user requests more vcores than the allocation limit (e.g. mapreduce.map.cpu.vcores is larger than yarn.scheduler.maximum-allocation-vcores), then InvalidResourceRequestException is thrown - https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java

          if (resReq.getCapability().getVirtualCores() < 0 ||
              resReq.getCapability().getVirtualCores() >
              maximumResource.getVirtualCores()) {
            throw new InvalidResourceRequestException("Invalid resource request"
                + ", requested virtual cores < 0"
                + ", or requested virtual cores > max configured"
                + ", requestedVirtualCores="
                + resReq.getCapability().getVirtualCores()
                + ", maxVirtualCores=" + maximumResource.getVirtualCores());
          }
      

      According to documentation - yarn-default.xml http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-common/yarn-default.xml, the request should be capped to the allocation limit.

        <property>
          <description>The maximum allocation for every container request at the RM,
          in terms of virtual CPU cores. Requests higher than this won't take effect,
          and will get capped to this value.</description>
          <name>yarn.scheduler.maximum-allocation-vcores</name>
          <value>32</value>
        </property>
      

      This means that:

      • Either documentation or code should be corrected (unless this exception is handled elsewhere accordingly, but it looks that it is not).

      This behavior is confusing, because when such a job (with mapreduce.map.cpu.vcores is larger than yarn.scheduler.maximum-allocation-vcores) is submitted, it does not make any progress. The warnings/exceptions are thrown at the scheduler (RM) side e.g.

      2014-06-29 00:34:51,469 WARN org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: Invalid resource ask by application appattempt_1403993411503_0002_000001
      org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid resource request, requested virtual cores < 0, or requested virtual cores > max configured, requestedVirtualCores=32, maxVirtualCores=3
      	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:237)
      	at org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.validateResourceRequests(RMServerUtils.java:80)
      	at org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:420)
              .....
      	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
      	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
      	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1986)
      	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1982)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:416)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
      	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1980)
      
      • IMHO, such an exception should be forwarded to client. Otherwise, it is non obvious to discover why a job does not make any progress.

      The same looks to be related to memory.

      Attachments

        1. YARN-2230.002.patch
          2 kB
          Vijay Bhat
        2. YARN-2230.001.patch
          1 kB
          Vijay Bhat

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            vijaysbhat Vijay Bhat
            kawaa Adam Kawa
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment