Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-8720

CapacityScheduler does not enforce max resource allocation check at queue level

    Details

    • Hadoop Flags:
      Reviewed

      Description

      The value of yarn.scheduler.capacity.<queue-path>.maximum-allocation-mb/vcores is not strictly enforced when applications request containers. An InvalidResourceRequestException is thrown only when the ResourceRequest is greater than the global value of yarn.scheduler.maximum-allocation-mb/vcores . So for an example configuration such as below,

       

      yarn.scheduler.maximum-allocation-mb=4096
      yarn.scheduler.capacity.root.test.maximum-allocation-mb=2048
      

       

      The below DSShell command runs successfully and asks an AM container of size 4096MB which is greater than max 2048MB configured in test queue.

      yarn jar $YARN_HOME/hadoop-yarn-applications-distributedshell.jar -num_containers 1 -jar $YARN_HOME/hadoop-yarn-applications-distributedshell.jar -shell_command "sleep 60" -container_memory=4096 -master_memory=4096 -queue=test

      Instead it should not launch the application and fail with InvalidResourceRequestException . The child container however will be requested with size 2048MB as DSShell AppMaster does the below check before ResourceRequest ask with RM.

      // A resource ask cannot exceed the max.
      if (containerMemory > maxMem) {
       LOG.info("Container memory specified above max threshold of cluster."
       + " Using max value." + ", specified=" + containerMemory + ", max="
       + maxMem);
       containerMemory = maxMem;
      }

       

        Attachments

        1. YARN-8720.001.patch
          2 kB
          Tarun Parimi
        2. YARN-8720.002.patch
          5 kB
          Tarun Parimi
        3. YARN-8720-branch-2.8.001.patch
          5 kB
          Tarun Parimi

          Issue Links

            Activity

              People

              • Assignee:
                tarunparimi Tarun Parimi
                Reporter:
                tarunparimi Tarun Parimi
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: