Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-9698 [Umbrella] Tools to help migration from Fair Scheduler to Capacity Scheduler
  3. YARN-10108

FS-CS converter: nestedUserQueue with default rule results in invalid queue mapping

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.3.0
    • 3.4.0, 3.3.1
    • None
    • Reviewed

    Description

      FS Queue Placement Policy

      <queuePlacementPolicy>
      <rule name="nestedUserQueue">
      <rule name="default" queue="users"/>
      </rule>
      </queuePlacementPolicy>
       

      gets mapped to an invalid CS queue mapping "u:%user:root.users.%user"

      RM fails to start with above queue mapping in CS

      2020-01-28 00:19:12,889 FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting ResourceManager
      org.apache.hadoop.service.ServiceStateException: java.io.IOException: mapping contains invalid or non-leaf queue [%user] and invalid parent queue [root.users]
      	at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:105)
      	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:173)
      	at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
      	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:829)
      	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
      	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1247)
      	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:324)
      	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
      	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1534)
      Caused by: java.io.IOException: mapping contains invalid or non-leaf queue [%user] and invalid parent queue [root.users]
      	at org.apache.hadoop.yarn.server.resourcemanager.placement.QueuePlacementRuleUtils.validateQueueMappingUnderParentQueue(QueuePlacementRuleUtils.java:48)
      	at org.apache.hadoop.yarn.server.resourcemanager.placement.UserGroupMappingPlacementRule.validateAndGetAutoCreatedQueueMapping(UserGroupMappingPlacementRule.java:363)
      	at org.apache.hadoop.yarn.server.resourcemanager.placement.UserGroupMappingPlacementRule.initialize(UserGroupMappingPlacementRule.java:300)
      	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.getUserGroupMappingPlacementRule(CapacityScheduler.java:671)
      	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.updatePlacementRules(CapacityScheduler.java:712)
      	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:753)
      	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:361)
      	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:426)
      	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
      	... 7 more
      

      QueuePlacementConverter#handleNestedRule has to be fixed.

      else if (pr instanceof DefaultPlacementRule) {
            DefaultPlacementRule defaultRule = (DefaultPlacementRule) pr;
            mapping.append("u:" + USER + ":")
              .append(defaultRule.defaultQueueName)
              .append("." + USER);
          }
      

      Attachments

        1. YARN-10108.branch-3.3.002.patch
          4 kB
          Gergely Pollák
        2. YARN-10108.branch-3.3.001.patch
          4 kB
          Szilard Nemeth
        3. YARN-10108.004.patch
          4 kB
          Gergely Pollák
        4. YARN-10108.003.patch
          4 kB
          Gergely Pollák
        5. YARN-10108.002.patch
          5 kB
          Gergely Pollák
        6. YARN-10108.001.patch
          2 kB
          Gergely Pollák

        Issue Links

          Activity

            People

              shuzirra Gergely Pollák
              prabhujoseph Prabhu Joseph
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: