Details

    • Sub-task
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • None
    • 3.3.0
    • None
    • None
    • Reviewed

    Description

      Before YARN-8967, QueuePlacementRule class had a method called isTerminal(). However, sometimes this method was hard-coded to return false, accepting such configurations as:

          <queuePlacementPolicy>
              <rule name="specified" create="true"/>
              <rule name="nestedUserQueue" create="true">
                  <rule name="default" create="true" queue="users"/>
              </rule>
              <rule name="default"/>
          </queuePlacementPolicy>
      

      It's because NestedUserQueue.isTerminal() always returns false.

      This changed after YARN-8967. Now, this configuration is not accepted because QueuePlacementPolicy.fromXml() calculates a list of terminal rules differently:

      https://github.com/apache/hadoop/blob/5257f50abb71905ef3068fd45541d00ce9e8f355/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueuePlacementPolicy.java#L176-L183

      In order to migrate existing configuration that were created before YARN-8967, we need a new switch (at least in migration mode) in FS to turn off this validation, otherwise the tool will not be able to migrate these configs and the following exception will be thrown:

      ~$ ./yarn fs2cs -y /tmp/yarn-site.xml -f /tmp/fair-scheduler.xml -o /tmp
      WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.
      20/01/13 05:48:20 INFO converter.FSConfigToCSConfigConverter: Output directory for yarn-site.xml and capacity-scheduler.xml is: /tmp
      20/01/13 05:48:20 INFO converter.FSConfigToCSConfigConverter: Conversion rules file is not defined, using default conversion config!
      20/01/13 05:48:21 INFO converter.FSConfigToCSConfigConverter: Using explicitly defined fair-scheduler.xml
      WARNING: This feature is experimental and not intended for production use!
      20/01/13 05:48:21 INFO conf.Configuration: resource-types.xml not found
      20/01/13 05:48:21 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
      20/01/13 05:48:21 INFO security.YarnAuthorizationProvider: org.apache.hadoop.yarn.security.ConfiguredYarnAuthorizer is instantiated.
      20/01/13 05:48:21 INFO scheduler.AbstractYarnScheduler: Minimum allocation = <memory:1024, vCores:1>
      20/01/13 05:48:21 INFO scheduler.AbstractYarnScheduler: Maximum allocation = <memory:8192, vCores:2>
      20/01/13 05:48:21 INFO placement.PlacementFactory: Creating PlacementRule implementation: class org.apache.hadoop.yarn.server.resourcemanager.placement.SpecifiedPlacementRule
      20/01/13 05:48:21 INFO placement.PlacementFactory: Creating PlacementRule implementation: class org.apache.hadoop.yarn.server.resourcemanager.placement.UserPlacementRule
      20/01/13 05:48:21 INFO fair.AllocationFileLoaderService: Loading allocation file file:/tmp/fair-scheduler.xml
      20/01/13 05:48:22 INFO placement.PlacementFactory: Creating PlacementRule implementation: class org.apache.hadoop.yarn.server.resourcemanager.placement.SpecifiedPlacementRule
      20/01/13 05:48:22 INFO placement.PlacementFactory: Creating PlacementRule implementation: class org.apache.hadoop.yarn.server.resourcemanager.placement.UserPlacementRule
      20/01/13 05:48:22 INFO placement.PlacementFactory: Creating PlacementRule implementation: class org.apache.hadoop.yarn.server.resourcemanager.placement.DefaultPlacementRule
      20/01/13 05:48:22 INFO placement.PlacementFactory: Creating PlacementRule implementation: class org.apache.hadoop.yarn.server.resourcemanager.placement.DefaultPlacementRule
      20/01/13 05:48:22 INFO service.AbstractService: Service org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler failed in state INITED
      java.io.IOException: Failed to initialize FairScheduler
      at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.initScheduler(FairScheduler.java:1438)
      at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.serviceInit(FairScheduler.java:1479)
      at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
      at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverter.convert(FSConfigToCSConfigConverter.java:206)
      at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverter.convert(FSConfigToCSConfigConverter.java:101)
      at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigArgumentHandler.parseAndConvert(FSConfigToCSConfigArgumentHandler.java:116)
      at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverterMain.main(FSConfigToCSConfigConverterMain.java:44)
      Caused by: org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationConfigurationException: Rules after rule 2 in queue placement policy can never be reached
      at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueuePlacementPolicy.updateRuleSet(QueuePlacementPolicy.java:110)
      at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueuePlacementPolicy.fromXml(QueuePlacementPolicy.java:187)
      at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService.getQueuePlacementPolicy(AllocationFileLoaderService.java:276)
      at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService.reloadAllocations(AllocationFileLoaderService.java:257)
      at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.initScheduler(FairScheduler.java:1436)
      ... 6 more
      20/01/13 05:48:22 ERROR converter.FSConfigToCSConfigConverterMain: Error while starting FS configuration conversion!
      org.apache.hadoop.service.ServiceStateException: java.io.IOException: Failed to initialize FairScheduler
      at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:105)
      at org.apache.hadoop.service.AbstractService.init(AbstractService.java:173)
      at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverter.convert(FSConfigToCSConfigConverter.java:206)
      at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverter.convert(FSConfigToCSConfigConverter.java:101)
      at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigArgumentHandler.parseAndConvert(FSConfigToCSConfigArgumentHandler.java:116)
      at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverterMain.main(FSConfigToCSConfigConverterMain.java:44)
      Caused by: java.io.IOException: Failed to initialize FairScheduler
      at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.initScheduler(FairScheduler.java:1438)
      at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.serviceInit(FairScheduler.java:1479)
      at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
      ... 4 more
      Caused by: org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationConfigurationException: Rules after rule 2 in queue placement policy can never be reached
      at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueuePlacementPolicy.updateRuleSet(QueuePlacementPolicy.java:110)
      at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueuePlacementPolicy.fromXml(QueuePlacementPolicy.java:187)
      at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService.getQueuePlacementPolicy(AllocationFileLoaderService.java:276)
      at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService.reloadAllocations(AllocationFileLoaderService.java:257)
      at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.initScheduler(FairScheduler.java:1436)
      ... 6 more
      

      Attachments

        1. YARN-10082-001.patch
          9 kB
          Peter Bacsko
        2. YARN-10082-002.patch
          19 kB
          Peter Bacsko

        Activity

          People

            pbacsko Peter Bacsko
            pbacsko Peter Bacsko
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: