Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-9204

RM fails to start if absolute resource is specified for partition capacity in CS queues

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 3.1.3
    • Fix Version/s: 3.1.2, 3.3.0, 3.2.1
    • Component/s: yarn
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      When I set yarn.scheduler.capacity.<queue-path>.capacity and yarn.scheduler.capacity.<queue-path>.accessible-node-labels.<node-label>.capacity to absolute resource value, staring RM fails, and throw following exception, and after diving into relate code, I found the logic of checking absolute resource value maybe wrong.

      2019-01-17 20:25:45,716 FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting ResourceManager
      java.lang.NumberFormatException: For input string: "[memory=40960,vcore=48]"
              at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2043)
              at sun.misc.FloatingDecimal.parseFloat(FloatingDecimal.java:122)
              at java.lang.Float.parseFloat(Float.java:451)
              at org.apache.hadoop.conf.Configuration.getFloat(Configuration.java:1606)
              at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.internalGetLabeledQueue
      Capacity(CapacitySchedulerConfiguration.java:655)
              at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getLabeledQueueCapacity
      (CapacitySchedulerConfiguration.java:670)
              at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadCapacitiesByLabelsFromConf(CSQueueUti
      ls.java:135)
              at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadUpdateAndCheckCapacities(CSQueueUtils
      .java:110)
              at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupConfigurableCapacities(AbstractCS
      Queue.java:179)
              at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupQueueConfigs(AbstractCSQueue.java
      :356)
              at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupQueueConfigs(AbstractCSQueue.java
      :323)
              at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:130)
              at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.<init>(ParentQueue.java:112)
              at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySched
      ulerQueueManager.java:275)
              at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(Capacit
      ySchedulerQueueManager.java:158)
              at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.j
      ava:715)
              at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java
      :360)
              at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:4
      25)
              at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
              at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
              at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:817)
              at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
              at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1218)
              at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:317)
              at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
              at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1500)
      2019-01-17 20:25:45,719 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: SHUTDOWN_MSG:
      

        Attachments

        1. YARN-9204.001.patch
          2 kB
          Jiandan Yang
        2. YARN-9204.002.patch
          8 kB
          Jiandan Yang
        3. YARN-9204.003.patch
          7 kB
          Jiandan Yang
        4. YARN-9204.004.patch
          7 kB
          Jiandan Yang
        5. YARN-9204.005.patch
          7 kB
          Jiandan Yang
        6. YARN-9204.006.patch
          7 kB
          Jiandan Yang

          Issue Links

            Activity

              People

              • Assignee:
                yangjiandan Jiandan Yang
                Reporter:
                yangjiandan Jiandan Yang
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: