Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-10497

Fix an issue in CapacityScheduler which fails to delete queues

    XMLWordPrintableJSON

    Details

      Description

      We saw an exception when using queue mutation APIs:

      2020-11-13 16:47:46,327 WARN org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices: CapacityScheduler configuration validation failed:java.io.IOException: Queue root.am2cmQueueSecond not found
      

      Which comes from this code:

              List<String> siblingQueues = getSiblingQueues(queueToRemove,
                  proposedConf);
              if (!siblingQueues.contains(queueName)) {
                throw new IOException("Queue " + queueToRemove + " not found");
              } 
      

      (Inside MutableCSConfigurationProvider)

      If you look at the method:

       
        private List<String> getSiblingQueues(String queuePath, Configuration conf) {
          String parentQueue = queuePath.substring(0, queuePath.lastIndexOf('.'));
          String childQueuesKey = CapacitySchedulerConfiguration.PREFIX +
              parentQueue + CapacitySchedulerConfiguration.DOT +
              CapacitySchedulerConfiguration.QUEUES;
          return new ArrayList<>(conf.getStringCollection(childQueuesKey));
        }
      

      And here's capacity-scheduler.xml I got

      <property><name>yarn.scheduler.capacity.root.queues</name><value>default, q1, q2</value></property>
      

      You can notice there're spaces between default, q1, a2

      So conf.getStringCollection returns:

      default
      <space>q1
      ...
      

      Which causes match issue when we try to delete the queue.

        Attachments

        1. YARN-10497.001.patch
          6 kB
          Wangda Tan
        2. YARN-10497.002.patch
          6 kB
          Wangda Tan
        3. YARN-10497.003.patch
          8 kB
          Wangda Tan
        4. YARN-10497.004.patch
          8 kB
          Wangda Tan
        5. YARN-10497.005.patch
          5 kB
          Qi Zhu
        6. YARN-10497.006.patch
          4 kB
          Qi Zhu

          Activity

            People

            • Assignee:
              wangda Wangda Tan
              Reporter:
              wangda Wangda Tan
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: