Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-10458

Hive On Tez queries fails upon submission to dynamically created pools



    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.4.0, 3.3.1
    • resourcemanager
    • None


      While using Dynamic Auto-Creation and Management of Leaf Queues, we could see that the queue creation fails because ACL submit application check couldn't succeed.

      We tried setting acl_submit_applications to '*' for managed parent queues. For static queues, this worked but failed for dynamic queues. Also tried setting the below property but it didn't help either.

      RM error log shows the following :

      2020-09-18 01:08:40,579 INFO org.apache.hadoop.yarn.server.resourcemanager.placement.UserGroupMappingPlacementRule: Application application_1600399068816_0460 user user1 mapping [default] to [queue1] override false
      2020-09-18 01:08:40,579 WARN org.apache.hadoop.yarn.server.resourcemanager.RMAppManager: User 'user1' from application tag does not have access to queue 'user1'. The placement is done for user 'hive'

      Checking the code, scheduler#checkAccess() bails out even before checking the ACL permissions for that particular queue because the CSQueue is null.

      public boolean checkAccess(UserGroupInformation callerUGI,
      QueueACL acl, String queueName) {
      CSQueue queue = getQueue(queueName);
      if (queue == null) {
      if (LOG.isDebugEnabled())
      { LOG.debug("ACL not found for queue access-type " + acl + " for queue " + queueName); }
      return false;                    *<-- the method returns false here.*
      return queue.hasAccess(acl, callerUGI);

      As this is an auto created queue, CSQueue may be null in this case. May be scheduler#checkAccess() should have a logic to differentiate when CSQueue is null and if queue mapping is involved and if so, check if the parent queue exists and is a managed parent and if so, check if the parent queue has valid ACL's instead of returning false ?



        1. YARN-10458-001.patch
          3 kB
          Peter Bacsko
        2. YARN-10458-002.patch
          13 kB
          Peter Bacsko
        3. YARN-10458-003.patch
          19 kB
          Peter Bacsko
        4. YARN-10458-004.patch
          19 kB
          Peter Bacsko
        5. YARN-10458-branch-3.3.001.patch
          19 kB
          Peter Bacsko



            pbacsko Peter Bacsko
            anand.srinivasan Anand Srinivasan
            0 Vote for this issue
            6 Start watching this issue