Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-10458

Hive On Tez queries fails upon submission to dynamically created pools

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.4.0, 3.3.1
    • Component/s: resourcemanager
    • Labels:
      None

      Description

      While using Dynamic Auto-Creation and Management of Leaf Queues, we could see that the queue creation fails because ACL submit application check couldn't succeed.

      We tried setting acl_submit_applications to '*' for managed parent queues. For static queues, this worked but failed for dynamic queues. Also tried setting the below property but it didn't help either.
      yarn.scheduler.capacity.root.parent-queue-name.leaf-queue-template.acl_submit_applications=*.

      RM error log shows the following :

      2020-09-18 01:08:40,579 INFO org.apache.hadoop.yarn.server.resourcemanager.placement.UserGroupMappingPlacementRule: Application application_1600399068816_0460 user user1 mapping [default] to [queue1] override false
      2020-09-18 01:08:40,579 WARN org.apache.hadoop.yarn.server.resourcemanager.RMAppManager: User 'user1' from application tag does not have access to queue 'user1'. The placement is done for user 'hive'
       

      Checking the code, scheduler#checkAccess() bails out even before checking the ACL permissions for that particular queue because the CSQueue is null.

      public boolean checkAccess(UserGroupInformation callerUGI,
      QueueACL acl, String queueName) {
      CSQueue queue = getQueue(queueName);
      if (queue == null) {
      if (LOG.isDebugEnabled())
      
      { LOG.debug("ACL not found for queue access-type " + acl + " for queue " + queueName); }
      return false;                    *<-- the method returns false here.*
      }
      return queue.hasAccess(acl, callerUGI);
      }
      

      As this is an auto created queue, CSQueue may be null in this case. May be scheduler#checkAccess() should have a logic to differentiate when CSQueue is null and if queue mapping is involved and if so, check if the parent queue exists and is a managed parent and if so, check if the parent queue has valid ACL's instead of returning false ?

      Thanks

        Attachments

        1. YARN-10458-001.patch
          3 kB
          Peter Bacsko
        2. YARN-10458-002.patch
          13 kB
          Peter Bacsko
        3. YARN-10458-003.patch
          19 kB
          Peter Bacsko
        4. YARN-10458-004.patch
          19 kB
          Peter Bacsko
        5. YARN-10458-branch-3.3.001.patch
          19 kB
          Peter Bacsko

          Activity

            People

            • Assignee:
              pbacsko Peter Bacsko
              Reporter:
              anand.srinivasan Anand Srinivasan

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment