Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5722

FairScheduler hides group resolution exceptions when assigning queue

    Details

    • Hadoop Flags:
      Reviewed

      Description

      When a group based placement rule is used and the user does not have any groups the reason for rejecting the application is hidden. An assignment will fail as follows:

      <rule name="primaryGroup" create="true" /> 
      <rule name="reject" /> 
      

      The error logged on the client side:

      09/30 15:59:27 INFO mapreduce.JobSubmitter: Cleaning up the staging area /user/test_user/.staging/job_1475223610304_6043 
      16/09/30 15:59:27 WARN security.UserGroupInformation: PriviledgedActionException as:test_user (auth:SIMPLE) cause:java.io.IOException: Failed to run job : Error assigning app to queue default 
      java.io.IOException: Failed to run job : Error assigning app to queue default 
      at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:301) 
      at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:244) 
      at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1307) 
      

      The default queue name is passed in as part of the application submission and not really the queue that is tried.

      1. YARN-5722.1.patch
        1 kB
        Wilfred Spiegelenburg
      2. YARN-5722.2.patch
        1 kB
        Wilfred Spiegelenburg

        Activity

        Hide
        wilfreds Wilfred Spiegelenburg added a comment -

        Simple change to pass back the message from the IOException and not the meaningless queue name.

        Show
        wilfreds Wilfred Spiegelenburg added a comment - Simple change to pass back the message from the IOException and not the meaningless queue name.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 18s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 6m 50s trunk passed
        +1 compile 0m 32s trunk passed
        +1 checkstyle 0m 20s trunk passed
        +1 mvnsite 0m 38s trunk passed
        +1 mvneclipse 0m 17s trunk passed
        +1 findbugs 0m 57s trunk passed
        +1 javadoc 0m 20s trunk passed
        +1 mvninstall 0m 30s the patch passed
        +1 compile 0m 30s the patch passed
        +1 javac 0m 30s the patch passed
        +1 checkstyle 0m 17s the patch passed
        +1 mvnsite 0m 35s the patch passed
        +1 mvneclipse 0m 13s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 3s the patch passed
        +1 javadoc 0m 18s the patch passed
        +1 unit 38m 23s hadoop-yarn-server-resourcemanager in the patch passed.
        +1 asflicense 0m 15s The patch does not generate ASF License warnings.
        52m 53s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832817/YARN-5722.1.patch
        JIRA Issue YARN-5722
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 853789c34bcc 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 7ba7092
        Default Java 1.8.0_101
        findbugs v3.0.0
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13355/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/13355/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 50s trunk passed +1 compile 0m 32s trunk passed +1 checkstyle 0m 20s trunk passed +1 mvnsite 0m 38s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 0m 57s trunk passed +1 javadoc 0m 20s trunk passed +1 mvninstall 0m 30s the patch passed +1 compile 0m 30s the patch passed +1 javac 0m 30s the patch passed +1 checkstyle 0m 17s the patch passed +1 mvnsite 0m 35s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 3s the patch passed +1 javadoc 0m 18s the patch passed +1 unit 38m 23s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 52m 53s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832817/YARN-5722.1.patch JIRA Issue YARN-5722 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 853789c34bcc 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 7ba7092 Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13355/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/13355/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        wilfreds Wilfred Spiegelenburg added a comment -

        There is no new test because it only exposes the error message properly in the log and responses

        Show
        wilfreds Wilfred Spiegelenburg added a comment - There is no new test because it only exposes the error message properly in the log and responses
        Hide
        yufeigu Yufei Gu added a comment -

        Thanks Wilfred Spiegelenburg for working on this. LGTM.

        Show
        yufeigu Yufei Gu added a comment - Thanks Wilfred Spiegelenburg for working on this. LGTM.
        Hide
        templedf Daniel Templeton added a comment -

        Thanks, Wilfred Spiegelenburg! In that patch, the error will now explain that the user has no primary group, but it won't say which queue was the cause. Maybe add that info to the exception?

        Show
        templedf Daniel Templeton added a comment - Thanks, Wilfred Spiegelenburg ! In that patch, the error will now explain that the user has no primary group, but it won't say which queue was the cause. Maybe add that info to the exception?
        Hide
        wilfreds Wilfred Spiegelenburg added a comment -

        Sorry for the late reply. I was doing other internal work for a couple of days.

        The queue has not been assigned yet because we have failed in one of the placement rules.

        That means the queue as we know it at that point will be the passed in queue to the assingToQueue call. This will most likely be the "default" queue unless the user has passed in a queue name in the configuration when submitted. In neither case the queue name will add or explain anything and it might even be confusing since the queue is irrelevant.

        Let me know if you still want to add the queue name to the error message.

        Show
        wilfreds Wilfred Spiegelenburg added a comment - Sorry for the late reply. I was doing other internal work for a couple of days. The queue has not been assigned yet because we have failed in one of the placement rules. That means the queue as we know it at that point will be the passed in queue to the assingToQueue call. This will most likely be the "default" queue unless the user has passed in a queue name in the configuration when submitted. In neither case the queue name will add or explain anything and it might even be confusing since the queue is irrelevant. Let me know if you still want to add the queue name to the error message.
        Hide
        wilfreds Wilfred Spiegelenburg added a comment -

        After discussion with Daniel Templeton off-line we came to the conclusion that a slight change of the message makes sense. Since we do not know the queue we now say "Error assigning app to a queue: " to remove the ambiguity around the queue.

        updating the patch with the agreed change

        Show
        wilfreds Wilfred Spiegelenburg added a comment - After discussion with Daniel Templeton off-line we came to the conclusion that a slight change of the message makes sense. Since we do not know the queue we now say "Error assigning app to a queue: " to remove the ambiguity around the queue. updating the patch with the agreed change
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 18s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 7m 37s trunk passed
        +1 compile 0m 32s trunk passed
        +1 checkstyle 0m 21s trunk passed
        +1 mvnsite 0m 40s trunk passed
        +1 mvneclipse 0m 17s trunk passed
        +1 findbugs 1m 0s trunk passed
        +1 javadoc 0m 22s trunk passed
        +1 mvninstall 0m 32s the patch passed
        +1 compile 0m 31s the patch passed
        +1 javac 0m 31s the patch passed
        +1 checkstyle 0m 18s the patch passed
        +1 mvnsite 0m 37s the patch passed
        +1 mvneclipse 0m 14s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 6s the patch passed
        +1 javadoc 0m 19s the patch passed
        -1 unit 40m 12s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 16s The patch does not generate ASF License warnings.
        56m 30s



        Reason Tests
        Failed junit tests hadoop.yarn.server.resourcemanager.TestTokenClientRMService



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-5722
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838911/YARN-5722.2.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux ed22528de432 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 99c2bbd
        Default Java 1.8.0_101
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-YARN-Build/13912/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13912/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/13912/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 37s trunk passed +1 compile 0m 32s trunk passed +1 checkstyle 0m 21s trunk passed +1 mvnsite 0m 40s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 1m 0s trunk passed +1 javadoc 0m 22s trunk passed +1 mvninstall 0m 32s the patch passed +1 compile 0m 31s the patch passed +1 javac 0m 31s the patch passed +1 checkstyle 0m 18s the patch passed +1 mvnsite 0m 37s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 6s the patch passed +1 javadoc 0m 19s the patch passed -1 unit 40m 12s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 56m 30s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestTokenClientRMService Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5722 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838911/YARN-5722.2.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux ed22528de432 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 99c2bbd Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/13912/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13912/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/13912/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        wilfreds Wilfred Spiegelenburg added a comment -

        TestTokenClientRMService failure is tracked in YARN-5875 and is not caused by this change.
        As commented earlier: there is no new test because it only exposes the error message properly

        Show
        wilfreds Wilfred Spiegelenburg added a comment - TestTokenClientRMService failure is tracked in YARN-5875 and is not caused by this change. As commented earlier: there is no new test because it only exposes the error message properly
        Hide
        templedf Daniel Templeton added a comment -

        +1 I'll check this in later today.

        Show
        templedf Daniel Templeton added a comment - +1 I'll check this in later today.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10874 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10874/)
        YARN-5722. FairScheduler hides group resolution exceptions when (templedf: rev 83cc7263af632939dc3b2ee58d8f03f98ed4d96a)

        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10874 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10874/ ) YARN-5722 . FairScheduler hides group resolution exceptions when (templedf: rev 83cc7263af632939dc3b2ee58d8f03f98ed4d96a) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
        Hide
        templedf Daniel Templeton added a comment -

        Thanks for the patch, Wilfred Spiegelenburg! Committed to trunk and branch-2.

        Show
        templedf Daniel Templeton added a comment - Thanks for the patch, Wilfred Spiegelenburg ! Committed to trunk and branch-2.
        Hide
        wilfreds Wilfred Spiegelenburg added a comment -

        Thank you Daniel Templeton for the review and commit

        Show
        wilfreds Wilfred Spiegelenburg added a comment - Thank you Daniel Templeton for the review and commit

          People

          • Assignee:
            wilfreds Wilfred Spiegelenburg
            Reporter:
            wilfreds Wilfred Spiegelenburg
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development