Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4235

FairScheduler PrimaryGroup does not handle empty groups returned for a user

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: fairscheduler
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      We see NPE if empty groups are returned for a user. This causes a NPE and cause RM to crash as below

      2015-09-22 16:51:52,780  FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in handling event type APP_ADDED to the scheduler
      java.lang.IndexOutOfBoundsException: Index: 0
      	at java.util.Collections$EmptyList.get(Collections.java:3212)
      	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueuePlacementRule$PrimaryGroup.getQueueForApp(QueuePlacementRule.java:149)
      	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueuePlacementRule.assignAppToQueue(QueuePlacementRule.java:74)
      	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueuePlacementPolicy.assignAppToQueue(QueuePlacementPolicy.java:167)
      	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.assignToQueue(FairScheduler.java:689)
      	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.addApplication(FairScheduler.java:595)
      	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:1180)
      	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:111)
      	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:684)
      	at java.lang.Thread.run(Thread.java:745)
      2015-09-22 16:51:52,797  INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting, bbye..
      

        Activity

        Hide
        adhoot Anubhav Dhoot added a comment -

        Thanks Rohith Sharma K S for review and commit!

        Show
        adhoot Anubhav Dhoot added a comment - Thanks Rohith Sharma K S for review and commit!
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2447 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2447/)
        YARN-4235. FairScheduler PrimaryGroup does not handle empty groups (rohithsharmaks: rev 8f195387a4a4a5a278119bf4c2f15cad61f0e2c7)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestQueuePlacementPolicy.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueuePlacementRule.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2447 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2447/ ) YARN-4235 . FairScheduler PrimaryGroup does not handle empty groups (rohithsharmaks: rev 8f195387a4a4a5a278119bf4c2f15cad61f0e2c7) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestQueuePlacementPolicy.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueuePlacementRule.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #503 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/503/)
        YARN-4235. FairScheduler PrimaryGroup does not handle empty groups (rohithsharmaks: rev 8f195387a4a4a5a278119bf4c2f15cad61f0e2c7)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestQueuePlacementPolicy.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueuePlacementRule.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #503 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/503/ ) YARN-4235 . FairScheduler PrimaryGroup does not handle empty groups (rohithsharmaks: rev 8f195387a4a4a5a278119bf4c2f15cad61f0e2c7) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestQueuePlacementPolicy.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueuePlacementRule.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #1240 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1240/)
        YARN-4235. FairScheduler PrimaryGroup does not handle empty groups (rohithsharmaks: rev 8f195387a4a4a5a278119bf4c2f15cad61f0e2c7)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueuePlacementRule.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestQueuePlacementPolicy.java
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1240 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1240/ ) YARN-4235 . FairScheduler PrimaryGroup does not handle empty groups (rohithsharmaks: rev 8f195387a4a4a5a278119bf4c2f15cad61f0e2c7) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueuePlacementRule.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestQueuePlacementPolicy.java hadoop-yarn-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #513 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/513/)
        YARN-4235. FairScheduler PrimaryGroup does not handle empty groups (rohithsharmaks: rev 8f195387a4a4a5a278119bf4c2f15cad61f0e2c7)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestQueuePlacementPolicy.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueuePlacementRule.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #513 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/513/ ) YARN-4235 . FairScheduler PrimaryGroup does not handle empty groups (rohithsharmaks: rev 8f195387a4a4a5a278119bf4c2f15cad61f0e2c7) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestQueuePlacementPolicy.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueuePlacementRule.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8599 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8599/)
        YARN-4235. FairScheduler PrimaryGroup does not handle empty groups (rohithsharmaks: rev 8f195387a4a4a5a278119bf4c2f15cad61f0e2c7)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueuePlacementRule.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestQueuePlacementPolicy.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8599 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8599/ ) YARN-4235 . FairScheduler PrimaryGroup does not handle empty groups (rohithsharmaks: rev 8f195387a4a4a5a278119bf4c2f15cad61f0e2c7) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueuePlacementRule.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestQueuePlacementPolicy.java
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        committed to branch-2/trunk. thanks Anubhav Dhoot!!

        Show
        rohithsharma Rohith Sharma K S added a comment - committed to branch-2/trunk. thanks Anubhav Dhoot!!
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        +1 lgtm

        Show
        rohithsharma Rohith Sharma K S added a comment - +1 lgtm
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 17m 7s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 7m 58s There were no new javac warning messages.
        +1 javadoc 10m 16s There were no new javadoc warning messages.
        -1 release audit 0m 19s The applied patch generated 1 release audit warnings.
        +1 checkstyle 0m 51s There were no new checkstyle issues.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 29s mvn install still works.
        +1 eclipse:eclipse 0m 36s The patch built with eclipse:eclipse.
        +1 findbugs 1m 28s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 yarn tests 56m 36s Tests passed in hadoop-yarn-server-resourcemanager.
            96m 43s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12765465/YARN-4235.001.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 6dd47d7
        Release Audit https://builds.apache.org/job/PreCommit-YARN-Build/9373/artifact/patchprocess/patchReleaseAuditProblems.txt
        hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9373/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9373/testReport/
        Java 1.7.0_55
        uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/9373/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 7s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 58s There were no new javac warning messages. +1 javadoc 10m 16s There were no new javadoc warning messages. -1 release audit 0m 19s The applied patch generated 1 release audit warnings. +1 checkstyle 0m 51s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 29s mvn install still works. +1 eclipse:eclipse 0m 36s The patch built with eclipse:eclipse. +1 findbugs 1m 28s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 56m 36s Tests passed in hadoop-yarn-server-resourcemanager.     96m 43s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12765465/YARN-4235.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 6dd47d7 Release Audit https://builds.apache.org/job/PreCommit-YARN-Build/9373/artifact/patchprocess/patchReleaseAuditProblems.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9373/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9373/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9373/console This message was automatically generated.
        Hide
        adhoot Anubhav Dhoot added a comment -

        Handle empty groups

        Show
        adhoot Anubhav Dhoot added a comment - Handle empty groups

          People

          • Assignee:
            adhoot Anubhav Dhoot
            Reporter:
            adhoot Anubhav Dhoot
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development