Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: webapp
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      =application_1457010932347_0121
      2016-03-04 10:16:27,016 INFO org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer: application_1457010932347_0121 found existing hdfs token Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:hacluster, Ident: (HDFS_DELEGATION_TOKEN token 128 for yarn with renewer yarn)
      2016-03-04 10:16:27,029 ERROR org.apache.hadoop.yarn.webapp.Dispatcher: error handling URI: /cluster/apps
      java.lang.NullPointerException
              at org.apache.hadoop.yarn.server.resourcemanager.webapp.RMAppsBlock.renderData(RMAppsBlock.java:100)
              at org.apache.hadoop.yarn.server.webapp.AppsBlock.render(AppsBlock.java:140)
              at org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:69)
              at org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:79)
              at org.apache.hadoop.yarn.webapp.View.render(View.java:235)
              at org.apache.hadoop.yarn.webapp.view.HtmlBlock$Block.subView(HtmlBlock.java:43)
              at org.apache.hadoop.yarn.webapp.hamlet.Hamlet._(Hamlet.java:30354)
              at org.apache.hadoop.yarn.server.resourcemanager.webapp.AppsBlockWithMetrics.render(AppsBlockWithMetrics.java:30)
              at org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:69)
              at org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:79)
              at org.apache.hadoop.yarn.webapp.View.render(View.java:235)
              at org.apache.hadoop.yarn.webapp.view.HtmlPage$Page.subView(HtmlPage.java:49)
              at org.apache.hadoop.yarn.webapp.hamlet.HamletImpl$EImp._v(HamletImpl.java:117)
              at org.apache.hadoop.yarn.webapp.hamlet.Hamlet$TD._(Hamlet.java:848)
              at org.apache.hadoop.yarn.webapp.view.TwoColumnLayout.render(TwoColumnLayout.java:71)
              at org.apache.hadoop.yarn.webapp.view.HtmlPage.render(HtmlPage.java:82)
              at org.apache.hadoop.yarn.webapp.Dispatcher.render(Dispatcher.java:197)
              at org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:156)
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
              at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
      
      

      Application state is NEW the apptempts can be empty as per inital analysis

      rm.getRMContext().getRMApps()
                .get(appAttemptId.getApplicationId()).getAppAttempts()
                .get(appAttemptId)
      
      1. 0001-YARN-4763.patch
        2 kB
        Bibin A Chundatt

        Issue Links

          Activity

          Hide
          sunilg Sunil G added a comment -
                String blacklistedNodesCount = "N/A";
                Set<String> nodes = rm.getRMContext().getRMApps()
                    .get(appAttemptId.getApplicationId()).getAppAttempts()
                    .get(appAttemptId).getBlacklistedNodes();
                if (nodes != null) {
                  blacklistedNodesCount = String.valueOf(nodes.size());
                }
          

          This code will cause pblm. This will cause exception as attempt was not created in initial states such as NEW, NEW_SAVING etc. And we can reproduce this pblm by placing a debug point in RMAppImpl.

          Can u share a patch for this? Rohith Sharma K S cud u pls take a look.

          Show
          sunilg Sunil G added a comment - String blacklistedNodesCount = "N/A" ; Set< String > nodes = rm.getRMContext().getRMApps() .get(appAttemptId.getApplicationId()).getAppAttempts() .get(appAttemptId).getBlacklistedNodes(); if (nodes != null ) { blacklistedNodesCount = String .valueOf(nodes.size()); } This code will cause pblm. This will cause exception as attempt was not created in initial states such as NEW, NEW_SAVING etc. And we can reproduce this pblm by placing a debug point in RMAppImpl. Can u share a patch for this? Rohith Sharma K S cud u pls take a look.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 6m 52s trunk passed
          +1 compile 0m 26s trunk passed with JDK v1.8.0_74
          +1 compile 0m 29s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 18s trunk passed
          +1 mvnsite 0m 35s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 1m 5s trunk passed
          +1 javadoc 0m 21s trunk passed with JDK v1.8.0_74
          +1 javadoc 0m 25s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 29s the patch passed
          +1 compile 0m 24s the patch passed with JDK v1.8.0_74
          +1 javac 0m 24s the patch passed
          +1 compile 0m 26s the patch passed with JDK v1.7.0_95
          +1 javac 0m 26s the patch passed
          +1 checkstyle 0m 16s the patch passed
          +1 mvnsite 0m 31s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 1m 14s the patch passed
          +1 javadoc 0m 19s the patch passed with JDK v1.8.0_74
          +1 javadoc 0m 23s the patch passed with JDK v1.7.0_95
          -1 unit 70m 48s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_74.
          -1 unit 72m 33s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 19s Patch does not generate ASF License warnings.
          159m 52s



          Reason Tests
          JDK v1.8.0_74 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
          JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12791501/0001-YARN-4763.patch
          JIRA Issue YARN-4763
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 7a9f6fca3ce2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 3e8099a
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10711/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_74.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10711/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10711/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-YARN-Build/10711/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10711/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10711/console
          Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 52s trunk passed +1 compile 0m 26s trunk passed with JDK v1.8.0_74 +1 compile 0m 29s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 18s trunk passed +1 mvnsite 0m 35s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 1m 5s trunk passed +1 javadoc 0m 21s trunk passed with JDK v1.8.0_74 +1 javadoc 0m 25s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 29s the patch passed +1 compile 0m 24s the patch passed with JDK v1.8.0_74 +1 javac 0m 24s the patch passed +1 compile 0m 26s the patch passed with JDK v1.7.0_95 +1 javac 0m 26s the patch passed +1 checkstyle 0m 16s the patch passed +1 mvnsite 0m 31s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 14s the patch passed +1 javadoc 0m 19s the patch passed with JDK v1.8.0_74 +1 javadoc 0m 23s the patch passed with JDK v1.7.0_95 -1 unit 70m 48s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_74. -1 unit 72m 33s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 19s Patch does not generate ASF License warnings. 159m 52s Reason Tests JDK v1.8.0_74 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12791501/0001-YARN-4763.patch JIRA Issue YARN-4763 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 7a9f6fca3ce2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3e8099a Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/10711/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_74.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10711/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10711/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-YARN-Build/10711/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10711/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/10711/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          +1, will commit it

          Show
          rohithsharma Rohith Sharma K S added a comment - +1, will commit it
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Committed to trunk/branch-2. thanks Bibin A Chundatt for the patch and Sunil G for the review!!

          Show
          rohithsharma Rohith Sharma K S added a comment - Committed to trunk/branch-2. thanks Bibin A Chundatt for the patch and Sunil G for the review!!
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9428 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9428/)
          YARN-4763. RMApps Page crashes with NPE. (Bibin A Chundatt via (rohithsharmaks: rev 19ee1859071509bba9ecd0a8a7dc6a47e2979c88)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMAppsBlock.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9428 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9428/ ) YARN-4763 . RMApps Page crashes with NPE. (Bibin A Chundatt via (rohithsharmaks: rev 19ee1859071509bba9ecd0a8a7dc6a47e2979c88) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMAppsBlock.java
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Rohith Sharma K S
          Thank you for review and commit

          Show
          bibinchundatt Bibin A Chundatt added a comment - Rohith Sharma K S Thank you for review and commit
          Hide
          sandflee sandflee added a comment -

          I think a general way fix for this is we should get rmappattempt even the state is NEW.

          Show
          sandflee sandflee added a comment - I think a general way fix for this is we should get rmappattempt even the state is NEW.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          As I see this issue is not because of rmappattempt is in NEW state where as Attempt itself is not created yet. In such case, application report send current attempt id by using dummy application attempt number. This time period is very very small may be fraction of seconds!!

          if (currentApplicationAttemptId == null) {
                  currentApplicationAttemptId = 
                      BuilderUtils.newApplicationAttemptId(this.applicationId, 
                          DUMMY_APPLICATION_ATTEMPT_NUMBER);
                }
          
          Show
          rohithsharma Rohith Sharma K S added a comment - As I see this issue is not because of rmappattempt is in NEW state where as Attempt itself is not created yet. In such case, application report send current attempt id by using dummy application attempt number. This time period is very very small may be fraction of seconds!! if (currentApplicationAttemptId == null ) { currentApplicationAttemptId = BuilderUtils.newApplicationAttemptId( this .applicationId, DUMMY_APPLICATION_ATTEMPT_NUMBER); }
          Hide
          sandflee sandflee added a comment -

          yes, thanks for pointing this

          Show
          sandflee sandflee added a comment - yes, thanks for pointing this
          Hide
          jlowe Jason Lowe added a comment -

          Thanks, Bibin A Chundatt! I committed this to branch-2.8 since the JIRA that broke this, YARN-4307, was committed there as well.

          Show
          jlowe Jason Lowe added a comment - Thanks, Bibin A Chundatt ! I committed this to branch-2.8 since the JIRA that broke this, YARN-4307 , was committed there as well.

            People

            • Assignee:
              bibinchundatt Bibin A Chundatt
              Reporter:
              bibinchundatt Bibin A Chundatt
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development