Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.7.1, 2.6.2
    • Fix Version/s: 2.8.0, 2.7.3, 2.6.4, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      As reported in the forum by Wen Lin (wlin@pivotal.io)

      [gpadmin@master simple-yarn-app]$ hadoop jar
      ~/hadoop/singlecluster/hadoop/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.6.0.3.0.0.0-120.jar
      Client --classpath ./target/simple-yarn-app-1.1.0.jar -cmd "java
      com.hortonworks.simpleyarnapp.ApplicationMaster /bin/date 2"

      error is coming as

      org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in handling event type REGISTERED for applicationAttempt
      application_1450079798629_0001
      664 java.lang.NullPointerException
      665     at
      org.apache.hadoop.yarn.server.resourcemanager.metrics.SystemMetricsPublisher.appAttemptRegistered(SystemMetricsPublisher.java:143)
      666     at
      org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl$AMRegisteredTransition.transition(RMAppAttemptImpl.java:1365)
      667     at
      org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl$AMRegisteredTransition.transition(RMAppAttemptImpl.java:1341)
      668     at
      org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:362)
      
      1. YARN-4452.v1.001.patch
        4 kB
        Naganarasimha G R
      2. YARN-4452.v1.002.patch
        7 kB
        Naganarasimha G R
      3. YARN-4452-branch-2.6.3.v1.001.patch
        6 kB
        Naganarasimha G R

        Issue Links

          Activity

          Hide
          wlin Lin Wen added a comment -

          Here is how to reproduce it.
          1. On Hadoop Yarn, timeline server is started/enabled and "yarn.resourcemanager.system-metrics-publisher.enabled" is enable in yarn-site.xml.
          <property>
          <description>The hostname of the timeline server web application.</description>
          <name>yarn.timeline-service.hostname</name>
          <value>master</value>
          </property>
          <property>
          <description>Enable or disable the GHS</description>
          <name>yarn.resourcemanager.system-metrics-publisher.enabled</name>
          <value>true</value>
          </property>
          <property>
          <description>Enable or disable the Timeline Server.</description>
          <name>yarn.timeline-service.enabled</name>
          <value>true</value>
          </property>
          <property>
          <description>Store class name for timeline store</description>
          <name>yarn.timeline-service.store-class</name>
          <value>org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore</value>
          </property>
          <property>
          <description>Store file name for leveldb timeline store</description>
          <name>yarn.timeline-service.leveldb-timeline-store.path</name>
          <value>/data/1/yarn/logs/timeline</value>
          </property>
          2. Use hortonworks' simple-yarn-app(https://github.com/hortonworks/simple-yarn-app), and start it in "unmanaged AM" mode.
          hadoop jar ~/hadoop/singlecluster/hadoop/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.6.0.3.0.0.0-120.jar Client --classpath ./target/simple-yarn-app-1.1.0.jar -cmd "java com.hortonworks.simpleyarnapp.ApplicationMaster /bin/date 2"

          Show
          wlin Lin Wen added a comment - Here is how to reproduce it. 1. On Hadoop Yarn, timeline server is started/enabled and "yarn.resourcemanager.system-metrics-publisher.enabled" is enable in yarn-site.xml. <property> <description>The hostname of the timeline server web application.</description> <name>yarn.timeline-service.hostname</name> <value>master</value> </property> <property> <description>Enable or disable the GHS</description> <name>yarn.resourcemanager.system-metrics-publisher.enabled</name> <value>true</value> </property> <property> <description>Enable or disable the Timeline Server.</description> <name>yarn.timeline-service.enabled</name> <value>true</value> </property> <property> <description>Store class name for timeline store</description> <name>yarn.timeline-service.store-class</name> <value>org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore</value> </property> <property> <description>Store file name for leveldb timeline store</description> <name>yarn.timeline-service.leveldb-timeline-store.path</name> <value>/data/1/yarn/logs/timeline</value> </property> 2. Use hortonworks' simple-yarn-app( https://github.com/hortonworks/simple-yarn-app ), and start it in "unmanaged AM" mode. hadoop jar ~/hadoop/singlecluster/hadoop/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.6.0.3.0.0.0-120.jar Client --classpath ./target/simple-yarn-app-1.1.0.jar -cmd "java com.hortonworks.simpleyarnapp.ApplicationMaster /bin/date 2"
          Hide
          wlin Lin Wen added a comment -

          I can see below information in log in Yarn's log file:
          2015-12-10 02:52:19,025 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: Storing attempt: AppId: application_1449744734026_0001 AttemptId: appattempt_1449744734026_0001_000001 MasterContainer: null
          ...
          2015-12-10 02:52:19,946 ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in handling event type REGISTERED for applicationAttempt application_1449744734026_0001
          java.lang.NullPointerException
          at org.apache.hadoop.yarn.server.resourcemanager.metrics.SystemMetricsPublisher.appAttemptRegistered(SystemMetricsPublisher.java:145)

          I guess since there is no container allocated for "unmanaged" application master, so MasterContainer is null. But when Yarn register this application into SystemMetricsPublisher, it requires a container and its id. That's why this null exception happens.
          private void storeAttempt()

          { // store attempt data in a non-blocking manner to prevent dispatcher // thread starvation and wait for state to be saved LOG.info("Storing attempt: AppId: " + getAppAttemptId().getApplicationId() + " AttemptId: " + getAppAttemptId() + " MasterContainer: " + masterContainer); rmContext.getStateStore().storeNewApplicationAttempt(this); }

          public void appAttemptRegistered(RMAppAttempt appAttempt,
          long registeredTime) {
          if (publishSystemMetrics)

          { dispatcher.getEventHandler().handle( new AppAttemptRegisteredEvent( appAttempt.getAppAttemptId(), appAttempt.getHost(), appAttempt.getRpcPort(), appAttempt.getTrackingUrl(), appAttempt.getOriginalTrackingUrl(), appAttempt.getMasterContainer().getId(), registeredTime)); }

          }
          In a word, if a unmanaged AM tries to register in Yarn, when timeline server is configured and "yarn.resourcemanager.system-metrics-publisher.enabled" is enable, a java NullPointerException occurs in Yarn.

          Show
          wlin Lin Wen added a comment - I can see below information in log in Yarn's log file: 2015-12-10 02:52:19,025 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: Storing attempt: AppId: application_1449744734026_0001 AttemptId: appattempt_1449744734026_0001_000001 MasterContainer: null ... 2015-12-10 02:52:19,946 ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in handling event type REGISTERED for applicationAttempt application_1449744734026_0001 java.lang.NullPointerException at org.apache.hadoop.yarn.server.resourcemanager.metrics.SystemMetricsPublisher.appAttemptRegistered(SystemMetricsPublisher.java:145) I guess since there is no container allocated for "unmanaged" application master, so MasterContainer is null. But when Yarn register this application into SystemMetricsPublisher, it requires a container and its id. That's why this null exception happens. private void storeAttempt() { // store attempt data in a non-blocking manner to prevent dispatcher // thread starvation and wait for state to be saved LOG.info("Storing attempt: AppId: " + getAppAttemptId().getApplicationId() + " AttemptId: " + getAppAttemptId() + " MasterContainer: " + masterContainer); rmContext.getStateStore().storeNewApplicationAttempt(this); } public void appAttemptRegistered(RMAppAttempt appAttempt, long registeredTime) { if (publishSystemMetrics) { dispatcher.getEventHandler().handle( new AppAttemptRegisteredEvent( appAttempt.getAppAttemptId(), appAttempt.getHost(), appAttempt.getRpcPort(), appAttempt.getTrackingUrl(), appAttempt.getOriginalTrackingUrl(), appAttempt.getMasterContainer().getId(), registeredTime)); } } In a word, if a unmanaged AM tries to register in Yarn, when timeline server is configured and "yarn.resourcemanager.system-metrics-publisher.enabled" is enable, a java NullPointerException occurs in Yarn.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Junping Du & Jian He,
          Verified this issue against 2.6.3, it throws NPE.

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Junping Du & Jian He , Verified this issue against 2.6.3, it throws NPE.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          simple WIP trunk patch ( test cases pending )

          Show
          Naganarasimha Naganarasimha G R added a comment - simple WIP trunk patch ( test cases pending )
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Trunk patch with test case

          Show
          Naganarasimha Naganarasimha G R added a comment - Trunk patch with test case
          Hide
          djp Junping Du added a comment -

          Thanks Naganarasimha G R for reporting this and delivering patch to fix it. Given unmanaged application with ATS enabled is not quite mainstream case. How about we target for next 2.6/2.7 release and make it critical here (not blocker)?
          002 patch LGTM. +1 pending on Jenkins.

          Show
          djp Junping Du added a comment - Thanks Naganarasimha G R for reporting this and delivering patch to fix it. Given unmanaged application with ATS enabled is not quite mainstream case. How about we target for next 2.6/2.7 release and make it critical here (not blocker)? 002 patch LGTM. +1 pending on Jenkins.
          Hide
          djp Junping Du added a comment -

          Just think it again. Another thing we may want to fix here is: the NPE happen at SystemMetricsPublisher shouldn't bring down RM rudely. Isn't it?

          Show
          djp Junping Du added a comment - Just think it again. Another thing we may want to fix here is: the NPE happen at SystemMetricsPublisher shouldn't bring down RM rudely. Isn't it?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 39s trunk passed
          +1 compile 0m 31s trunk passed with JDK v1.8.0_66
          +1 compile 0m 33s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 14s trunk passed
          +1 mvnsite 0m 41s trunk passed
          +1 mvneclipse 0m 16s trunk passed
          +1 findbugs 1m 31s trunk passed
          +1 javadoc 0m 25s trunk passed with JDK v1.8.0_66
          +1 javadoc 0m 28s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 42s the patch passed
          +1 compile 0m 34s the patch passed with JDK v1.8.0_66
          +1 javac 0m 34s the patch passed
          +1 compile 0m 40s the patch passed with JDK v1.7.0_91
          +1 javac 0m 40s the patch passed
          +1 checkstyle 0m 19s the patch passed
          +1 mvnsite 0m 49s the patch passed
          +1 mvneclipse 0m 20s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 1m 26s the patch passed
          +1 javadoc 0m 28s the patch passed with JDK v1.8.0_66
          +1 javadoc 0m 29s the patch passed with JDK v1.7.0_91
          -1 unit 59m 47s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66.
          -1 unit 60m 22s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_91.
          +1 asflicense 0m 22s Patch does not generate ASF License warnings.
          140m 44s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
          JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12777824/YARN-4452.v1.002.patch
          JIRA Issue YARN-4452
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 1b185afae204 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 8602692
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-YARN-Build/9990/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/9990/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/9990/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/9990/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9990/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Max memory used 75MB
          Powered by Apache Yetus 0.1.0 http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9990/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 39s trunk passed +1 compile 0m 31s trunk passed with JDK v1.8.0_66 +1 compile 0m 33s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 14s trunk passed +1 mvnsite 0m 41s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 1m 31s trunk passed +1 javadoc 0m 25s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 28s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 42s the patch passed +1 compile 0m 34s the patch passed with JDK v1.8.0_66 +1 javac 0m 34s the patch passed +1 compile 0m 40s the patch passed with JDK v1.7.0_91 +1 javac 0m 40s the patch passed +1 checkstyle 0m 19s the patch passed +1 mvnsite 0m 49s the patch passed +1 mvneclipse 0m 20s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 26s the patch passed +1 javadoc 0m 28s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 29s the patch passed with JDK v1.7.0_91 -1 unit 59m 47s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66. -1 unit 60m 22s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 140m 44s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12777824/YARN-4452.v1.002.patch JIRA Issue YARN-4452 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 1b185afae204 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8602692 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/9990/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/9990/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/9990/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/9990/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9990/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Max memory used 75MB Powered by Apache Yetus 0.1.0 http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/9990/console This message was automatically generated.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Thanks for review comments Junping Du,

          Given unmanaged application with ATS enabled is not quite mainstream case. How about we target for next 2.6/2.7 release and make it critical here (not blocker)?

          Yes should not be a problem, had only converted from critical to blocker as one application was getting the RM down.

          Another thing we may want to fix here is: the NPE happen at SystemMetricsPublisher shouldn't bring down RM rudely. Isn't it?

          So do we try catch in each of these publishing events ? generally it will not happen this was a direct bug in the code . If you feel its required then can add it but felt it was not required.

          Show
          Naganarasimha Naganarasimha G R added a comment - Thanks for review comments Junping Du , Given unmanaged application with ATS enabled is not quite mainstream case. How about we target for next 2.6/2.7 release and make it critical here (not blocker)? Yes should not be a problem, had only converted from critical to blocker as one application was getting the RM down. Another thing we may want to fix here is: the NPE happen at SystemMetricsPublisher shouldn't bring down RM rudely. Isn't it? So do we try catch in each of these publishing events ? generally it will not happen this was a direct bug in the code . If you feel its required then can add it but felt it was not required.
          Hide
          djp Junping Du added a comment -

          Adding try catch for each event push may be not necessary. But may be we can consider to add some unexpectedExceptionHandler to some non-critical component (like metrics) so NPE or other exceptions on these component won't be necessary to bring down RM. We can have a separate JIRA to fix it if you don't want to address here.

          Show
          djp Junping Du added a comment - Adding try catch for each event push may be not necessary. But may be we can consider to add some unexpectedExceptionHandler to some non-critical component (like metrics) so NPE or other exceptions on these component won't be necessary to bring down RM. We can have a separate JIRA to fix it if you don't want to address here.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Junping Du, thanks for sharing your views.

          But may be we can consider to add some unexpectedExceptionHandler to some non-critical component (like metrics) so NPE or other exceptions on these component won't be necessary to bring down RM. We can have a separate JIRA to fix it if you don't want to address here.

          +1 for this approach but I believe it can be done in other jira (as this critical jira), if you are fine will create a new jira and work in that.

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Junping Du , thanks for sharing your views. But may be we can consider to add some unexpectedExceptionHandler to some non-critical component (like metrics) so NPE or other exceptions on these component won't be necessary to bring down RM. We can have a separate JIRA to fix it if you don't want to address here. +1 for this approach but I believe it can be done in other jira (as this critical jira), if you are fine will create a new jira and work in that.
          Hide
          djp Junping Du added a comment -

          Filed YARN-4466. Committing 002 patch.

          Show
          djp Junping Du added a comment - Filed YARN-4466 . Committing 002 patch.
          Hide
          djp Junping Du added a comment -

          I just committed 002 patch to trunk and branch-2. However, I meet some conflict/build error in backporting to 2.6. Naga, can you put a patch for 2.6? Thanks!

          Show
          djp Junping Du added a comment - I just committed 002 patch to trunk and branch-2. However, I meet some conflict/build error in backporting to 2.6. Naga, can you put a patch for 2.6? Thanks!
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #8975 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8975/)
          YARN-4452. NPE when submit Unmanaged application. Contributed by (junping_du: rev 50bd067e1d63d4c80dc1e7bf4024bfaf42cf4416)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisher.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/SystemMetricsPublisher.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #8975 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8975/ ) YARN-4452 . NPE when submit Unmanaged application. Contributed by (junping_du: rev 50bd067e1d63d4c80dc1e7bf4024bfaf42cf4416) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisher.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/SystemMetricsPublisher.java
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          will provide for it shortly !

          Show
          Naganarasimha Naganarasimha G R added a comment - will provide for it shortly !
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Uploading the patch based on branch-2.6.3

          Show
          Naganarasimha Naganarasimha G R added a comment - Uploading the patch based on branch-2.6.3
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 11m 9s branch-2.6.3 passed
          +1 compile 0m 25s branch-2.6.3 passed with JDK v1.8.0_66
          +1 compile 0m 25s branch-2.6.3 passed with JDK v1.7.0_91
          +1 checkstyle 0m 18s branch-2.6.3 passed
          +1 mvnsite 0m 34s branch-2.6.3 passed
          +1 mvneclipse 0m 20s branch-2.6.3 passed
          -1 findbugs 1m 2s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager in branch-2.6.3 has 2 extant Findbugs warnings.
          -1 javadoc 0m 22s hadoop-yarn-server-resourcemanager in branch-2.6.3 failed with JDK v1.8.0_66.
          +1 javadoc 0m 21s branch-2.6.3 passed with JDK v1.7.0_91
          +1 mvninstall 0m 29s the patch passed
          +1 compile 0m 21s the patch passed with JDK v1.8.0_66
          +1 javac 0m 21s the patch passed
          +1 compile 0m 27s the patch passed with JDK v1.7.0_91
          +1 javac 0m 27s the patch passed
          +1 checkstyle 0m 12s the patch passed
          +1 mvnsite 0m 32s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          -1 whitespace 0m 1s The patch has 1450 line(s) that end in whitespace. Use git apply --whitespace=fix.
          -1 whitespace 0m 40s The patch has 101 line(s) with tabs.
          +1 findbugs 1m 2s the patch passed
          -1 javadoc 0m 18s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66.
          +1 javadoc 0m 22s the patch passed with JDK v1.7.0_91
          -1 unit 45m 16s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66.
          -1 unit 46m 25s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_91.
          -1 asflicense 0m 43s Patch generated 123 ASF License warnings.
          113m 25s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
          JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore
            hadoop.yarn.server.resourcemanager.TestAMAuthorization



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:date2015-12-17
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778223/YARN-4452-branch-2.6.3.v1.001.patch
          JIRA Issue YARN-4452
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 4296e0251093 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2.6.3 / 95d8146
          findbugs v1.3.9
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/10018/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-warnings.html
          javadoc https://builds.apache.org/job/PreCommit-YARN-Build/10018/artifact/patchprocess/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/10018/artifact/patchprocess/whitespace-eol.txt
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/10018/artifact/patchprocess/whitespace-tabs.txt
          javadoc https://builds.apache.org/job/PreCommit-YARN-Build/10018/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10018/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10018/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10018/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/10018/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10018/testReport/
          asflicense https://builds.apache.org/job/PreCommit-YARN-Build/10018/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Max memory used 70MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10018/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 11m 9s branch-2.6.3 passed +1 compile 0m 25s branch-2.6.3 passed with JDK v1.8.0_66 +1 compile 0m 25s branch-2.6.3 passed with JDK v1.7.0_91 +1 checkstyle 0m 18s branch-2.6.3 passed +1 mvnsite 0m 34s branch-2.6.3 passed +1 mvneclipse 0m 20s branch-2.6.3 passed -1 findbugs 1m 2s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager in branch-2.6.3 has 2 extant Findbugs warnings. -1 javadoc 0m 22s hadoop-yarn-server-resourcemanager in branch-2.6.3 failed with JDK v1.8.0_66. +1 javadoc 0m 21s branch-2.6.3 passed with JDK v1.7.0_91 +1 mvninstall 0m 29s the patch passed +1 compile 0m 21s the patch passed with JDK v1.8.0_66 +1 javac 0m 21s the patch passed +1 compile 0m 27s the patch passed with JDK v1.7.0_91 +1 javac 0m 27s the patch passed +1 checkstyle 0m 12s the patch passed +1 mvnsite 0m 32s the patch passed +1 mvneclipse 0m 14s the patch passed -1 whitespace 0m 1s The patch has 1450 line(s) that end in whitespace. Use git apply --whitespace=fix. -1 whitespace 0m 40s The patch has 101 line(s) with tabs. +1 findbugs 1m 2s the patch passed -1 javadoc 0m 18s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66. +1 javadoc 0m 22s the patch passed with JDK v1.7.0_91 -1 unit 45m 16s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66. -1 unit 46m 25s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_91. -1 asflicense 0m 43s Patch generated 123 ASF License warnings. 113m 25s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore   hadoop.yarn.server.resourcemanager.TestAMAuthorization JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:date2015-12-17 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778223/YARN-4452-branch-2.6.3.v1.001.patch JIRA Issue YARN-4452 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 4296e0251093 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.6.3 / 95d8146 findbugs v1.3.9 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/10018/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-warnings.html javadoc https://builds.apache.org/job/PreCommit-YARN-Build/10018/artifact/patchprocess/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/10018/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/10018/artifact/patchprocess/whitespace-tabs.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/10018/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10018/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10018/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10018/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/10018/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10018/testReport/ asflicense https://builds.apache.org/job/PreCommit-YARN-Build/10018/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Max memory used 70MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/10018/console This message was automatically generated.
          Hide
          djp Junping Du added a comment -

          Thanks Naga. A correction here is the patch will be commit to 2.6 (for 2.6.4) instead of 2.6.3 given 2.6.3 RC is out and pass the vote. I have commit the patch to trunk, branch-2, branch-2.6, branch-2.7 and branch-2.8. Thanks Naga for the contribution.

          Show
          djp Junping Du added a comment - Thanks Naga. A correction here is the patch will be commit to 2.6 (for 2.6.4) instead of 2.6.3 given 2.6.3 RC is out and pass the vote. I have commit the patch to trunk, branch-2, branch-2.6, branch-2.7 and branch-2.8. Thanks Naga for the contribution.
          Hide
          wlin Lin Wen added a comment -

          Naga and Junping,
          Thank you very much!

          Show
          wlin Lin Wen added a comment - Naga and Junping, Thank you very much!
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Thanks for reviewing and committing the patch Junping Du, and Lin Wen for raising the issue.

          patch will be commit to 2.6 (for 2.6.4) instead of 2.6.3 given 2.6.3 RC is out and pass the vote.

          As i had already branch for 2.6.3 in my machine, created the patch for the issue on top of it (also its same as 2.6 branch).

          Show
          Naganarasimha Naganarasimha G R added a comment - Thanks for reviewing and committing the patch Junping Du , and Lin Wen for raising the issue. patch will be commit to 2.6 (for 2.6.4) instead of 2.6.3 given 2.6.3 RC is out and pass the vote. As i had already branch for 2.6.3 in my machine, created the patch for the issue on top of it (also its same as 2.6 branch).
          Hide
          jlowe Jason Lowe added a comment -

          I recently came across the same type of NPE in a 2.7.3-based build. The cherry-picks for branch-2.8 and branch-2.7 were performed incorrectly, as if it was using the patch for 2.6. Only one of the two null checks from the trunk patch were present. I reverted the original commits and re-picked the trunk patch to branch-2.8 and branch-2.7.

          Show
          jlowe Jason Lowe added a comment - I recently came across the same type of NPE in a 2.7.3-based build. The cherry-picks for branch-2.8 and branch-2.7 were performed incorrectly, as if it was using the patch for 2.6. Only one of the two null checks from the trunk patch were present. I reverted the original commits and re-picked the trunk patch to branch-2.8 and branch-2.7.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Closing the JIRA as part of 2.7.3 release.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Closing the JIRA as part of 2.7.3 release.

            People

            • Assignee:
              Naganarasimha Naganarasimha G R
              Reporter:
              Naganarasimha Naganarasimha G R
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development