Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4478 [Umbrella] : Track all the Test failures in YARN
  3. YARN-4623

TestSystemMetricsPublisher#testPublishAppAttemptMetricsForUnmanagedAM fails with NPE

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.3
    • Fix Version/s: 2.7.3
    • Component/s: yarn
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      testPublishAppAttemptMetricsForUnmanagedAM is failing with a NullPointerException on branch-2.7.

      Running org.apache.hadoop.yarn.server.resourcemanager.metrics.TestSystemMetricsPublisher
      Tests run: 4, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 6.393 sec <<< FAILURE! - in org.apache.hadoop.yarn.server.resourcemanager.metrics.TestSystemMetricsPublisher
      testPublishAppAttemptMetricsForUnmanagedAM(org.apache.hadoop.yarn.server.resourcemanager.metrics.TestSystemMetricsPublisher)  Time elapsed: 0.026 sec  <<< ERROR!
      java.lang.NullPointerException: null
      	at org.apache.hadoop.yarn.server.resourcemanager.metrics.SystemMetricsPublisher.appAttemptFinished(SystemMetricsPublisher.java:163)
      	at org.apache.hadoop.yarn.server.resourcemanager.metrics.TestSystemMetricsPublisher.testPublishAppAttemptMetricsForUnmanagedAM(TestSystemMetricsPublisher.java:204)
      

        Issue Links

          Activity

          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Closing the JIRA as part of 2.7.3 release.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Closing the JIRA as part of 2.7.3 release.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Thanks for looking into it Jason Lowe, sorry missed to check in other place !

          Show
          Naganarasimha Naganarasimha G R added a comment - Thanks for looking into it Jason Lowe , sorry missed to check in other place !
          Hide
          jlowe Jason Lowe added a comment -

          After this commit we're seeing NPEs in the ResourceManager similar to that reported in YARN-4452. Looks like not all of the fixes in trunk patch for YARN-4452 made it to branch-2.7, and that was the real culprit of this test failure. This change restored part of the missing trunk patch for branch-2.7 but not all of it. I think we can fix this on YARN-4452 by committing the trunk version of the patch to branch-2.7.

          Show
          jlowe Jason Lowe added a comment - After this commit we're seeing NPEs in the ResourceManager similar to that reported in YARN-4452 . Looks like not all of the fixes in trunk patch for YARN-4452 made it to branch-2.7, and that was the real culprit of this test failure. This change restored part of the missing trunk patch for branch-2.7 but not all of it. I think we can fix this on YARN-4452 by committing the trunk version of the patch to branch-2.7.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Thanks for the review and commit Junping Du, Jason Lowe & Rohith Sharma K S !

          Show
          Naganarasimha Naganarasimha G R added a comment - Thanks for the review and commit Junping Du , Jason Lowe & Rohith Sharma K S !
          Hide
          djp Junping Du added a comment -

          I have commit the patch to branch-2.7. Thanks Naganarasimha G R for patch, Jason for report it and Rohith for review!

          Show
          djp Junping Du added a comment - I have commit the patch to branch-2.7. Thanks Naganarasimha G R for patch, Jason for report it and Rohith for review!
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 7m 31s branch-2.7 passed
          +1 compile 0m 25s branch-2.7 passed with JDK v1.8.0_66
          +1 compile 0m 29s branch-2.7 passed with JDK v1.7.0_91
          +1 checkstyle 0m 18s branch-2.7 passed
          +1 mvnsite 0m 33s branch-2.7 passed
          +1 mvneclipse 0m 19s branch-2.7 passed
          -1 findbugs 1m 15s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager in branch-2.7 has 1 extant Findbugs warnings.
          +1 javadoc 0m 23s branch-2.7 passed with JDK v1.8.0_66
          +1 javadoc 0m 25s branch-2.7 passed with JDK v1.7.0_91
          +1 mvninstall 0m 28s the patch passed
          +1 compile 0m 22s the patch passed with JDK v1.8.0_66
          +1 javac 0m 22s the patch passed
          +1 compile 0m 24s the patch passed with JDK v1.7.0_91
          +1 javac 0m 24s the patch passed
          -1 checkstyle 0m 13s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: patch generated 1 new + 95 unchanged - 0 fixed = 96 total (was 95)
          +1 mvnsite 0m 32s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          -1 whitespace 0m 2s The patch has 1115 line(s) that end in whitespace. Use git apply --whitespace=fix.
          -1 whitespace 0m 33s The patch has 97 line(s) with tabs.
          +1 findbugs 1m 18s the patch passed
          +1 javadoc 0m 17s the patch passed with JDK v1.8.0_66
          +1 javadoc 0m 21s the patch passed with JDK v1.7.0_91
          -1 unit 52m 59s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66.
          -1 unit 53m 17s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_91.
          -1 asflicense 44m 8s Patch generated 77 ASF License warnings.
          185m 15s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
          JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestRMRestart
            hadoop.yarn.server.resourcemanager.TestAMAuthorization



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:date2016-01-21
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12783602/YARN-4623-branch-2.7.v1.001.patch
          JIRA Issue YARN-4623
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 4055aa89d81a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2.7 / bade7f0
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/10356/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-warnings.html
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10356/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/10356/artifact/patchprocess/whitespace-eol.txt
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/10356/artifact/patchprocess/whitespace-tabs.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10356/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10356/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10356/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/10356/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10356/testReport/
          asflicense https://builds.apache.org/job/PreCommit-YARN-Build/10356/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Max memory used 71MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10356/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 31s branch-2.7 passed +1 compile 0m 25s branch-2.7 passed with JDK v1.8.0_66 +1 compile 0m 29s branch-2.7 passed with JDK v1.7.0_91 +1 checkstyle 0m 18s branch-2.7 passed +1 mvnsite 0m 33s branch-2.7 passed +1 mvneclipse 0m 19s branch-2.7 passed -1 findbugs 1m 15s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager in branch-2.7 has 1 extant Findbugs warnings. +1 javadoc 0m 23s branch-2.7 passed with JDK v1.8.0_66 +1 javadoc 0m 25s branch-2.7 passed with JDK v1.7.0_91 +1 mvninstall 0m 28s the patch passed +1 compile 0m 22s the patch passed with JDK v1.8.0_66 +1 javac 0m 22s the patch passed +1 compile 0m 24s the patch passed with JDK v1.7.0_91 +1 javac 0m 24s the patch passed -1 checkstyle 0m 13s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: patch generated 1 new + 95 unchanged - 0 fixed = 96 total (was 95) +1 mvnsite 0m 32s the patch passed +1 mvneclipse 0m 13s the patch passed -1 whitespace 0m 2s The patch has 1115 line(s) that end in whitespace. Use git apply --whitespace=fix. -1 whitespace 0m 33s The patch has 97 line(s) with tabs. +1 findbugs 1m 18s the patch passed +1 javadoc 0m 17s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 21s the patch passed with JDK v1.7.0_91 -1 unit 52m 59s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66. -1 unit 53m 17s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_91. -1 asflicense 44m 8s Patch generated 77 ASF License warnings. 185m 15s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:date2016-01-21 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12783602/YARN-4623-branch-2.7.v1.001.patch JIRA Issue YARN-4623 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 4055aa89d81a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.7 / bade7f0 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/10356/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-warnings.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10356/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/10356/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/10356/artifact/patchprocess/whitespace-tabs.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10356/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10356/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10356/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/10356/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10356/testReport/ asflicense https://builds.apache.org/job/PreCommit-YARN-Build/10356/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Max memory used 71MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/10356/console This message was automatically generated.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Thanks for confirmation.. +1

          Show
          rohithsharma Rohith Sharma K S added a comment - Thanks for confirmation.. +1
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          +1 LGTM too

          Show
          rohithsharma Rohith Sharma K S added a comment - +1 LGTM too
          Hide
          djp Junping Du added a comment -

          Thanks Jason Lowe for reporting on this and Naga give a quick patch update. The patch LGTM. Will commit it shortly if no objects.

          Show
          djp Junping Du added a comment - Thanks Jason Lowe for reporting on this and Naga give a quick patch update. The patch LGTM. Will commit it shortly if no objects.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Thanks for pointing it out Rohith Sharma K S, Code is different from trunk & 2.6, hence had give a different patch in YARN-4452. In 2.6 SystemMetricsPublisher#appAttemptFinished is not storing the master containerId, i verified the same again by running the testcase and checking the code in branch 2.6.

          Show
          Naganarasimha Naganarasimha G R added a comment - Thanks for pointing it out Rohith Sharma K S , Code is different from trunk & 2.6, hence had give a different patch in YARN-4452 . In 2.6 SystemMetricsPublisher#appAttemptFinished is not storing the master containerId, i verified the same again by running the testcase and checking the code in branch 2.6.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Naganarasimha G R can you check in branch-2.6 also for this issue existence? Because hadoop-2.6 patch in YARN-4452 does not contain modification in SystemMetricsPublisher#appAttemptFinished

          Show
          rohithsharma Rohith Sharma K S added a comment - Naganarasimha G R can you check in branch-2.6 also for this issue existence? Because hadoop-2.6 patch in YARN-4452 does not contain modification in SystemMetricsPublisher#appAttemptFinished
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Jason Lowe & Junping Du, seems like the complete patch was not applied in 2.7, Only test case exists without the fix which was given in the trunk. Have uploaded the patch for the Branch 2.7. Please review

          Show
          Naganarasimha Naganarasimha G R added a comment - Jason Lowe & Junping Du , seems like the complete patch was not applied in 2.7, Only test case exists without the fix which was given in the trunk. Have uploaded the patch for the Branch 2.7. Please review
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Thanks for reporting Jason Lowe, am working on it will upload a patch at the earliest!

          Show
          Naganarasimha Naganarasimha G R added a comment - Thanks for reporting Jason Lowe , am working on it will upload a patch at the earliest!
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Thanks for reporting Jason Lowe, am working on it will upload a patch at the earliest!

          Show
          Naganarasimha Naganarasimha G R added a comment - Thanks for reporting Jason Lowe , am working on it will upload a patch at the earliest!
          Hide
          jlowe Jason Lowe added a comment -

          Test failure appears to have been introduced by YARN-4452. Naganarasimha G R or Junping Du could you take a look?

          Show
          jlowe Jason Lowe added a comment - Test failure appears to have been introduced by YARN-4452 . Naganarasimha G R or Junping Du could you take a look?

            People

            • Assignee:
              Naganarasimha Naganarasimha G R
              Reporter:
              jlowe Jason Lowe
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development