Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4880

Running TestZKRMStateStorePerf with real zookeeper cluster throws NPE

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      While going throw TestZKRMStateStorePerf class , found that we are not initializing variable TestingServer curatorTestingServer if real zookeeper cluster are passed to utility. But down the line , this variables are used which causes NPE

      I tested by passing program arguments which result in NPE

       
      SLF4J: Class path contains multiple SLF4J bindings.
      SLF4J: Found binding in [jar:file:/C:/Users/r00902292/.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
      SLF4J: Found binding in [jar:file:/C:/Users/r00902292/.m2/repository/ch/qos/logback/logback-classic/1.1.2/logback-classic-1.1.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
      SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
      SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
      2016-03-26 14:47:50,937 INFO  [main] recovery.TestZKRMStateStore (TestZKRMStateStorePerf.java:run(119)) - Starting ZKRMStateStorePerf ver.0.1
      java.lang.NullPointerException
      	at org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStorePerf.initStore(TestZKRMStateStorePerf.java:102)
      	at org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStorePerf.run(TestZKRMStateStorePerf.java:156)
      	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
      	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
      	at org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStorePerf.main(TestZKRMStateStorePerf.java:273)
      

      There are 2 places variable curatorTestingServer used that need to be guarded with null check.

        Activity

        Hide
        sunilg Sunil G added a comment -

        Hi Rohith Sharma K S
        May be we can put curatorTestingServer initializing @BeforeClass instead of @Before. we can use {{curatorTestingServer#start()} from @Before.

        Show
        sunilg Sunil G added a comment - Hi Rohith Sharma K S May be we can put curatorTestingServer initializing @BeforeClass instead of @Before. we can use {{curatorTestingServer#start()} from @Before.
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        I do not think changing annotation would help here rather null check. Annotated methods are called while running as JUnit Test applications. But when launched with Java applications, still problem would remain same.

        Show
        rohithsharma Rohith Sharma K S added a comment - I do not think changing annotation would help here rather null check. Annotated methods are called while running as JUnit Test applications. But when launched with Java applications, still problem would remain same.
        Hide
        sunilg Sunil G added a comment -

        Oh Yes. This can run an independent java app also.We also can try initialize it while declaring itself, because null check will have to be done in all inits. Its fine both ways

        Show
        sunilg Sunil G added a comment - Oh Yes. This can run an independent java app also.We also can try initialize it while declaring itself, because null check will have to be done in all inits. Its fine both ways
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        I thought about this initially but since it start new ZK server I felt it is not necessarily required to start new ZK server when user is specifying external ZK host and port. And also default port at which curatorTestingServer try to start is 2181. But if external zk is running with default port and this test is in same machine, then we may face port issues.

        Show
        rohithsharma Rohith Sharma K S added a comment - I thought about this initially but since it start new ZK server I felt it is not necessarily required to start new ZK server when user is specifying external ZK host and port. And also default port at which curatorTestingServer try to start is 2181. But if external zk is running with default port and this test is in same machine, then we may face port issues.
        Hide
        sunilg Sunil G added a comment -

        Yes, that make sense. Thanks for sharing.

        Show
        sunilg Sunil G added a comment - Yes, that make sense. Thanks for sharing.
        Hide
        sunilg Sunil G added a comment -

        HI Rohith. I would like to give a try if you are not working on same. Pls reassign otherwise.

        Show
        sunilg Sunil G added a comment - HI Rohith. I would like to give a try if you are not working on same. Pls reassign otherwise.
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        Sure, go ahead

        Show
        rohithsharma Rohith Sharma K S added a comment - Sure, go ahead
        Hide
        sunilg Sunil G added a comment -

        Thanks Rohith Sharma K S.
        Attaching an initial patch to address the issue.

        Thank You.

        Show
        sunilg Sunil G added a comment - Thanks Rohith Sharma K S . Attaching an initial patch to address the issue. Thank You.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 9s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 6m 43s trunk passed
        +1 compile 0m 25s trunk passed with JDK v1.8.0_77
        +1 compile 0m 28s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 18s trunk passed
        +1 mvnsite 0m 33s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        +1 findbugs 1m 5s trunk passed
        +1 javadoc 0m 21s trunk passed with JDK v1.8.0_77
        +1 javadoc 0m 26s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 29s the patch passed
        +1 compile 0m 23s the patch passed with JDK v1.8.0_77
        +1 javac 0m 23s the patch passed
        +1 compile 0m 26s the patch passed with JDK v1.7.0_95
        +1 javac 0m 26s the patch passed
        +1 checkstyle 0m 15s the patch passed
        +1 mvnsite 0m 32s the patch passed
        +1 mvneclipse 0m 12s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 1m 16s the patch passed
        +1 javadoc 0m 17s the patch passed with JDK v1.8.0_77
        +1 javadoc 0m 23s the patch passed with JDK v1.7.0_95
        -1 unit 59m 48s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_77.
        -1 unit 61m 6s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95.
        +1 asflicense 0m 17s Patch does not generate ASF License warnings.
        137m 6s



        Reason Tests
        JDK v1.8.0_77 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization
          hadoop.yarn.server.resourcemanager.TestRMAdminService
          hadoop.yarn.server.resourcemanager.TestClientRMTokens
        JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization
          hadoop.yarn.server.resourcemanager.TestRMAdminService
          hadoop.yarn.server.resourcemanager.TestClientRMTokens



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:fbe3e86
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12796743/0001-YARN-4880.patch
        JIRA Issue YARN-4880
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 2dc1c84bf9ad 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 0ecdd4c
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-YARN-Build/10949/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_77.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/10949/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10949/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-YARN-Build/10949/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10949/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/10949/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 9s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 43s trunk passed +1 compile 0m 25s trunk passed with JDK v1.8.0_77 +1 compile 0m 28s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 18s trunk passed +1 mvnsite 0m 33s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 5s trunk passed +1 javadoc 0m 21s trunk passed with JDK v1.8.0_77 +1 javadoc 0m 26s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 29s the patch passed +1 compile 0m 23s the patch passed with JDK v1.8.0_77 +1 javac 0m 23s the patch passed +1 compile 0m 26s the patch passed with JDK v1.7.0_95 +1 javac 0m 26s the patch passed +1 checkstyle 0m 15s the patch passed +1 mvnsite 0m 32s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 16s the patch passed +1 javadoc 0m 17s the patch passed with JDK v1.8.0_77 +1 javadoc 0m 23s the patch passed with JDK v1.7.0_95 -1 unit 59m 48s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_77. -1 unit 61m 6s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 17s Patch does not generate ASF License warnings. 137m 6s Reason Tests JDK v1.8.0_77 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestRMAdminService   hadoop.yarn.server.resourcemanager.TestClientRMTokens JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestRMAdminService   hadoop.yarn.server.resourcemanager.TestClientRMTokens Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12796743/0001-YARN-4880.patch JIRA Issue YARN-4880 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 2dc1c84bf9ad 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0ecdd4c Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/10949/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_77.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10949/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10949/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-YARN-Build/10949/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10949/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/10949/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        sunilg Sunil G added a comment -

        Rohith Sharma K S, could you please take a look.

        Show
        sunilg Sunil G added a comment - Rohith Sharma K S , could you please take a look.
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        +1 lgtm

        Show
        rohithsharma Rohith Sharma K S added a comment - +1 lgtm
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        committed to trunk/branch-2.. thanks Sunil G

        Show
        rohithsharma Rohith Sharma K S added a comment - committed to trunk/branch-2.. thanks Sunil G
        Hide
        sunilg Sunil G added a comment -

        Thank you very much Rohith for the review and commit.

        Show
        sunilg Sunil G added a comment - Thank you very much Rohith for the review and commit.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #9556 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9556/)
        YARN-4880. Running TestZKRMStateStorePerf with real zookeeper cluster (rohithsharmaks: rev 552237d4a34ab10fa5f9ec7aad7942f2a110993e)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStorePerf.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9556 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9556/ ) YARN-4880 . Running TestZKRMStateStorePerf with real zookeeper cluster (rohithsharmaks: rev 552237d4a34ab10fa5f9ec7aad7942f2a110993e) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStorePerf.java

          People

          • Assignee:
            sunilg Sunil G
            Reporter:
            rohithsharma Rohith Sharma K S
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development