Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.95.2
    • Fix Version/s: None
    • Component/s: build, master, regionserver
    • Labels:
      None
    • Release Note:
      Hide
      Fixing TestInfoServers by correcting hbase-server/pom.xml in include the right resources. This then caused an issue that was noticed that tests can be run in other folders if they aren't the right packaging (so the test would still fail in -PlocalTests), so fixed that too.

      When doing the right thing with the resources, it borked the webapp in the packaged deployment, so the webapp classpath addition in bin/hbase had to be fixed.

      It all works locally on my macbook.
      Show
      Fixing TestInfoServers by correcting hbase-server/pom.xml in include the right resources. This then caused an issue that was noticed that tests can be run in other folders if they aren't the right packaging (so the test would still fail in -PlocalTests), so fixed that too. When doing the right thing with the resources, it borked the webapp in the packaged deployment, so the webapp classpath addition in bin/hbase had to be fixed. It all works locally on my macbook.

      Description

      With the recent port to modules, we broke a couple of tests, including this one. The fix needs to ensure that the webapp still works from the in-situ and packaged running of HBase.

      1. hbase-6110-v3.patch
        8 kB
        Jesse Yates
      2. hbase-6110-v2.patch
        8 kB
        Jesse Yates
      3. hbase-6110-v1.patch
        5 kB
        Jesse Yates
      4. hbase-6110-v0.patch
        5 kB
        Jesse Yates

        Activity

        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #1757 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1757/)
        HBASE-6110 adding more slow action log in critical write path (Liang Xie via stack) (stack: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1597633)

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
        Show
        Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #1757 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1757/ ) HBASE-6110 adding more slow action log in critical write path (Liang Xie via stack) (stack: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1597633 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #1783 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1783/)
        HBASE-6110 adding more slow action log in critical write path (Liang Xie via stack) (stack: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1597633)

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
        Show
        Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1783 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1783/ ) HBASE-6110 adding more slow action log in critical write path (Liang Xie via stack) (stack: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1597633 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #565 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/565/)
        HBASE-6110 adding more slow action log in critical write path (Liang Xie via stack) (stack: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1597633)

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
        Show
        Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #565 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/565/ ) HBASE-6110 adding more slow action log in critical write path (Liang Xie via stack) (stack: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1597633 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hadoop-trunk-Commit #5610 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5610/)
        HBASE-6110 adding more slow action log in critical write path (Liang Xie via stack) (stack: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1597633)

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
        Show
        Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #5610 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5610/ ) HBASE-6110 adding more slow action log in critical write path (Liang Xie via stack) (stack: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1597633 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
        Hide
        Jesse Yates added a comment -

        Committed to trunk.

        Show
        Jesse Yates added a comment - Committed to trunk.
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #28 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/28/)
        HBASE-6110 Fix TestInfoServers, patch v3 (Jesse) (Revision 1343110)

        Result = FAILURE
        tedyu :
        Files :

        • /hbase/trunk/bin/hbase
        • /hbase/trunk/hbase-assembly/pom.xml
        • /hbase/trunk/hbase-server/pom.xml
        • /hbase/trunk/hbase-server/src/main/resources/hbase-default.xml
        • /hbase/trunk/hbase-site/pom.xml
        • /hbase/trunk/pom.xml
        Show
        Hudson added a comment - Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #28 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/28/ ) HBASE-6110 Fix TestInfoServers, patch v3 (Jesse) (Revision 1343110) Result = FAILURE tedyu : Files : /hbase/trunk/bin/hbase /hbase/trunk/hbase-assembly/pom.xml /hbase/trunk/hbase-server/pom.xml /hbase/trunk/hbase-server/src/main/resources/hbase-default.xml /hbase/trunk/hbase-site/pom.xml /hbase/trunk/pom.xml
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #2935 (See https://builds.apache.org/job/HBase-TRUNK/2935/)
        HBASE-6110 Fix TestInfoServers, patch v3 (Jesse) (Revision 1343110)

        Result = FAILURE
        tedyu :
        Files :

        • /hbase/trunk/bin/hbase
        • /hbase/trunk/hbase-assembly/pom.xml
        • /hbase/trunk/hbase-server/pom.xml
        • /hbase/trunk/hbase-server/src/main/resources/hbase-default.xml
        • /hbase/trunk/hbase-site/pom.xml
        • /hbase/trunk/pom.xml
        Show
        Hudson added a comment - Integrated in HBase-TRUNK #2935 (See https://builds.apache.org/job/HBase-TRUNK/2935/ ) HBASE-6110 Fix TestInfoServers, patch v3 (Jesse) (Revision 1343110) Result = FAILURE tedyu : Files : /hbase/trunk/bin/hbase /hbase/trunk/hbase-assembly/pom.xml /hbase/trunk/hbase-server/pom.xml /hbase/trunk/hbase-server/src/main/resources/hbase-default.xml /hbase/trunk/hbase-site/pom.xml /hbase/trunk/pom.xml
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12529909/hbase-6110-v3.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 7 new or modified tests.

        +1 hadoop23. The patch compiles against the hadoop 0.23.x profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.io.hfile.TestForceCacheImportantBlocks
        org.apache.hadoop.hbase.master.TestSplitLogManager

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2019//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2019//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12529909/hbase-6110-v3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 7 new or modified tests. +1 hadoop23. The patch compiles against the hadoop 0.23.x profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.io.hfile.TestForceCacheImportantBlocks org.apache.hadoop.hbase.master.TestSplitLogManager Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2019//testReport/ Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2019//console This message is automatically generated.
        Hide
        Ted Yu added a comment -

        I ran the following command:

        mvn -P runAllTests -Dtest=TestClusterBootOrder,TestHBaseTestingUtility,TestHFileReaderV1,TestMultiVersions -DfailIfNoTests=false clean test
        

        the tests passed.

        Patch v3 integrated to trunk.

        Thanks for your effort, Jesse.

        Show
        Ted Yu added a comment - I ran the following command: mvn -P runAllTests -Dtest=TestClusterBootOrder,TestHBaseTestingUtility,TestHFileReaderV1,TestMultiVersions -DfailIfNoTests= false clean test the tests passed. Patch v3 integrated to trunk. Thanks for your effort, Jesse.
        Hide
        Jesse Yates added a comment -

        Attaching version that imports all the test resources. Fixes TestHFileV1 and still has TestInfoServers passing locally.

        Show
        Jesse Yates added a comment - Attaching version that imports all the test resources. Fixes TestHFileV1 and still has TestInfoServers passing locally.
        Hide
        Jesse Yates added a comment -

        hmm, I wonder what happens if we just include the whole directory. Let me try one more time.

        Show
        Jesse Yates added a comment - hmm, I wonder what happens if we just include the whole directory. Let me try one more time.
        Hide
        Ted Yu added a comment -

        Looks like Hadoop QA is working again.
        I saw the following after applying patch v2:

          <testcase time="0.002" classname="org.apache.hadoop.hbase.io.hfile.TestHFileReaderV1" name="testReadingExistingVersion1HFile">
            <error type="java.lang.NullPointerException">java.lang.NullPointerException
          at org.apache.hadoop.hbase.io.hfile.TestHFileReaderV1.testReadingExistingVersion1HFile(TestHFileReaderV1.java:71)
        

        The test couldn't find hbase-server/src/test/resources/org/apache/hadoop/hbase/io/hfile/8e8ab58dcf39412da19833fcd8f687ac

        Show
        Ted Yu added a comment - Looks like Hadoop QA is working again. I saw the following after applying patch v2: <testcase time= "0.002" classname= "org.apache.hadoop.hbase.io.hfile.TestHFileReaderV1" name= "testReadingExistingVersion1HFile" > <error type= "java.lang.NullPointerException" >java.lang.NullPointerException at org.apache.hadoop.hbase.io.hfile.TestHFileReaderV1.testReadingExistingVersion1HFile(TestHFileReaderV1.java:71) The test couldn't find hbase-server/src/test/resources/org/apache/hadoop/hbase/io/hfile/8e8ab58dcf39412da19833fcd8f687ac
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12529906/hbase-6110-v2.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 7 new or modified tests.

        +1 hadoop23. The patch compiles against the hadoop 0.23.x profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.io.hfile.TestHFileReaderV1

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2018//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2018//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12529906/hbase-6110-v2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 7 new or modified tests. +1 hadoop23. The patch compiles against the hadoop 0.23.x profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.io.hfile.TestHFileReaderV1 Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2018//testReport/ Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2018//console This message is automatically generated.
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #2934 (See https://builds.apache.org/job/HBase-TRUNK/2934/)
        HBASE-6110 Revert due to increased number of failed tests (Revision 1343081)

        Result = FAILURE
        tedyu :
        Files :

        • /hbase/trunk/bin/hbase
        • /hbase/trunk/hbase-assembly/pom.xml
        • /hbase/trunk/hbase-server/pom.xml
        • /hbase/trunk/hbase-site/pom.xml
        • /hbase/trunk/pom.xml
        Show
        Hudson added a comment - Integrated in HBase-TRUNK #2934 (See https://builds.apache.org/job/HBase-TRUNK/2934/ ) HBASE-6110 Revert due to increased number of failed tests (Revision 1343081) Result = FAILURE tedyu : Files : /hbase/trunk/bin/hbase /hbase/trunk/hbase-assembly/pom.xml /hbase/trunk/hbase-server/pom.xml /hbase/trunk/hbase-site/pom.xml /hbase/trunk/pom.xml
        Hide
        Jesse Yates added a comment -

        Attaching new patch. Does everything in v1, plus fixes the resource processing such that:

        • src/test/resources/hbase-site.xml is included
        • @@@VERSION@@@ is processed on inclusion, rather than in a separate step (this came up in testing the fix)
        Show
        Jesse Yates added a comment - Attaching new patch. Does everything in v1, plus fixes the resource processing such that: src/test/resources/hbase-site.xml is included @@@VERSION@@@ is processed on inclusion, rather than in a separate step (this came up in testing the fix)
        Hide
        Ted Yu added a comment -

        Confirmed with Jesse that we roll back the patch until we have clear idea with packaging.

        Reverted in trunk.

        Show
        Ted Yu added a comment - Confirmed with Jesse that we roll back the patch until we have clear idea with packaging. Reverted in trunk.
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #24 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/24/)
        HBASE-6110 Revert due to increased number of failed tests (Revision 1343081)

        Result = FAILURE
        tedyu :
        Files :

        • /hbase/trunk/bin/hbase
        • /hbase/trunk/hbase-assembly/pom.xml
        • /hbase/trunk/hbase-server/pom.xml
        • /hbase/trunk/hbase-site/pom.xml
        • /hbase/trunk/pom.xml
        Show
        Hudson added a comment - Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #24 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/24/ ) HBASE-6110 Revert due to increased number of failed tests (Revision 1343081) Result = FAILURE tedyu : Files : /hbase/trunk/bin/hbase /hbase/trunk/hbase-assembly/pom.xml /hbase/trunk/hbase-server/pom.xml /hbase/trunk/hbase-site/pom.xml /hbase/trunk/pom.xml
        Hide
        Jesse Yates added a comment -

        @stack: yeah, that's the case now. My recent patch here should have fixed the bin/hbase running to use the right classpath. Something has gone wonky now with the classpath in maven.

        Show
        Jesse Yates added a comment - @stack: yeah, that's the case now. My recent patch here should have fixed the bin/hbase running to use the right classpath. Something has gone wonky now with the classpath in maven.
        Hide
        stack added a comment -

        On hbase-webapps, in tgz, we used have hbase-webapps that the top-level beside the hbase-*.jar and beside LICENSE and CHANGES. That still happening?

        Show
        stack added a comment - On hbase-webapps, in tgz, we used have hbase-webapps that the top-level beside the hbase-*.jar and beside LICENSE and CHANGES. That still happening?
        Hide
        stack added a comment -

        I feel like hbasetestutility's minicluster should have the webui off by default, unless otherwise desired

        It used to be off. The way we turned it off was by putting the test hbase-site.xml in the classpath ahead of all other configs: i.e. src/test folder. That must not be the case now after modules.

        Show
        stack added a comment - I feel like hbasetestutility's minicluster should have the webui off by default, unless otherwise desired It used to be off. The way we turned it off was by putting the test hbase-site.xml in the classpath ahead of all other configs: i.e. src/test folder. That must not be the case now after modules.
        Hide
        Jesse Yates added a comment -

        Back to the real issue, it seems like a concurrency bug, like surefire isn't doing the right then when parallelizing the tests. There is the fork factor of 4 (perThread and then threads set to 4). Originally, this was done in HBASE-5064; I'm surprised that we didn't break things there. Conceptually, the parallelization shouldn't work unless we get consistently lucky with test order - the minicluster will have problems if it can't find the right port, which it shouldn't if another jvm is also running a mini-cluster.

        The this wouldn't be a big issue except that the master goes down if the http server port is already taken:

        java.net.BindException: Address already in use
        at sun.nio.ch.Net.bind(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:126)
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
        at org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216)
        at org.apache.hadoop.http.HttpServer.start(HttpServer.java:581)
        at org.apache.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:929)
        at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:585)
        at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:386)
        at java.lang.Thread.run(Thread.java:680)

        Should this be the case? I feel like hbasetestutility's minicluster should have the webui off by default, unless otherwise desired (I tend to turn it off in the tests I write, just to speed things up). Also, if the webui is taken, it should just bump the port number to the next available.

        Thoughts?

        Show
        Jesse Yates added a comment - Back to the real issue, it seems like a concurrency bug, like surefire isn't doing the right then when parallelizing the tests. There is the fork factor of 4 (perThread and then threads set to 4). Originally, this was done in HBASE-5064 ; I'm surprised that we didn't break things there. Conceptually, the parallelization shouldn't work unless we get consistently lucky with test order - the minicluster will have problems if it can't find the right port, which it shouldn't if another jvm is also running a mini-cluster. The this wouldn't be a big issue except that the master goes down if the http server port is already taken: java.net.BindException: Address already in use at sun.nio.ch.Net.bind(Native Method) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:126) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59) at org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216) at org.apache.hadoop.http.HttpServer.start(HttpServer.java:581) at org.apache.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:929) at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:585) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:386) at java.lang.Thread.run(Thread.java:680) Should this be the case? I feel like hbasetestutility's minicluster should have the webui off by default, unless otherwise desired (I tend to turn it off in the tests I write, just to speed things up). Also, if the webui is taken, it should just bump the port number to the next available. Thoughts?
        Hide
        Jesse Yates added a comment -

        @Ted: yeah, that's my bad. Hadoop version bump. Filed HBASE-6112

        Show
        Jesse Yates added a comment - @Ted: yeah, that's my bad. Hadoop version bump. Filed HBASE-6112
        Hide
        Ted Yu added a comment -

        I tried '-Dhadoop.profile=2.0' (build #21) and '-Dhadoop.version=2.0' (build #22)
        For build #22, I saw:

        [ERROR] Failed to execute goal on project hbase-server: Could not resolve dependencies for project org.apache.hbase:hbase-server:jar:0.95-SNAPSHOT: The following artifacts could not be resolved: org.apache.hadoop:hadoop-core:jar:2.0, org.apache.hadoop:hadoop-test:jar:2.0: Could not find artifact org.apache.hadoop:hadoop-core:jar:2.0 in apache release (https://repository.apache.org/content/repositories/releases/) -> [Help 1]
        
        Show
        Ted Yu added a comment - I tried '-Dhadoop.profile=2.0' (build #21) and '-Dhadoop.version=2.0' (build #22) For build #22, I saw: [ERROR] Failed to execute goal on project hbase-server: Could not resolve dependencies for project org.apache.hbase:hbase-server:jar:0.95-SNAPSHOT: The following artifacts could not be resolved: org.apache.hadoop:hadoop-core:jar:2.0, org.apache.hadoop:hadoop-test:jar:2.0: Could not find artifact org.apache.hadoop:hadoop-core:jar:2.0 in apache release (https: //repository.apache.org/content/repositories/releases/) -> [Help 1]
        Hide
        Jesse Yates added a comment -

        As an aside, the Hadoop-2.0 build is using the wrong command and that's why its failing.

        -Dhadoop.version=2.0.0-SNAPSHOT

        should be

        -Dhadoop.version=2.0
        

        that should get that build running at least.

        Show
        Jesse Yates added a comment - As an aside, the Hadoop-2.0 build is using the wrong command and that's why its failing. -Dhadoop.version=2.0.0-SNAPSHOT should be -Dhadoop.version=2.0 that should get that build running at least.
        Hide
        Jesse Yates added a comment -

        I get this issue sometimes on my local box when doing testing in eclipse - for some reason the previous cluster hasn't completely come down, causing the new one to be upset because the port is in use. I'll look into seeing if I can reproduce.

        Show
        Jesse Yates added a comment - I get this issue sometimes on my local box when doing testing in eclipse - for some reason the previous cluster hasn't completely come down, causing the new one to be upset because the port is in use. I'll look into seeing if I can reproduce.
        Hide
        Ted Yu added a comment -

        From https://builds.apache.org/view/G-L/view/HBase/job/HBase-TRUNK/2932/testReport/junit/org.apache.hadoop.hbase/TestClusterBootOrder/testBootRegionServerFirst/:

        2012-05-27 06:51:36,454 FATAL [Master:0;juno.apache.org,49080,1338101493265] master.HMaster(1605): Master server abort: loaded coprocessors are: []
        2012-05-27 06:51:36,454 FATAL [Master:0;juno.apache.org,49080,1338101493265] master.HMaster(1610): Unhandled exception. Starting shutdown.
        java.net.BindException: Address already in use
        	at sun.nio.ch.Net.bind(Native Method)
        	at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:126)
        	at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
        	at org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216)
        	at org.apache.hadoop.http.HttpServer.start(HttpServer.java:581)
        	at org.apache.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:929)
        	at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:585)
        	at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:386)
        	at java.lang.Thread.run(Thread.java:662)
        ...
        2012-05-27 06:51:51,339 WARN  [pool-1-thread-1] zookeeper.RecoverableZooKeeper(218): Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/shutdown
        2012-05-27 06:51:51,340 ERROR [pool-1-thread-1] zookeeper.RecoverableZooKeeper(220): ZooKeeper delete failed after 3 retries
        2012-05-27 06:51:51,340 ERROR [pool-1-thread-1] master.HMaster(1753): ZooKeeper exception trying to set cluster as down in ZK
        org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/shutdown
        	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
        	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        	at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:868)
        	at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:132)
        	at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:974)
        	at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:963)
        	at org.apache.hadoop.hbase.zookeeper.ClusterStatusTracker.setClusterDown(ClusterStatusTracker.java:84)
        	at org.apache.hadoop.hbase.master.HMaster.shutdown(HMaster.java:1750)
        	at org.apache.hadoop.hbase.util.JVMClusterUtil.shutdown(JVMClusterUtil.java:236)
        	at org.apache.hadoop.hbase.LocalHBaseCluster.shutdown(LocalHBaseCluster.java:431)
        	at org.apache.hadoop.hbase.TestClusterBootOrder.tearDown(TestClusterBootOrder.java:55)
        

        From https://builds.apache.org/view/G-L/view/HBase/job/HBase-TRUNK/2932/artifact/trunk/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.TestMultiVersions.txt:

        Caused by: java.lang.RuntimeException: Master not initialized after 200 seconds
        	at org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:208)
        	at org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:424)
        	at org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:204)
        

        I couldn't reproduce the above failure running TestMultiVersions standalone. There is no test output for TestMultiVersions under https://builds.apache.org/view/G-L/view/HBase/job/HBase-TRUNK/2932/artifact/trunk/hbase-server/target/surefire-reports/

        Show
        Ted Yu added a comment - From https://builds.apache.org/view/G-L/view/HBase/job/HBase-TRUNK/2932/testReport/junit/org.apache.hadoop.hbase/TestClusterBootOrder/testBootRegionServerFirst/: 2012-05-27 06:51:36,454 FATAL [Master:0;juno.apache.org,49080,1338101493265] master.HMaster(1605): Master server abort: loaded coprocessors are: [] 2012-05-27 06:51:36,454 FATAL [Master:0;juno.apache.org,49080,1338101493265] master.HMaster(1610): Unhandled exception. Starting shutdown. java.net.BindException: Address already in use at sun.nio.ch.Net.bind(Native Method) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:126) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59) at org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216) at org.apache.hadoop.http.HttpServer.start(HttpServer.java:581) at org.apache.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:929) at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:585) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:386) at java.lang. Thread .run( Thread .java:662) ... 2012-05-27 06:51:51,339 WARN [pool-1-thread-1] zookeeper.RecoverableZooKeeper(218): Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/shutdown 2012-05-27 06:51:51,340 ERROR [pool-1-thread-1] zookeeper.RecoverableZooKeeper(220): ZooKeeper delete failed after 3 retries 2012-05-27 06:51:51,340 ERROR [pool-1-thread-1] master.HMaster(1753): ZooKeeper exception trying to set cluster as down in ZK org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/shutdown at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:868) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:132) at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:974) at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:963) at org.apache.hadoop.hbase.zookeeper.ClusterStatusTracker.setClusterDown(ClusterStatusTracker.java:84) at org.apache.hadoop.hbase.master.HMaster.shutdown(HMaster.java:1750) at org.apache.hadoop.hbase.util.JVMClusterUtil.shutdown(JVMClusterUtil.java:236) at org.apache.hadoop.hbase.LocalHBaseCluster.shutdown(LocalHBaseCluster.java:431) at org.apache.hadoop.hbase.TestClusterBootOrder.tearDown(TestClusterBootOrder.java:55) From https://builds.apache.org/view/G-L/view/HBase/job/HBase-TRUNK/2932/artifact/trunk/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.TestMultiVersions.txt: Caused by: java.lang.RuntimeException: Master not initialized after 200 seconds at org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:208) at org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:424) at org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:204) I couldn't reproduce the above failure running TestMultiVersions standalone. There is no test output for TestMultiVersions under https://builds.apache.org/view/G-L/view/HBase/job/HBase-TRUNK/2932/artifact/trunk/hbase-server/target/surefire-reports/
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #19 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/19/)
        HBASE-6110 Fix TestInfoServers (Jesse Yates) (Revision 1342995)

        Result = FAILURE
        tedyu :
        Files :

        • /hbase/trunk/bin/hbase
        • /hbase/trunk/hbase-assembly/pom.xml
        • /hbase/trunk/hbase-server/pom.xml
        • /hbase/trunk/hbase-site/pom.xml
        • /hbase/trunk/pom.xml
        Show
        Hudson added a comment - Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #19 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/19/ ) HBASE-6110 Fix TestInfoServers (Jesse Yates) (Revision 1342995) Result = FAILURE tedyu : Files : /hbase/trunk/bin/hbase /hbase/trunk/hbase-assembly/pom.xml /hbase/trunk/hbase-server/pom.xml /hbase/trunk/hbase-site/pom.xml /hbase/trunk/pom.xml
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #2932 (See https://builds.apache.org/job/HBase-TRUNK/2932/)
        HBASE-6110 Fix TestInfoServers (Jesse Yates) (Revision 1342995)

        Result = FAILURE
        tedyu :
        Files :

        • /hbase/trunk/bin/hbase
        • /hbase/trunk/hbase-assembly/pom.xml
        • /hbase/trunk/hbase-server/pom.xml
        • /hbase/trunk/hbase-site/pom.xml
        • /hbase/trunk/pom.xml
        Show
        Hudson added a comment - Integrated in HBase-TRUNK #2932 (See https://builds.apache.org/job/HBase-TRUNK/2932/ ) HBASE-6110 Fix TestInfoServers (Jesse Yates) (Revision 1342995) Result = FAILURE tedyu : Files : /hbase/trunk/bin/hbase /hbase/trunk/hbase-assembly/pom.xml /hbase/trunk/hbase-server/pom.xml /hbase/trunk/hbase-site/pom.xml /hbase/trunk/pom.xml
        Hide
        Ted Yu added a comment -

        I tried patch v1 locally and TestInfoServers passed.

        Patch integrated to trunk.

        Thanks for the patch, Jesse.
        Let's see if next trunk build passes.

        Show
        Ted Yu added a comment - I tried patch v1 locally and TestInfoServers passed. Patch integrated to trunk. Thanks for the patch, Jesse. Let's see if next trunk build passes.
        Hide
        Jesse Yates added a comment -

        submitting patch for hadoopqa

        Show
        Jesse Yates added a comment - submitting patch for hadoopqa
        Hide
        Jesse Yates added a comment -

        patch works for both insitu and packaged deployment. Problem was classpath issues in bin/hbase when looking for the webapps folder.

        Don't see a good way of unit testing this. Fortunately, its pretty obvious when its borked. And since its fixed now, we don't need to worry anymore (sarcasm)

        Show
        Jesse Yates added a comment - patch works for both insitu and packaged deployment. Problem was classpath issues in bin/hbase when looking for the webapps folder. Don't see a good way of unit testing this. Fortunately, its pretty obvious when its borked. And since its fixed now, we don't need to worry anymore (sarcasm)
        Hide
        Jesse Yates added a comment -

        Test passes locally with this patch and the in-situ seems to work fine. Seems like there is a classpath issue that I can't seem to figure out when running from the packaged version (obtained via 'mvn clean package -DskipTests').

        Show
        Jesse Yates added a comment - Test passes locally with this patch and the in-situ seems to work fine. Seems like there is a classpath issue that I can't seem to figure out when running from the packaged version (obtained via 'mvn clean package -DskipTests').

          People

          • Assignee:
            Jesse Yates
            Reporter:
            Jesse Yates
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development