HBase
  1. HBase
  2. HBASE-5212

Fix test TestTableMapReduce against 0.23.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.92.0
    • Fix Version/s: 0.92.1, 0.94.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      As reported by Andrew on the hadoop mailing list, mvn -Dhadoop.profile=23 clean test -Dtest=org.apache.hadoop.hbase.mapreduce.TestTableMapReduce fails on 0.92 branch. There are minor changes to HBase poms required to fix that.

      1. HBASE-5212-v3.patch
        4 kB
        Gregory Chanan
      2. 5212-v2.txt
        3 kB
        Ted Yu
      3. HBASE-5212.patch
        1 kB
        Mahadev konar

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in HBase-0.92-security #90 (See https://builds.apache.org/job/HBase-0.92-security/90/)
          HBASE-5212 Fix test TestTableMapReduce against 0.23 (Ted and Gregory)

          jmhsieh :
          Files :

          • /hbase/branches/0.92/CHANGES.txt
          • /hbase/branches/0.92/pom.xml
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java
          Show
          Hudson added a comment - Integrated in HBase-0.92-security #90 (See https://builds.apache.org/job/HBase-0.92-security/90/ ) HBASE-5212 Fix test TestTableMapReduce against 0.23 (Ted and Gregory) jmhsieh : Files : /hbase/branches/0.92/CHANGES.txt /hbase/branches/0.92/pom.xml /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java
          Hide
          Hudson added a comment -

          Integrated in HBase-0.92 #270 (See https://builds.apache.org/job/HBase-0.92/270/)
          HBASE-5212 Fix test TestTableMapReduce against 0.23 (Ted and Gregory)

          jmhsieh :
          Files :

          • /hbase/branches/0.92/CHANGES.txt
          • /hbase/branches/0.92/pom.xml
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java
          Show
          Hudson added a comment - Integrated in HBase-0.92 #270 (See https://builds.apache.org/job/HBase-0.92/270/ ) HBASE-5212 Fix test TestTableMapReduce against 0.23 (Ted and Gregory) jmhsieh : Files : /hbase/branches/0.92/CHANGES.txt /hbase/branches/0.92/pom.xml /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java
          Hide
          Jonathan Hsieh added a comment -

          I've committed to 0.92 branch as well.

          Show
          Jonathan Hsieh added a comment - I've committed to 0.92 branch as well.
          Hide
          Ted Yu added a comment -

          Go ahead and apply the patch to 0.92

          Thanks

          Show
          Ted Yu added a comment - Go ahead and apply the patch to 0.92 Thanks
          Hide
          Jonathan Hsieh added a comment -

          @Ted - any reason why this isn't applicable to 0.92 as well?

          Show
          Jonathan Hsieh added a comment - @Ted - any reason why this isn't applicable to 0.92 as well?
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #2650 (See https://builds.apache.org/job/HBase-TRUNK/2650/)
          HBASE-5212 Fix test TestTableMapReduce against 0.23 (Ted and Gregory)

          tedyu :
          Files :

          • /hbase/trunk/pom.xml
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #2650 (See https://builds.apache.org/job/HBase-TRUNK/2650/ ) HBASE-5212 Fix test TestTableMapReduce against 0.23 (Ted and Gregory) tedyu : Files : /hbase/trunk/pom.xml /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java
          Hide
          Mahadev konar added a comment -

          Thanks Gregory and Ted for taking care of this!

          Show
          Mahadev konar added a comment - Thanks Gregory and Ted for taking care of this!
          Hide
          Ted Yu added a comment -

          Integrated to TRUNK.

          Thanks for the patch, Mahadev and Gregory.

          Show
          Ted Yu added a comment - Integrated to TRUNK. Thanks for the patch, Mahadev and Gregory.
          Hide
          Ted Yu added a comment -

          @Gregory:
          I agree with your points.

          Will integrate momentarily.

          Show
          Ted Yu added a comment - @Gregory: I agree with your points. Will integrate momentarily.
          Hide
          Gregory Chanan added a comment -

          @Zhihong: good point. It looks like testLogRollOnPipelineRestart is a separate issue, see
          https://issues.apache.org/jira/browse/HDFS-2288
          https://issues.apache.org/jira/browse/HBASE-4254
          Todd has brought that issue back up.

          I think this patch addresses:
          1) HBASE-5212 since it does fix TestTableMapReduce against 0.23
          2) HBASE-5191 since it fixes the compilation issue and causes that test (testLogRollOnDatanodeDeath) to pass against 0.23.

          Also, I ran the above failed unit tests locally with the patch applied (not against 0.23) and they passed.

          Show
          Gregory Chanan added a comment - @Zhihong: good point. It looks like testLogRollOnPipelineRestart is a separate issue, see https://issues.apache.org/jira/browse/HDFS-2288 https://issues.apache.org/jira/browse/HBASE-4254 Todd has brought that issue back up. I think this patch addresses: 1) HBASE-5212 since it does fix TestTableMapReduce against 0.23 2) HBASE-5191 since it fixes the compilation issue and causes that test (testLogRollOnDatanodeDeath) to pass against 0.23. Also, I ran the above failed unit tests locally with the patch applied (not against 0.23) and they passed.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12512511/HBASE-5212-v3.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          -1 javadoc. The javadoc tool appears to have generated -140 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 161 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.io.hfile.TestHFileBlock
          org.apache.hadoop.hbase.mapreduce.TestImportTsv
          org.apache.hadoop.hbase.mapred.TestTableMapReduce
          org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/878//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/878//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/878//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12512511/HBASE-5212-v3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -140 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 161 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.io.hfile.TestHFileBlock org.apache.hadoop.hbase.mapreduce.TestImportTsv org.apache.hadoop.hbase.mapred.TestTableMapReduce org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/878//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/878//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/878//console This message is automatically generated.
          Hide
          Ted Yu added a comment -

          Thanks for the help, Gregory.
          The following command:

          mvn -Dhadoop.profile=23 test -P localTests -Dtest=TestLogRolling
          

          gave me:

          Tests in error: 
            testLogRollOnPipelineRestart(org.apache.hadoop.hbase.regionserver.wal.TestLogRolling): Cannot obtain block length for LocatedBlock{BP-1538911267-10.246.12.139-1327971561136:blk_-5190883400664819989_1014; getBlockSize()=1482; corrupt=false; offset=0; locs=[127.0.0.1:64493, 127.0.0.1:64499]}
          

          TestLogRolling did pass for hadoop 1.0

          Show
          Ted Yu added a comment - Thanks for the help, Gregory. The following command: mvn -Dhadoop.profile=23 test -P localTests -Dtest=TestLogRolling gave me: Tests in error: testLogRollOnPipelineRestart(org.apache.hadoop.hbase.regionserver.wal.TestLogRolling): Cannot obtain block length for LocatedBlock{BP-1538911267-10.246.12.139-1327971561136:blk_-5190883400664819989_1014; getBlockSize()=1482; corrupt= false ; offset=0; locs=[127.0.0.1:64493, 127.0.0.1:64499]} TestLogRolling did pass for hadoop 1.0
          Hide
          Gregory Chanan added a comment -

          This fixes TestTableMapReduce against 0.23 (addresses HBASE-5212) while not breaking TestLogRolling (HBASE-5191).

          Problem with previous patch was it stopped 3 data nodes, rather than stopping the data nodes that existed before the new ones were started.

          Show
          Gregory Chanan added a comment - This fixes TestTableMapReduce against 0.23 (addresses HBASE-5212 ) while not breaking TestLogRolling ( HBASE-5191 ). Problem with previous patch was it stopped 3 data nodes, rather than stopping the data nodes that existed before the new ones were started.
          Hide
          Ted Yu added a comment -

          The patch would break TestLogRolling against hadoop 1.0:

          java.lang.AssertionError: DataNodes 2 default replication 2
          	at org.junit.Assert.fail(Assert.java:93)
          	at org.junit.Assert.assertTrue(Assert.java:43)
          	at org.apache.hadoop.hbase.regionserver.wal.TestLogRolling.testLogRollOnDatanodeDeath(TestLogRolling.java:335)
          

          See HBASE-5191

          Show
          Ted Yu added a comment - The patch would break TestLogRolling against hadoop 1.0: java.lang.AssertionError: DataNodes 2 default replication 2 at org.junit.Assert.fail(Assert.java:93) at org.junit.Assert.assertTrue(Assert.java:43) at org.apache.hadoop.hbase.regionserver.wal.TestLogRolling.testLogRollOnDatanodeDeath(TestLogRolling.java:335) See HBASE-5191
          Hide
          stack added a comment -

          +1 on patch.

          Show
          stack added a comment - +1 on patch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12510879/5212-v2.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          -1 javadoc. The javadoc tool appears to have generated -145 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 82 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.replication.TestReplication
          org.apache.hadoop.hbase.regionserver.wal.TestLogRolling

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/795//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/795//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/795//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12510879/5212-v2.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -145 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 82 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.replication.TestReplication org.apache.hadoop.hbase.regionserver.wal.TestLogRolling Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/795//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/795//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/795//console This message is automatically generated.
          Hide
          Ted Yu added a comment -

          mapreduce.TestTableMapReduce also passes based on hadoop 1.0

          Show
          Ted Yu added a comment - mapreduce.TestTableMapReduce also passes based on hadoop 1.0
          Hide
          Ted Yu added a comment -

          Patch v2 allows TestTableMapReduce to pass based on hadoop 0.23

          Show
          Ted Yu added a comment - Patch v2 allows TestTableMapReduce to pass based on hadoop 0.23
          Hide
          Mahadev konar added a comment -

          @Ted,
          Yup, thats why I marked it as a dependency on this jira.

          Show
          Mahadev konar added a comment - @Ted, Yup, thats why I marked it as a dependency on this jira.
          Hide
          Ted Yu added a comment -

          After fixing the above compilation error, you're likely to see:

          [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.0.2:testCompile (default-testCompile) on project hbase: Compilation failure
          [ERROR] /Users/zhihyu/trunk-hbase/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java:[331,33] cannot find symbol
          [ERROR] symbol  : variable dnRegistration
          [ERROR] location: class org.apache.hadoop.hdfs.server.datanode.DataNode
          

          So there is some overlap between HBASE-5191 and this JIRA.

          Show
          Ted Yu added a comment - After fixing the above compilation error, you're likely to see: [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.0.2:testCompile ( default -testCompile) on project hbase: Compilation failure [ERROR] /Users/zhihyu/trunk-hbase/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java:[331,33] cannot find symbol [ERROR] symbol : variable dnRegistration [ERROR] location: class org.apache.hadoop.hdfs.server.datanode.DataNode So there is some overlap between HBASE-5191 and this JIRA.
          Hide
          Mahadev konar added a comment -

          Looks like trunk is quite different from 0.92. Will upload a patch for trunk.

          Show
          Mahadev konar added a comment - Looks like trunk is quite different from 0.92. Will upload a patch for trunk.
          Hide
          Ted Yu added a comment -
          [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.0.2:testCompile (default-testCompile) on project hbase: Compilation failure
          [ERROR] /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java:[1279,4] cannot find symbol
          [ERROR] symbol  : variable c
          [ERROR] location: class org.apache.hadoop.hbase.HBaseTestingUtility
          
          Show
          Ted Yu added a comment - [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.0.2:testCompile ( default -testCompile) on project hbase: Compilation failure [ERROR] /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java:[1279,4] cannot find symbol [ERROR] symbol : variable c [ERROR] location: class org.apache.hadoop.hbase.HBaseTestingUtility
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12510776/HBASE-5212.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          -1 javadoc. The javadoc tool appears to have generated -145 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 82 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/782//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/782//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/782//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12510776/HBASE-5212.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -145 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 82 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/782//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/782//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/782//console This message is automatically generated.
          Hide
          Mahadev konar added a comment -

          Thanks to Hitesh for helping me out on this. This should fix most of the issue with 0.23 tests.

          Show
          Mahadev konar added a comment - Thanks to Hitesh for helping me out on this. This should fix most of the issue with 0.23 tests.

            People

            • Assignee:
              Gregory Chanan
              Reporter:
              Mahadev konar
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development