Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-8146

IntegrationTestBigLinkedList does not work on distributed setup

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.98.0, 0.95.0
    • Fix Version/s: 0.98.0, 0.94.7, 0.95.0
    • Component/s: test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Two issues:

      • IntegrationTestBigLinkedList.java does not add dependency jars, for it's first job (my bad for not testing it properly)
      • TableMapReduceUtil does not add guava, and htrace dependency to the job classpath. This affects all mapreduce jobs in trunk.

      I have to look into whether the patch for HBASE-8140 would fix the second condition.

      1. hbase-8146_v1-0.94.patch
        3 kB
        Enis Soztutar
      2. hbase-8146_v1.patch
        3 kB
        Enis Soztutar

        Issue Links

          Activity

          Hide
          enis Enis Soztutar added a comment -

          Simple patch.

          Show
          enis Enis Soztutar added a comment - Simple patch.
          Hide
          stack stack added a comment -

          +1 if it works on your cluster Enis (maybe wait on hadoopqa just in case)

          Show
          stack stack added a comment - +1 if it works on your cluster Enis (maybe wait on hadoopqa just in case)
          Hide
          stack stack added a comment -

          +1 on backport to 0.95 if this test is in 0.95.

          Show
          stack stack added a comment - +1 on backport to 0.95 if this test is in 0.95.
          Hide
          enis Enis Soztutar added a comment -

          Thanks for the quick look. We've discovered with Nick offline that the reason we were able to get away for the mapred classpath issues was that most of the mapred utils (importtsv, etc) was adding those classes manually.
          Will commit if qa gives a go. Will also fix for 0.94, sans the TableMapReduceUtil changes.

          Show
          enis Enis Soztutar added a comment - Thanks for the quick look. We've discovered with Nick offline that the reason we were able to get away for the mapred classpath issues was that most of the mapred utils (importtsv, etc) was adding those classes manually. Will commit if qa gives a go. Will also fix for 0.94, sans the TableMapReduceUtil changes.
          Hide
          ndimiduk Nick Dimiduk added a comment -

          With this patch, you can remove the line in ImportTsv that explicitly adds guava as a dependency for that job.

          Are there any other dependencies we're not shipping?

          Show
          ndimiduk Nick Dimiduk added a comment - With this patch, you can remove the line in ImportTsv that explicitly adds guava as a dependency for that job. Are there any other dependencies we're not shipping?
          Hide
          enis Enis Soztutar added a comment -

          Are there any other dependencies we're not shipping?

          mvn dependency:tree
          

          lists these dependencies for hbase-client:

          INFO] --- maven-dependency-plugin:2.1:tree (default-cli) @ hbase-client ---
          [INFO] org.apache.hbase:hbase-client:jar:0.97-SNAPSHOT
          [INFO] +- org.apache.hbase:hbase-common:jar:0.97-SNAPSHOT:compile
          [INFO] |  +- com.google.guava:guava:jar:12.0.1:compile
          [INFO] |  |  \- com.google.code.findbugs:jsr305:jar:1.3.9:compile
          [INFO] |  +- commons-logging:commons-logging:jar:1.1.1:compile
          [INFO] |  +- org.slf4j:slf4j-api:jar:1.4.3:compile
          [INFO] |  \- org.slf4j:slf4j-log4j12:jar:1.4.3:compile
          [INFO] +- org.apache.hbase:hbase-common:test-jar:tests:0.97-SNAPSHOT:test
          [INFO] +- org.apache.hbase:hbase-protocol:jar:0.97-SNAPSHOT:compile
          [INFO] +- com.google.protobuf:protobuf-java:jar:2.4.1:compile
          [INFO] +- org.apache.zookeeper:zookeeper:jar:3.4.5:compile
          [INFO] |  \- log4j:log4j:jar:1.2.17:compile
          [INFO] +- org.cloudera.htrace:htrace:jar:1.50:compile
          [INFO] |  \- org.mortbay.jetty:jetty-util:jar:6.1.26:compile
          [INFO] +- org.apache.hadoop:hadoop-core:jar:1.1.1:compile
          [INFO] |  +- commons-cli:commons-cli:jar:1.2:compile
          [INFO] |  +- xmlenc:xmlenc:jar:0.52:compile
          [INFO] |  +- commons-io:commons-io:jar:2.4:compile (version managed from 2.1)
          [INFO] |  +- commons-httpclient:commons-httpclient:jar:3.0.1:compile
          [INFO] |  +- commons-codec:commons-codec:jar:1.7:compile
          [INFO] |  +- org.apache.commons:commons-math:jar:2.1:compile
          [INFO] |  +- commons-configuration:commons-configuration:jar:1.6:compile
          [INFO] |  |  +- commons-collections:commons-collections:jar:3.2.1:compile
          [INFO] |  |  +- commons-lang:commons-lang:jar:2.6:compile (version managed from 2.4)
          [INFO] |  |  +- commons-digester:commons-digester:jar:1.8:compile
          [INFO] |  |  |  \- commons-beanutils:commons-beanutils:jar:1.7.0:compile
          [INFO] |  |  \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
          [INFO] |  +- commons-net:commons-net:jar:1.4.1:compile
          [INFO] |  +- org.mortbay.jetty:jetty:jar:6.1.26:compile
          [INFO] |  +- tomcat:jasper-runtime:jar:5.5.23:runtime (version managed from 5.5.12; scope managed from compile)
          [INFO] |  +- tomcat:jasper-compiler:jar:5.5.23:runtime (version managed from 5.5.12; scope managed from compile)
          [INFO] |  +- org.mortbay.jetty:jsp-api-2.1:jar:6.1.14:compile
          [INFO] |  |  \- org.mortbay.jetty:servlet-api-2.5:jar:6.1.14:compile
          [INFO] |  +- org.mortbay.jetty:jsp-2.1:jar:6.1.14:compile
          [INFO] |  +- commons-el:commons-el:jar:1.0:compile
          [INFO] |  \- org.codehaus.jackson:jackson-mapper-asl:jar:1.8.8:compile
          [INFO] |     \- org.codehaus.jackson:jackson-core-asl:jar:1.8.8:compile
          [INFO] +- com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:compile
          [INFO] +- junit:junit:jar:4.11:test
          [INFO] |  \- org.hamcrest:hamcrest-core:jar:1.3:test
          [INFO] \- org.mockito:mockito-all:jar:1.9.0:test
          

          Mainly, it seems we are depending on guava, htrace, protobuf, findbugs. I am guessing findbugs is not a runtime dependency(nope not). We should be good to go.

          Show
          enis Enis Soztutar added a comment - Are there any other dependencies we're not shipping? mvn dependency:tree lists these dependencies for hbase-client: INFO] --- maven-dependency-plugin:2.1:tree ( default -cli) @ hbase-client --- [INFO] org.apache.hbase:hbase-client:jar:0.97-SNAPSHOT [INFO] +- org.apache.hbase:hbase-common:jar:0.97-SNAPSHOT:compile [INFO] | +- com.google.guava:guava:jar:12.0.1:compile [INFO] | | \- com.google.code.findbugs:jsr305:jar:1.3.9:compile [INFO] | +- commons-logging:commons-logging:jar:1.1.1:compile [INFO] | +- org.slf4j:slf4j-api:jar:1.4.3:compile [INFO] | \- org.slf4j:slf4j-log4j12:jar:1.4.3:compile [INFO] +- org.apache.hbase:hbase-common:test-jar:tests:0.97-SNAPSHOT:test [INFO] +- org.apache.hbase:hbase-protocol:jar:0.97-SNAPSHOT:compile [INFO] +- com.google.protobuf:protobuf-java:jar:2.4.1:compile [INFO] +- org.apache.zookeeper:zookeeper:jar:3.4.5:compile [INFO] | \- log4j:log4j:jar:1.2.17:compile [INFO] +- org.cloudera.htrace:htrace:jar:1.50:compile [INFO] | \- org.mortbay.jetty:jetty-util:jar:6.1.26:compile [INFO] +- org.apache.hadoop:hadoop-core:jar:1.1.1:compile [INFO] | +- commons-cli:commons-cli:jar:1.2:compile [INFO] | +- xmlenc:xmlenc:jar:0.52:compile [INFO] | +- commons-io:commons-io:jar:2.4:compile (version managed from 2.1) [INFO] | +- commons-httpclient:commons-httpclient:jar:3.0.1:compile [INFO] | +- commons-codec:commons-codec:jar:1.7:compile [INFO] | +- org.apache.commons:commons-math:jar:2.1:compile [INFO] | +- commons-configuration:commons-configuration:jar:1.6:compile [INFO] | | +- commons-collections:commons-collections:jar:3.2.1:compile [INFO] | | +- commons-lang:commons-lang:jar:2.6:compile (version managed from 2.4) [INFO] | | +- commons-digester:commons-digester:jar:1.8:compile [INFO] | | | \- commons-beanutils:commons-beanutils:jar:1.7.0:compile [INFO] | | \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile [INFO] | +- commons-net:commons-net:jar:1.4.1:compile [INFO] | +- org.mortbay.jetty:jetty:jar:6.1.26:compile [INFO] | +- tomcat:jasper-runtime:jar:5.5.23:runtime (version managed from 5.5.12; scope managed from compile) [INFO] | +- tomcat:jasper-compiler:jar:5.5.23:runtime (version managed from 5.5.12; scope managed from compile) [INFO] | +- org.mortbay.jetty:jsp-api-2.1:jar:6.1.14:compile [INFO] | | \- org.mortbay.jetty:servlet-api-2.5:jar:6.1.14:compile [INFO] | +- org.mortbay.jetty:jsp-2.1:jar:6.1.14:compile [INFO] | +- commons-el:commons-el:jar:1.0:compile [INFO] | \- org.codehaus.jackson:jackson-mapper-asl:jar:1.8.8:compile [INFO] | \- org.codehaus.jackson:jackson-core-asl:jar:1.8.8:compile [INFO] +- com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:compile [INFO] +- junit:junit:jar:4.11:test [INFO] | \- org.hamcrest:hamcrest-core:jar:1.3:test [INFO] \- org.mockito:mockito-all:jar:1.9.0:test Mainly, it seems we are depending on guava, htrace, protobuf, findbugs. I am guessing findbugs is not a runtime dependency(nope not). We should be good to go.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12574411/hbase-8146_v1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          -1 site. The patch appears to cause mvn site goal to fail.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4897//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4897//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4897//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4897//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4897//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4897//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4897//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4897//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4897//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4897//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12574411/hbase-8146_v1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 site . The patch appears to cause mvn site goal to fail. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4897//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4897//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4897//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4897//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4897//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4897//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4897//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4897//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4897//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4897//console This message is automatically generated.
          Hide
          enis Enis Soztutar added a comment -

          Committed this to 0.95 and 0.98. Thanks for reviews.

          0.94 patch coming shortly, but I have to test it first.

          Show
          enis Enis Soztutar added a comment - Committed this to 0.95 and 0.98. Thanks for reviews. 0.94 patch coming shortly, but I have to test it first.
          Hide
          hudson Hudson added a comment -

          Integrated in hbase-0.95-on-hadoop2 #34 (See https://builds.apache.org/job/hbase-0.95-on-hadoop2/34/)
          HBASE-8146 IntegrationTestBigLinkedList does not work on distributed setup (Revision 1458586)

          Result = FAILURE
          enis :
          Files :

          • /hbase/branches/0.95/hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
          Show
          hudson Hudson added a comment - Integrated in hbase-0.95-on-hadoop2 #34 (See https://builds.apache.org/job/hbase-0.95-on-hadoop2/34/ ) HBASE-8146 IntegrationTestBigLinkedList does not work on distributed setup (Revision 1458586) Result = FAILURE enis : Files : /hbase/branches/0.95/hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
          Hide
          hudson Hudson added a comment -

          Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #455 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/455/)
          HBASE-8146 IntegrationTestBigLinkedList does not work on distributed setup (Revision 1458585)

          Result = FAILURE
          enis :
          Files :

          • /hbase/trunk/hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
          Show
          hudson Hudson added a comment - Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #455 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/455/ ) HBASE-8146 IntegrationTestBigLinkedList does not work on distributed setup (Revision 1458585) Result = FAILURE enis : Files : /hbase/trunk/hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
          Hide
          hudson Hudson added a comment -

          Integrated in hbase-0.95 #89 (See https://builds.apache.org/job/hbase-0.95/89/)
          HBASE-8146 IntegrationTestBigLinkedList does not work on distributed setup (Revision 1458586)

          Result = SUCCESS
          enis :
          Files :

          • /hbase/branches/0.95/hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
          Show
          hudson Hudson added a comment - Integrated in hbase-0.95 #89 (See https://builds.apache.org/job/hbase-0.95/89/ ) HBASE-8146 IntegrationTestBigLinkedList does not work on distributed setup (Revision 1458586) Result = SUCCESS enis : Files : /hbase/branches/0.95/hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
          Hide
          hudson Hudson added a comment -

          Integrated in HBase-TRUNK #3976 (See https://builds.apache.org/job/HBase-TRUNK/3976/)
          HBASE-8146 IntegrationTestBigLinkedList does not work on distributed setup (Revision 1458585)

          Result = FAILURE
          enis :
          Files :

          • /hbase/trunk/hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
          Show
          hudson Hudson added a comment - Integrated in HBase-TRUNK #3976 (See https://builds.apache.org/job/HBase-TRUNK/3976/ ) HBASE-8146 IntegrationTestBigLinkedList does not work on distributed setup (Revision 1458585) Result = FAILURE enis : Files : /hbase/trunk/hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
          Hide
          enis Enis Soztutar added a comment -

          Committed this to 0.94.7 as well. Attaching committed patch, which adds one class from main hbase jar to the job classpath.

          Show
          enis Enis Soztutar added a comment - Committed this to 0.94.7 as well. Attaching committed patch, which adds one class from main hbase jar to the job classpath.
          Hide
          hudson Hudson added a comment -

          Integrated in HBase-0.94 #918 (See https://builds.apache.org/job/HBase-0.94/918/)
          HBASE-8146 IntegrationTestBigLinkedList does not work on distributed setup (Revision 1459065)

          Result = SUCCESS
          enis :
          Files :

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java
          Show
          hudson Hudson added a comment - Integrated in HBase-0.94 #918 (See https://builds.apache.org/job/HBase-0.94/918/ ) HBASE-8146 IntegrationTestBigLinkedList does not work on distributed setup (Revision 1459065) Result = SUCCESS enis : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java
          Hide
          hudson Hudson added a comment -

          Integrated in HBase-0.94-security #129 (See https://builds.apache.org/job/HBase-0.94-security/129/)
          HBASE-8146 IntegrationTestBigLinkedList does not work on distributed setup (Revision 1459065)

          Result = FAILURE
          enis :
          Files :

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java
          Show
          hudson Hudson added a comment - Integrated in HBase-0.94-security #129 (See https://builds.apache.org/job/HBase-0.94-security/129/ ) HBASE-8146 IntegrationTestBigLinkedList does not work on distributed setup (Revision 1459065) Result = FAILURE enis : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java
          Hide
          hudson Hudson added a comment -

          Integrated in HBase-0.94-security-on-Hadoop-23 #13 (See https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/13/)
          HBASE-8146 IntegrationTestBigLinkedList does not work on distributed setup (Revision 1459065)

          Result = FAILURE
          enis :
          Files :

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java
          Show
          hudson Hudson added a comment - Integrated in HBase-0.94-security-on-Hadoop-23 #13 (See https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/13/ ) HBASE-8146 IntegrationTestBigLinkedList does not work on distributed setup (Revision 1459065) Result = FAILURE enis : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java

            People

            • Assignee:
              enis Enis Soztutar
              Reporter:
              enis Enis Soztutar
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development