Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.95.2
    • Fix Version/s: 0.98.0, 0.96.1, 0.94.14
    • Component/s: mapreduce
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Introduces the method TableMapReduceUtil#addHBaseDependencyJars for adding HBase and its direct dependencies (only) to the job configuration.

      This is intended as a low-level API, facilitating code reuse between this class and its mapred counterpart. It is also of use to external tools that need to build a MapReduce job that interacts with HBase but want fine-grained control over the jars shipped to the cluster. See also PIG-3285 and HIVE-2055.
      Show
      Introduces the method TableMapReduceUtil#addHBaseDependencyJars for adding HBase and its direct dependencies (only) to the job configuration. This is intended as a low-level API, facilitating code reuse between this class and its mapred counterpart. It is also of use to external tools that need to build a MapReduce job that interacts with HBase but want fine-grained control over the jars shipped to the cluster. See also PIG-3285 and HIVE-2055 .

      Description

      The way we support adding HBase dependencies to a MapReduce job in TableMapReduceUtils#addDependencyJars(job) is a bit monolithic. Advanced users need a way to add HBase and its dependencies to their job without us snooping around for ouput formats and the like (see PIG-3285). We can also benefit from a little more code reuse between our mapred and mapreduce namespaces.

      1. HBASE-9165-0.94.01.patch
        6 kB
        Nick Dimiduk
      2. HBASE-9165-0.96.00.patch
        36 kB
        Nick Dimiduk
      3. HBASE-9165.02.patch
        40 kB
        Nick Dimiduk
      4. HBASE-9165-0.94.00.patch
        6 kB
        Nick Dimiduk
      5. 0001-HBASE-9165-mapreduce-Modularize-building-dependency-.patch
        40 kB
        Nick Dimiduk
      6. 0001-HBASE-9165-mapreduce-Modularize-building-dependency-.patch
        40 kB
        Nick Dimiduk
      7. 0001-HBASE-9165-mapreduce-Modularize-building-dependency-.patch
        42 kB
        Nick Dimiduk

        Issue Links

          Activity

          Hide
          Nick Dimiduk added a comment -
          • Separate adding HBase and dependencies from adding other job dependencies, and expose it as a separate method that other projects can use (for PIG-3285).
          • Explicitly add hbase-server to the list of dependencies we ship with the job, for users who extend the classes we provide (see HBASE-9112).
          • Add integration test for addDependencyJars.
          • Code reuse for TestTableMapReduce.
          Show
          Nick Dimiduk added a comment - Separate adding HBase and dependencies from adding other job dependencies, and expose it as a separate method that other projects can use (for PIG-3285 ). Explicitly add hbase-server to the list of dependencies we ship with the job, for users who extend the classes we provide (see HBASE-9112 ). Add integration test for addDependencyJars. Code reuse for TestTableMapReduce.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12596999/0001-HBASE-9165-mapreduce-Modularize-building-dependency-.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 19 new or modified tests.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          -1 javadoc. The javadoc tool appears to have generated 20 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          -1 core tests. The patch failed these unit tests:

          -1 core zombie tests. There are 1 zombie test(s): at org.apache.hadoop.hbase.client.TestAdmin.testTableExist(TestAdmin.java:835)

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/6671//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6671//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6671//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6671//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6671//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6671//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6671//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6671//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6671//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6671//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12596999/0001-HBASE-9165-mapreduce-Modularize-building-dependency-.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 19 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. -1 javadoc . The javadoc tool appears to have generated 20 warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: -1 core zombie tests . There are 1 zombie test(s): at org.apache.hadoop.hbase.client.TestAdmin.testTableExist(TestAdmin.java:835) Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/6671//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6671//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6671//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6671//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6671//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6671//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6671//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6671//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6671//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6671//console This message is automatically generated.
          Hide
          Daniel Dai added a comment -

          +1. Pig need this feature to ship HBase dependent jars correctly. Currently Pig user have to register HBase dependent jars (such as protobuf.jar) to use HBaseStorage.

          Show
          Daniel Dai added a comment - +1. Pig need this feature to ship HBase dependent jars correctly. Currently Pig user have to register HBase dependent jars (such as protobuf.jar) to use HBaseStorage.
          Hide
          Nick Dimiduk added a comment -

          The patch has gone stale.

          Show
          Nick Dimiduk added a comment - The patch has gone stale.
          Hide
          Nick Dimiduk added a comment -

          Rebased onto trunk. Let's see what buildbot says.

          Show
          Nick Dimiduk added a comment - Rebased onto trunk. Let's see what buildbot says.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12613499/0001-HBASE-9165-mapreduce-Modularize-building-dependency-.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 19 new or modified tests.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          -1 site. The patch appears to cause mvn site goal to fail.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7831//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7831//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7831//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7831//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7831//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7831//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7831//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7831//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7831//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7831//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7831//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12613499/0001-HBASE-9165-mapreduce-Modularize-building-dependency-.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 19 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. -1 javadoc . The javadoc tool appears to have generated 1 warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 site . The patch appears to cause mvn site goal to fail. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7831//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7831//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7831//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7831//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7831//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7831//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7831//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7831//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7831//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7831//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7831//console This message is automatically generated.
          Hide
          Ted Yu added a comment -
          +      // when making changes here, consider also mapreduce.TableMapReduceUtils
          

          You meant TableMapReduceUtil, right ?

          +  public static class ProcessContentsMapper extends MapReduceBase implements
          

          The above class can be package-private, right ?

          Show
          Ted Yu added a comment - + // when making changes here, consider also mapreduce.TableMapReduceUtils You meant TableMapReduceUtil, right ? + public static class ProcessContentsMapper extends MapReduceBase implements The above class can be package-private, right ?
          Hide
          stack added a comment -

          Skimmed. lgtm

          Show
          stack added a comment - Skimmed. lgtm
          Hide
          Nick Dimiduk added a comment -

          Updated patch to address Ted's review comments. Both mapred and mapreduce.TestTableMapReduce pass locally on hadoop-1 and hadoop-2. Will let a buildbot run and commit if there's clean tests and no objections.

          Show
          Nick Dimiduk added a comment - Updated patch to address Ted's review comments. Both mapred and mapreduce.TestTableMapReduce pass locally on hadoop-1 and hadoop-2. Will let a buildbot run and commit if there's clean tests and no objections.
          Hide
          Nick Dimiduk added a comment -

          Lars Hofhansl do you want at least a partial backport to 0.94? I think we can leave out the test refactor if you'd prefer, keep only the new method. This should be helpful for downstream projects (Pig, Hive). I'd like to include the new method so I can also backport HBASE-8438.

          Show
          Nick Dimiduk added a comment - Lars Hofhansl do you want at least a partial backport to 0.94? I think we can leave out the test refactor if you'd prefer, keep only the new method. This should be helpful for downstream projects (Pig, Hive). I'd like to include the new method so I can also backport HBASE-8438 .
          Hide
          Nick Dimiduk added a comment -

          Here's a patch for 0.94. Just adds the new method and fixes the unit test on addDependencyJars. Both mapred, mapreduce.TestTableMapReduce pass locally on hadoop-1 and hadoop-2 profiles.

          Show
          Nick Dimiduk added a comment - Here's a patch for 0.94. Just adds the new method and fixes the unit test on addDependencyJars. Both mapred, mapreduce.TestTableMapReduce pass locally on hadoop-1 and hadoop-2 profiles.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12613968/HBASE-9165-0.94.00.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7867//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12613968/HBASE-9165-0.94.00.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified tests. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7867//console This message is automatically generated.
          Hide
          Lars Hofhansl added a comment - - edited

          So with mapred we are removing:

          -      org.apache.zookeeper.ZooKeeper.class,
          -      com.google.common.base.Function.class,
          -      com.google.protobuf.Message.class,
          

          And adding:

          +      org.apache.zookeeper.ZooKeeper.class,
          +      com.google.protobuf.Message.class,
          +      com.google.common.collect.ImmutableSet.class,
          +      org.apache.hadoop.hbase.util.Bytes.class
          

          Is that correct?

          Show
          Lars Hofhansl added a comment - - edited So with mapred we are removing: - org.apache.zookeeper.ZooKeeper.class, - com.google.common.base.Function.class, - com.google.protobuf.Message.class, And adding: + org.apache.zookeeper.ZooKeeper.class, + com.google.protobuf.Message.class, + com.google.common.collect.ImmutableSet.class, + org.apache.hadoop.hbase.util.Bytes.class Is that correct?
          Hide
          Nick Dimiduk added a comment -

          Yikes, I didn't realize the list of classes are different between the two. This is a bug, right? Mapred is just a pass-through to the mapreduce implementations.

          Zookeeper and protobuf jars are drawn by the same class in both cases. The same Guava jar is brought in by Function.class and ImmutableSet.class respectively. Not including the hbase.jar is a bug as reported by HBASE-9112, and fixed inadvertently by HBASE-8146.

          So yes, this is correct. Thanks for double-checking.

          Show
          Nick Dimiduk added a comment - Yikes, I didn't realize the list of classes are different between the two. This is a bug, right? Mapred is just a pass-through to the mapreduce implementations. Zookeeper and protobuf jars are drawn by the same class in both cases. The same Guava jar is brought in by Function.class and ImmutableSet.class respectively. Not including the hbase.jar is a bug as reported by HBASE-9112 , and fixed inadvertently by HBASE-8146 . So yes, this is correct. Thanks for double-checking.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12613968/HBASE-9165-0.94.00.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7869//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12613968/HBASE-9165-0.94.00.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified tests. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7869//console This message is automatically generated.
          Hide
          Nick Dimiduk added a comment -

          Reattaching most recent trunk patch for the benefit of the buildbot.

          Show
          Nick Dimiduk added a comment - Reattaching most recent trunk patch for the benefit of the buildbot.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12613983/HBASE-9165.02.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 19 new or modified tests.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          -1 site. The patch appears to cause mvn site goal to fail.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7871//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7871//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7871//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7871//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7871//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7871//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7871//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7871//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7871//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7871//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7871//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12613983/HBASE-9165.02.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 19 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 site . The patch appears to cause mvn site goal to fail. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7871//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7871//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7871//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7871//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7871//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7871//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7871//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7871//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7871//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7871//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7871//console This message is automatically generated.
          Hide
          Nick Dimiduk added a comment -

          Attaching the patch committed to 0.96. From the commit message:

          Note this patch differs from the one applied to TRUNK/0.98 in that it omits changes to TestTableMapReduceUtil which have not yet been backported from HBASE-8534. This can be addressed when that patch is backported (HBASE-9484).

          Show
          Nick Dimiduk added a comment - Attaching the patch committed to 0.96. From the commit message: Note this patch differs from the one applied to TRUNK/0.98 in that it omits changes to TestTableMapReduceUtil which have not yet been backported from HBASE-8534 . This can be addressed when that patch is backported ( HBASE-9484 ).
          Hide
          Nick Dimiduk added a comment -

          Lars Hofhansl are you okay with the 0.94 backport?

          Show
          Nick Dimiduk added a comment - Lars Hofhansl are you okay with the 0.94 backport?
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12614091/HBASE-9165-0.96.00.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 15 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7883//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12614091/HBASE-9165-0.96.00.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 15 new or modified tests. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7883//console This message is automatically generated.
          Hide
          Lars Hofhansl added a comment -

          +1
          You sure we won't need com.google.common.base.Function.

          Show
          Lars Hofhansl added a comment - +1 You sure we won't need com.google.common.base.Function.
          Hide
          Nick Dimiduk added a comment -

          Yes. This change is an effective noop; the same jar contains both classes. It's value is to bring the implementations across branches back into sync.

          soleil:hbase-94 ndimiduk$ grep \<guava\.version\> pom.xml
              <guava.version>11.0.2</guava.version>
          soleil:hbase-94 ndimiduk$ jar tf ~/.m2/repository/com/google/guava/guava/11.0.2/guava-11.0.2.jar | grep base\.Function\.class
          com/google/common/base/Function.class
          soleil:hbase-94 ndimiduk$ jar tf ~/.m2/repository/com/google/guava/guava/11.0.2/guava-11.0.2.jar | grep collect\.ImmutableSet\.class                                                                      
          com/google/common/collect/ImmutableSet.class
          
          Show
          Nick Dimiduk added a comment - Yes. This change is an effective noop; the same jar contains both classes. It's value is to bring the implementations across branches back into sync. soleil:hbase-94 ndimiduk$ grep \<guava\.version\> pom.xml <guava.version>11.0.2</guava.version> soleil:hbase-94 ndimiduk$ jar tf ~/.m2/repository/com/google/guava/guava/11.0.2/guava-11.0.2.jar | grep base\.Function\.class com/google/common/base/Function.class soleil:hbase-94 ndimiduk$ jar tf ~/.m2/repository/com/google/guava/guava/11.0.2/guava-11.0.2.jar | grep collect\.ImmutableSet\.class com/google/common/collect/ImmutableSet.class
          Hide
          Nick Dimiduk added a comment -

          I stood up a pseudo-distributed hadoop on my laptop, built the tarball from 0.94.14-SNAPSHOT, and launched PerformanceEvaluation sequentialWrite 2. In the job launch logs I see

          13/11/15 11:36:46 DEBUG mapreduce.TableMapReduceUtil: For class com.google.common.collect.ImmutableSet, using jar /private/tmp/hbase-0.94.14-SNAPSHOT/lib/guava-11.0.2.jar
          

          The submitted job.xml file contains the following value for "tmpjars":

          file:/private/tmp/hbase-0.94.14-SNAPSHOT/hbase-0.94.14-SNAPSHOT.jar,file:/private/tmp/hbase-0.94.14-SNAPSHOT/lib/protobuf-java-2.4.0a.jar,file:/private/tmp/hbase-0.94.14-SNAPSHOT/lib/zookeeper-3.4.5.jar,file:/private/tmp/hbase-0.94.14-SNAPSHOT/lib/guava-11.0.2.jar,file:/private/tmp/hbase-0.94.14-SNAPSHOT/lib/hadoop-core-1.0.4.jar,file:/private/tmp/hbase-0.94.14-SNAPSHOT/hbase-0.94.14-SNAPSHOT-tests.jar
          

          I think we're good

          Show
          Nick Dimiduk added a comment - I stood up a pseudo-distributed hadoop on my laptop, built the tarball from 0.94.14-SNAPSHOT, and launched PerformanceEvaluation sequentialWrite 2. In the job launch logs I see 13/11/15 11:36:46 DEBUG mapreduce.TableMapReduceUtil: For class com.google.common.collect.ImmutableSet, using jar /private/tmp/hbase-0.94.14-SNAPSHOT/lib/guava-11.0.2.jar The submitted job.xml file contains the following value for "tmpjars": file:/private/tmp/hbase-0.94.14-SNAPSHOT/hbase-0.94.14-SNAPSHOT.jar,file:/private/tmp/hbase-0.94.14-SNAPSHOT/lib/protobuf-java-2.4.0a.jar,file:/private/tmp/hbase-0.94.14-SNAPSHOT/lib/zookeeper-3.4.5.jar,file:/private/tmp/hbase-0.94.14-SNAPSHOT/lib/guava-11.0.2.jar,file:/private/tmp/hbase-0.94.14-SNAPSHOT/lib/hadoop-core-1.0.4.jar,file:/private/tmp/hbase-0.94.14-SNAPSHOT/hbase-0.94.14-SNAPSHOT-tests.jar I think we're good
          Hide
          Lars Hofhansl added a comment -

          If they are in the same jar, let's just add them both. Seems safer and it also calls out the actual dependencies.
          +1 otherwise.

          Show
          Lars Hofhansl added a comment - If they are in the same jar, let's just add them both. Seems safer and it also calls out the actual dependencies. +1 otherwise.
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #838 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/838/)
          HBASE-9165 [mapreduce] Modularize building dependency jars

          • Separate adding HBase and dependencies from adding other job
            dependencies, and expose it as a separate method that other
            projects can use (for PIG-3285).
          • Explicitly add hbase-server to the list of dependencies we ship
            with the job, for users who extend the classes we provide (see
            HBASE-9112).
          • Add integration test for addDependencyJars.
          • Code reuse for TestTableMapReduce. (ndimiduk: rev 1542341)
          • /hbase/trunk/hbase-it/src/test/java/org/apache/hadoop/hbase/mapreduce/IntegrationTestTableMapReduceUtil.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/IdentityTableMapper.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapred/TestTableMapReduce.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduceBase.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduceUtil.java
          Show
          Hudson added a comment - SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #838 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/838/ ) HBASE-9165 [mapreduce] Modularize building dependency jars Separate adding HBase and dependencies from adding other job dependencies, and expose it as a separate method that other projects can use (for PIG-3285 ). Explicitly add hbase-server to the list of dependencies we ship with the job, for users who extend the classes we provide (see HBASE-9112 ). Add integration test for addDependencyJars. Code reuse for TestTableMapReduce. (ndimiduk: rev 1542341) /hbase/trunk/hbase-it/src/test/java/org/apache/hadoop/hbase/mapreduce/IntegrationTestTableMapReduceUtil.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/IdentityTableMapper.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapred/TestTableMapReduce.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduceBase.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduceUtil.java
          Hide
          Nick Dimiduk added a comment -

          As you prefer. Attached the updated patch. If there's no further objection, I'll commit this afternoon.

          Thanks Lars Hofhansl.

          Show
          Nick Dimiduk added a comment - As you prefer. Attached the updated patch. If there's no further objection, I'll commit this afternoon. Thanks Lars Hofhansl .
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12614123/HBASE-9165-0.94.01.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7888//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12614123/HBASE-9165-0.94.01.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified tests. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7888//console This message is automatically generated.
          Hide
          Lars Hofhansl added a comment -

          +1

          Thanks Nick.

          Show
          Lars Hofhansl added a comment - +1 Thanks Nick.
          Hide
          Nick Dimiduk added a comment -

          Committed across all 3 branches. Thanks for the reviews everyone.

          Show
          Nick Dimiduk added a comment - Committed across all 3 branches. Thanks for the reviews everyone.
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in HBase-TRUNK #4681 (See https://builds.apache.org/job/HBase-TRUNK/4681/)
          HBASE-9165 [mapreduce] Modularize building dependency jars

          • Separate adding HBase and dependencies from adding other job
            dependencies, and expose it as a separate method that other
            projects can use (for PIG-3285).
          • Explicitly add hbase-server to the list of dependencies we ship
            with the job, for users who extend the classes we provide (see
            HBASE-9112).
          • Add integration test for addDependencyJars.
          • Code reuse for TestTableMapReduce. (ndimiduk: rev 1542341)
          • /hbase/trunk/hbase-it/src/test/java/org/apache/hadoop/hbase/mapreduce/IntegrationTestTableMapReduceUtil.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/IdentityTableMapper.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapred/TestTableMapReduce.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduceBase.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduceUtil.java
          Show
          Hudson added a comment - SUCCESS: Integrated in HBase-TRUNK #4681 (See https://builds.apache.org/job/HBase-TRUNK/4681/ ) HBASE-9165 [mapreduce] Modularize building dependency jars Separate adding HBase and dependencies from adding other job dependencies, and expose it as a separate method that other projects can use (for PIG-3285 ). Explicitly add hbase-server to the list of dependencies we ship with the job, for users who extend the classes we provide (see HBASE-9112 ). Add integration test for addDependencyJars. Code reuse for TestTableMapReduce. (ndimiduk: rev 1542341) /hbase/trunk/hbase-it/src/test/java/org/apache/hadoop/hbase/mapreduce/IntegrationTestTableMapReduceUtil.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/IdentityTableMapper.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapred/TestTableMapReduce.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduceBase.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduceUtil.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in hbase-0.96 #190 (See https://builds.apache.org/job/hbase-0.96/190/)
          HBASE-9165 [mapreduce] Modularize building dependency jars

          • Separate adding HBase and dependencies from adding other job
            dependencies, and expose it as a separate method that other
            projects can use (for PIG-3285).
          • Explicitly add hbase-server to the list of dependencies we ship
            with the job, for users who extend the classes we provide (see
            HBASE-9112).
          • Add integration test for addDependencyJars.
          • Code reuse for TestTableMapReduce.

          Note this patch differs from the one applied to TRUNK/0.98 in that it omits
          changes to TestTableMapReduceUtil which have not yet been backported from
          HBASE-8534. This can be addressed when that patch is backported (HBASE-9484). (ndimiduk: rev 1542342)

          • /hbase/branches/0.96/hbase-it/src/test/java/org/apache/hadoop/hbase/mapreduce/IntegrationTestTableMapReduceUtil.java
          • /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java
          • /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/IdentityTableMapper.java
          • /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
          • /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/mapred/TestTableMapReduce.java
          • /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java
          • /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduceBase.java
          Show
          Hudson added a comment - FAILURE: Integrated in hbase-0.96 #190 (See https://builds.apache.org/job/hbase-0.96/190/ ) HBASE-9165 [mapreduce] Modularize building dependency jars Separate adding HBase and dependencies from adding other job dependencies, and expose it as a separate method that other projects can use (for PIG-3285 ). Explicitly add hbase-server to the list of dependencies we ship with the job, for users who extend the classes we provide (see HBASE-9112 ). Add integration test for addDependencyJars. Code reuse for TestTableMapReduce. Note this patch differs from the one applied to TRUNK/0.98 in that it omits changes to TestTableMapReduceUtil which have not yet been backported from HBASE-8534 . This can be addressed when that patch is backported ( HBASE-9484 ). (ndimiduk: rev 1542342) /hbase/branches/0.96/hbase-it/src/test/java/org/apache/hadoop/hbase/mapreduce/IntegrationTestTableMapReduceUtil.java /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/IdentityTableMapper.java /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/mapred/TestTableMapReduce.java /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduceBase.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in HBase-0.94 #1203 (See https://builds.apache.org/job/HBase-0.94/1203/)
          HBASE-9165 [mapreduce] Modularize building dependency jars

          Separate adding HBase and dependencies from adding other job dependencies, and
          expose it as a separate method that other projects can use (for PIG-3285,
          HIVE-2055). (ndimiduk: rev 1542414)

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java
          Show
          Hudson added a comment - FAILURE: Integrated in HBase-0.94 #1203 (See https://builds.apache.org/job/HBase-0.94/1203/ ) HBASE-9165 [mapreduce] Modularize building dependency jars Separate adding HBase and dependencies from adding other job dependencies, and expose it as a separate method that other projects can use (for PIG-3285 , HIVE-2055 ). (ndimiduk: rev 1542414) /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in HBase-0.94-security #337 (See https://builds.apache.org/job/HBase-0.94-security/337/)
          HBASE-9165 [mapreduce] Modularize building dependency jars

          Separate adding HBase and dependencies from adding other job dependencies, and
          expose it as a separate method that other projects can use (for PIG-3285,
          HIVE-2055). (ndimiduk: rev 1542414)

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java
          Show
          Hudson added a comment - SUCCESS: Integrated in HBase-0.94-security #337 (See https://builds.apache.org/job/HBase-0.94-security/337/ ) HBASE-9165 [mapreduce] Modularize building dependency jars Separate adding HBase and dependencies from adding other job dependencies, and expose it as a separate method that other projects can use (for PIG-3285 , HIVE-2055 ). (ndimiduk: rev 1542414) /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in hbase-0.96-hadoop2 #120 (See https://builds.apache.org/job/hbase-0.96-hadoop2/120/)
          HBASE-9165 [mapreduce] Modularize building dependency jars

          • Separate adding HBase and dependencies from adding other job
            dependencies, and expose it as a separate method that other
            projects can use (for PIG-3285).
          • Explicitly add hbase-server to the list of dependencies we ship
            with the job, for users who extend the classes we provide (see
            HBASE-9112).
          • Add integration test for addDependencyJars.
          • Code reuse for TestTableMapReduce.

          Note this patch differs from the one applied to TRUNK/0.98 in that it omits
          changes to TestTableMapReduceUtil which have not yet been backported from
          HBASE-8534. This can be addressed when that patch is backported (HBASE-9484). (ndimiduk: rev 1542342)

          • /hbase/branches/0.96/hbase-it/src/test/java/org/apache/hadoop/hbase/mapreduce/IntegrationTestTableMapReduceUtil.java
          • /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java
          • /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/IdentityTableMapper.java
          • /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
          • /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/mapred/TestTableMapReduce.java
          • /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java
          • /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduceBase.java
          Show
          Hudson added a comment - FAILURE: Integrated in hbase-0.96-hadoop2 #120 (See https://builds.apache.org/job/hbase-0.96-hadoop2/120/ ) HBASE-9165 [mapreduce] Modularize building dependency jars Separate adding HBase and dependencies from adding other job dependencies, and expose it as a separate method that other projects can use (for PIG-3285 ). Explicitly add hbase-server to the list of dependencies we ship with the job, for users who extend the classes we provide (see HBASE-9112 ). Add integration test for addDependencyJars. Code reuse for TestTableMapReduce. Note this patch differs from the one applied to TRUNK/0.98 in that it omits changes to TestTableMapReduceUtil which have not yet been backported from HBASE-8534 . This can be addressed when that patch is backported ( HBASE-9484 ). (ndimiduk: rev 1542342) /hbase/branches/0.96/hbase-it/src/test/java/org/apache/hadoop/hbase/mapreduce/IntegrationTestTableMapReduceUtil.java /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/IdentityTableMapper.java /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/mapred/TestTableMapReduce.java /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduceBase.java
          Hide
          Lars Hofhansl added a comment - - edited
          java.lang.AssertionError
          	at org.junit.Assert.fail(Assert.java:92)
          	at org.junit.Assert.assertTrue(Assert.java:43)
          	at org.junit.Assert.assertTrue(Assert.java:54)
          	at org.apache.hadoop.hbase.mapreduce.TestTableMapReduce.testAddDependencyJars(TestTableMapReduce.java:272)
          

          Can you take a look Nick Dimiduk?

          Show
          Lars Hofhansl added a comment - - edited java.lang.AssertionError at org.junit.Assert.fail(Assert.java:92) at org.junit.Assert.assertTrue(Assert.java:43) at org.junit.Assert.assertTrue(Assert.java:54) at org.apache.hadoop.hbase.mapreduce.TestTableMapReduce.testAddDependencyJars(TestTableMapReduce.java:272) Can you take a look Nick Dimiduk ?
          Hide
          Lars Hofhansl added a comment -

          The reason is probably this from the logs:

          2013-11-16 17:57:00,896 DEBUG [pool-1-thread-1] mapreduce.TableMapReduceUtil(665): For class org.apache.hadoop.hbase.util.Bytes, using jar /x1/jenkins/jenkins-slave/workspace/HBase-0.94/trunk/target/test-dir/hadoop-1223249546787952422.jar
          

          This does not happen on my local box.

          Show
          Lars Hofhansl added a comment - The reason is probably this from the logs: 2013-11-16 17:57:00,896 DEBUG [pool-1-thread-1] mapreduce.TableMapReduceUtil(665): For class org.apache.hadoop.hbase.util.Bytes, using jar /x1/jenkins/jenkins-slave/workspace/HBase-0.94/trunk/target/test-dir/hadoop-1223249546787952422.jar This does not happen on my local box.
          Hide
          Lars Hofhansl added a comment -

          This only passes on my local box, because the directory I have my build in is called hbase-0.94 (on jenkins it is HBase-0.94). So even on my box it only passes by pure accident.

          Show
          Lars Hofhansl added a comment - This only passes on my local box, because the directory I have my build in is called hbase-0.94 (on jenkins it is HBase-0.94). So even on my box it only passes by pure accident.
          Hide
          Lars Hofhansl added a comment -

          Fixed in 0.94 for now, but removing the newly added test for "hbase" in the dependency jars.

          Show
          Lars Hofhansl added a comment - Fixed in 0.94 for now, but removing the newly added test for "hbase" in the dependency jars.
          Hide
          Lars Hofhansl added a comment -

          0.96 and trunk do not have this change TestTableMapreduce. Should be good now.

          Show
          Lars Hofhansl added a comment - 0.96 and trunk do not have this change TestTableMapreduce. Should be good now.
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in HBase-0.94-security #338 (See https://builds.apache.org/job/HBase-0.94-security/338/)
          HBASE-9165 Addendum, fix TestTableMapreduce (larsh: rev 1542642)

          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java
          Show
          Hudson added a comment - SUCCESS: Integrated in HBase-0.94-security #338 (See https://builds.apache.org/job/HBase-0.94-security/338/ ) HBASE-9165 Addendum, fix TestTableMapreduce (larsh: rev 1542642) /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in HBase-0.94 #1204 (See https://builds.apache.org/job/HBase-0.94/1204/)
          HBASE-9165 Addendum, fix TestTableMapreduce (larsh: rev 1542642)

          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java
          Show
          Hudson added a comment - SUCCESS: Integrated in HBase-0.94 #1204 (See https://builds.apache.org/job/HBase-0.94/1204/ ) HBASE-9165 Addendum, fix TestTableMapreduce (larsh: rev 1542642) /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java
          Hide
          Nick Dimiduk added a comment -

          This test is on 0.96 and trunk, but as an integration test. Sorry for the mistake.

          Show
          Nick Dimiduk added a comment - This test is on 0.96 and trunk, but as an integration test. Sorry for the mistake.
          Hide
          Lars Hofhansl added a comment -

          NP
          Are you OK with my "fix"? Just removed the test for HBase in tmpjars.

          Show
          Lars Hofhansl added a comment - NP Are you OK with my "fix"? Just removed the test for HBase in tmpjars.
          Hide
          Nick Dimiduk added a comment -

          Yeah, it's fine. Wish there was a way to include the check, but without an integration test I don't think we can.

          Show
          Nick Dimiduk added a comment - Yeah, it's fine. Wish there was a way to include the check, but without an integration test I don't think we can.

            People

            • Assignee:
              Nick Dimiduk
              Reporter:
              Nick Dimiduk
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development