Hive
  1. Hive
  2. HIVE-1235

use Ivy for fetching HBase dependencies

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.6.0
    • Fix Version/s: 0.7.0
    • Component/s: HBase Handler
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      For now I just checked in the .jars under hbase-handler/lib to avoid having to pull down the huge tarballs. If individual jars were available (HBASE-1933), we could use Ivy to pull those.

      1. HIVE-1235.3.patch
        24 kB
        John Sichi
      2. HIVE-1235.2.patch
        23 kB
        John Sichi
      3. HIVE-1235.1.patch
        10 kB
        John Sichi

        Issue Links

          Activity

          Hide
          Karthik K added a comment -

          The snapshots of hbase (0.21.0) should now be available in the apache snapshot repositories -

          https://repository.apache.org/content/repositories/snapshots/org/apache/hbase/hbase-core/0.21.0-SNAPSHOT/ .

          Private Repository here at - http://people.apache.org/~rawson/repo/ , hosts some additional dependencies ( modified hdfs + thrift 0.2.0).

          Hope it helps to integrate with hive.

          Show
          Karthik K added a comment - The snapshots of hbase (0.21.0) should now be available in the apache snapshot repositories - https://repository.apache.org/content/repositories/snapshots/org/apache/hbase/hbase-core/0.21.0-SNAPSHOT/ . Private Repository here at - http://people.apache.org/~rawson/repo/ , hosts some additional dependencies ( modified hdfs + thrift 0.2.0). Hope it helps to integrate with hive.
          Hide
          John Sichi added a comment -

          Awesome, I'll give that a try.

          Show
          John Sichi added a comment - Awesome, I'll give that a try.
          Hide
          John Sichi added a comment -

          Linking HIVE-1533 for Zookeeper.

          Show
          John Sichi added a comment - Linking HIVE-1533 for Zookeeper.
          Hide
          John Sichi added a comment -

          Starting on this one with HBase 0.89 and zookeeper 3.3.1 as part of addressing HIVE-1716.

          Show
          John Sichi added a comment - Starting on this one with HBase 0.89 and zookeeper 3.3.1 as part of addressing HIVE-1716 .
          Hide
          John Sichi added a comment -

          This patch (HIVE-1235.1.patch) is not ready for review yet, and ant test is hitting an OOM in one of the ivy calls right before running the HBase tests; just posting it here as a checkpoint. To apply, you also need:

          svn remove lib/hbase-0.20.3-test.jar
          svn remove lib/hbase-0.20.3.jar
          svn remove lib/zookeeper-3.2.2.jar

          Show
          John Sichi added a comment - This patch ( HIVE-1235 .1.patch) is not ready for review yet, and ant test is hitting an OOM in one of the ivy calls right before running the HBase tests; just posting it here as a checkpoint. To apply, you also need: svn remove lib/hbase-0.20.3-test.jar svn remove lib/hbase-0.20.3.jar svn remove lib/zookeeper-3.2.2.jar
          Hide
          John Sichi added a comment -

          Note that transitive dependencies were pulling down all kinds of odds and ends we don't need, so I had to turn them off and then get the only hard dependency (guava) explicitly. Slightly defeating the purpose of ivy, but...

          Show
          John Sichi added a comment - Note that transitive dependencies were pulling down all kinds of odds and ends we don't need, so I had to turn them off and then get the only hard dependency (guava) explicitly. Slightly defeating the purpose of ivy, but...
          Hide
          John Sichi added a comment -

          HIVE-1235.2.patch passes tests. A couple of the HBase tests had broken due to recent changes, so this fixes those. It also gets rid of redundant ivy-retrieve for hadoopcore (which is what was causing the OOM's).

          Show
          John Sichi added a comment - HIVE-1235 .2.patch passes tests. A couple of the HBase tests had broken due to recent changes, so this fixes those. It also gets rid of redundant ivy-retrieve for hadoopcore (which is what was causing the OOM's).
          Hide
          John Sichi added a comment -

          Review board:

          https://reviews.apache.org/r/367/

          Note this also makes the hbase-handler build non-conditional since we've dropped support for pre-0.20.

          Show
          John Sichi added a comment - Review board: https://reviews.apache.org/r/367/ Note this also makes the hbase-handler build non-conditional since we've dropped support for pre-0.20.
          Hide
          John Sichi added a comment -

          Would be nice to get this one in for 0.7.

          Show
          John Sichi added a comment - Would be nice to get this one in for 0.7.
          Hide
          John Sichi added a comment -

          Oops, got one failure in TestHBaseSerDe due to a change in the Result constructor contract (it now expects the kv's to be presorted). I'll uploaded another patch.

          Show
          John Sichi added a comment - Oops, got one failure in TestHBaseSerDe due to a change in the Result constructor contract (it now expects the kv's to be presorted). I'll uploaded another patch.
          Hide
          John Sichi added a comment -

          I updated review board as well.

          Show
          John Sichi added a comment - I updated review board as well.
          Hide
          John Sichi added a comment -

          HIVE-1235.3.patch

          Show
          John Sichi added a comment - HIVE-1235 .3.patch
          Hide
          Carl Steinbach added a comment -

          +1. Will commit if tests pass.

          Show
          Carl Steinbach added a comment - +1. Will commit if tests pass.
          Hide
          Carl Steinbach added a comment -

          Committed. Thanks John!

          Show
          Carl Steinbach added a comment - Committed. Thanks John!

            People

            • Assignee:
              John Sichi
              Reporter:
              John Sichi
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development