Hive
  1. Hive
  2. HIVE-1235

use Ivy for fetching HBase dependencies

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.6.0
    • Fix Version/s: 0.7.0
    • Component/s: HBase Handler
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      For now I just checked in the .jars under hbase-handler/lib to avoid having to pull down the huge tarballs. If individual jars were available (HBASE-1933), we could use Ivy to pull those.

      1. HIVE-1235.1.patch
        10 kB
        John Sichi
      2. HIVE-1235.2.patch
        23 kB
        John Sichi
      3. HIVE-1235.3.patch
        24 kB
        John Sichi

        Issue Links

          Activity

          John Sichi created issue -
          John Sichi made changes -
          Field Original Value New Value
          Link This issue relates to HIVE-705 [ HIVE-705 ]
          John Sichi made changes -
          Link This issue is blocked by HBASE-1933 [ HBASE-1933 ]
          John Sichi made changes -
          Link This issue is blocked by ZOOKEEPER-224 [ ZOOKEEPER-224 ]
          John Sichi made changes -
          Component/s HBase Handler [ 12313461 ]
          Component/s Build Infrastructure [ 12312593 ]
          Hide
          Karthik K added a comment -

          The snapshots of hbase (0.21.0) should now be available in the apache snapshot repositories -

          https://repository.apache.org/content/repositories/snapshots/org/apache/hbase/hbase-core/0.21.0-SNAPSHOT/ .

          Private Repository here at - http://people.apache.org/~rawson/repo/ , hosts some additional dependencies ( modified hdfs + thrift 0.2.0).

          Hope it helps to integrate with hive.

          Show
          Karthik K added a comment - The snapshots of hbase (0.21.0) should now be available in the apache snapshot repositories - https://repository.apache.org/content/repositories/snapshots/org/apache/hbase/hbase-core/0.21.0-SNAPSHOT/ . Private Repository here at - http://people.apache.org/~rawson/repo/ , hosts some additional dependencies ( modified hdfs + thrift 0.2.0). Hope it helps to integrate with hive.
          Hide
          John Sichi added a comment -

          Awesome, I'll give that a try.

          Show
          John Sichi added a comment - Awesome, I'll give that a try.
          John Sichi made changes -
          Fix Version/s 0.6.0 [ 12314524 ]
          John Sichi made changes -
          Link This issue blocks HIVE-1512 [ HIVE-1512 ]
          John Sichi made changes -
          Link This issue relates to HIVE-1533 [ HIVE-1533 ]
          Hide
          John Sichi added a comment -

          Linking HIVE-1533 for Zookeeper.

          Show
          John Sichi added a comment - Linking HIVE-1533 for Zookeeper.
          John Sichi made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          Hide
          John Sichi added a comment -

          Starting on this one with HBase 0.89 and zookeeper 3.3.1 as part of addressing HIVE-1716.

          Show
          John Sichi added a comment - Starting on this one with HBase 0.89 and zookeeper 3.3.1 as part of addressing HIVE-1716 .
          Hide
          John Sichi added a comment -

          This patch (HIVE-1235.1.patch) is not ready for review yet, and ant test is hitting an OOM in one of the ivy calls right before running the HBase tests; just posting it here as a checkpoint. To apply, you also need:

          svn remove lib/hbase-0.20.3-test.jar
          svn remove lib/hbase-0.20.3.jar
          svn remove lib/zookeeper-3.2.2.jar

          Show
          John Sichi added a comment - This patch ( HIVE-1235 .1.patch) is not ready for review yet, and ant test is hitting an OOM in one of the ivy calls right before running the HBase tests; just posting it here as a checkpoint. To apply, you also need: svn remove lib/hbase-0.20.3-test.jar svn remove lib/hbase-0.20.3.jar svn remove lib/zookeeper-3.2.2.jar
          John Sichi made changes -
          Attachment HIVE-1235.1.patch [ 12469334 ]
          Hide
          John Sichi added a comment -

          Note that transitive dependencies were pulling down all kinds of odds and ends we don't need, so I had to turn them off and then get the only hard dependency (guava) explicitly. Slightly defeating the purpose of ivy, but...

          Show
          John Sichi added a comment - Note that transitive dependencies were pulling down all kinds of odds and ends we don't need, so I had to turn them off and then get the only hard dependency (guava) explicitly. Slightly defeating the purpose of ivy, but...
          Hide
          John Sichi added a comment -

          HIVE-1235.2.patch passes tests. A couple of the HBase tests had broken due to recent changes, so this fixes those. It also gets rid of redundant ivy-retrieve for hadoopcore (which is what was causing the OOM's).

          Show
          John Sichi added a comment - HIVE-1235 .2.patch passes tests. A couple of the HBase tests had broken due to recent changes, so this fixes those. It also gets rid of redundant ivy-retrieve for hadoopcore (which is what was causing the OOM's).
          John Sichi made changes -
          Attachment HIVE-1235.2.patch [ 12469718 ]
          Hide
          John Sichi added a comment -

          Review board:

          https://reviews.apache.org/r/367/

          Note this also makes the hbase-handler build non-conditional since we've dropped support for pre-0.20.

          Show
          John Sichi added a comment - Review board: https://reviews.apache.org/r/367/ Note this also makes the hbase-handler build non-conditional since we've dropped support for pre-0.20.
          John Sichi made changes -
          Status In Progress [ 3 ] Patch Available [ 10002 ]
          Fix Version/s 0.7.0 [ 12315150 ]
          Hide
          John Sichi added a comment -

          Would be nice to get this one in for 0.7.

          Show
          John Sichi added a comment - Would be nice to get this one in for 0.7.
          Hide
          John Sichi added a comment -

          Oops, got one failure in TestHBaseSerDe due to a change in the Result constructor contract (it now expects the kv's to be presorted). I'll uploaded another patch.

          Show
          John Sichi added a comment - Oops, got one failure in TestHBaseSerDe due to a change in the Result constructor contract (it now expects the kv's to be presorted). I'll uploaded another patch.
          John Sichi made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          John Sichi added a comment -

          I updated review board as well.

          Show
          John Sichi added a comment - I updated review board as well.
          John Sichi made changes -
          Attachment HIVE-1235.3.patch [ 12469846 ]
          Hide
          John Sichi added a comment -

          HIVE-1235.3.patch

          Show
          John Sichi added a comment - HIVE-1235 .3.patch
          John Sichi made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Carl Steinbach added a comment -

          +1. Will commit if tests pass.

          Show
          Carl Steinbach added a comment - +1. Will commit if tests pass.
          Hide
          Carl Steinbach added a comment -

          Committed. Thanks John!

          Show
          Carl Steinbach added a comment - Committed. Thanks John!
          Carl Steinbach made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Resolution Fixed [ 1 ]
          Carl Steinbach made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open In Progress In Progress
          320d 3h 9m 1 John Sichi 25/Jan/11 00:15
          In Progress In Progress Patch Available Patch Available
          4d 2h 25m 1 John Sichi 29/Jan/11 02:40
          Patch Available Patch Available Open Open
          2d 16h 23m 1 John Sichi 31/Jan/11 19:04
          Open Open Patch Available Patch Available
          18m 34s 1 John Sichi 31/Jan/11 19:22
          Patch Available Patch Available Resolved Resolved
          11h 11m 1 Carl Steinbach 01/Feb/11 06:34
          Resolved Resolved Closed Closed
          318d 17h 25m 1 Carl Steinbach 16/Dec/11 23:59

            People

            • Assignee:
              John Sichi
              Reporter:
              John Sichi
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development