HBase
  1. HBase
  2. HBASE-2588

Add easier way to ship hbase dependencies to MR cluster with job

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.90.0
    • Fix Version/s: 0.90.0
    • Component/s: Client, mapreduce
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      It's a bit of a pain to ship the hbase, ZK, etc jars onto the cluster. This JIRA is to provide a nice programattic way to get these jars into the classpath of the MR child.

      1. hbase-2588.txt
        7 kB
        Todd Lipcon
      2. hbase-2588.txt
        8 kB
        Todd Lipcon

        Activity

        Hide
        Todd Lipcon added a comment -

        this should be unit tested, but appears to kinda work. We should also integrate it into the other builtin programs (eg import/export/etc). And possibly move it to a different container class. Just wanted to throw it up for people to look at.

        Show
        Todd Lipcon added a comment - this should be unit tested, but appears to kinda work. We should also integrate it into the other builtin programs (eg import/export/etc). And possibly move it to a different container class. Just wanted to throw it up for people to look at.
        Hide
        Lars George added a comment -

        As discussed on IRC, I still do not get how this can work without a final

        conf.set("tmpjars", tempJars);

        in getDependenyJars(). Am I missing something?

        Show
        Lars George added a comment - As discussed on IRC, I still do not get how this can work without a final conf.set( "tmpjars" , tempJars); in getDependenyJars(). Am I missing something?
        Hide
        stack added a comment -

        Thats a nice trick there Todd. It works? (At end of addDependencyJars, it makes a + tmpJars += "," + StringUtils.join(",", jars); but doesn't seem to do anything w/ it) How you going to make a unit test for that? It'll need to start so much stuff, it'll be hard to figure when its working or not (we have start minimrcluster in HBaseTestingUtility....) Just commit?

        Show
        stack added a comment - Thats a nice trick there Todd. It works? (At end of addDependencyJars, it makes a + tmpJars += "," + StringUtils.join(",", jars); but doesn't seem to do anything w/ it) How you going to make a unit test for that? It'll need to start so much stuff, it'll be hard to figure when its working or not (we have start minimrcluster in HBaseTestingUtility....) Just commit?
        Hide
        Todd Lipcon added a comment -

        yea, as Larse points out I somehow missed a line in this patch

        It appears to vaguely work on my system, but let's not commit yet til someone else verifies it's not just hocus pocus that does nothing. I can also write a unit test that at least verifies tmpjars gets set!

        Show
        Todd Lipcon added a comment - yea, as Larse points out I somehow missed a line in this patch It appears to vaguely work on my system, but let's not commit yet til someone else verifies it's not just hocus pocus that does nothing. I can also write a unit test that at least verifies tmpjars gets set!
        Hide
        stack added a comment -

        .bq I can also write a unit test that at least verifies tmpjars gets set!

        That seems fine. More than that would be tough to do and not worth the investment (IMO).

        Show
        stack added a comment - .bq I can also write a unit test that at least verifies tmpjars gets set! That seems fine. More than that would be tough to do and not worth the investment (IMO).
        Hide
        HBase Review Board added a comment -

        Message from: "Todd Lipcon" <todd@cloudera.com>

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        http://review.hbase.org/r/111/
        -----------------------------------------------------------

        Review request for hbase, stack and Lars George.

        Summary
        -------

        HBASE-2588. Provide a utility function to ship HBase dependency jars to the cluster in classpath

        Fix for tmpjars thing

        Unit test for HBASE-2588

        This addresses bug HBASE-2588.
        http://issues.apache.org/jira/browse/HBASE-2588

        Diffs


        src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java b332280
        src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java d76c75e
        src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java d8b11db

        Diff: http://review.hbase.org/r/111/diff

        Testing
        -------

        Been using with importtsv from bulk load for a while.

        Thanks,

        Todd

        Show
        HBase Review Board added a comment - Message from: "Todd Lipcon" <todd@cloudera.com> ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: http://review.hbase.org/r/111/ ----------------------------------------------------------- Review request for hbase, stack and Lars George. Summary ------- HBASE-2588 . Provide a utility function to ship HBase dependency jars to the cluster in classpath Fix for tmpjars thing Unit test for HBASE-2588 This addresses bug HBASE-2588 . http://issues.apache.org/jira/browse/HBASE-2588 Diffs src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java b332280 src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java d76c75e src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java d8b11db Diff: http://review.hbase.org/r/111/diff Testing ------- Been using with importtsv from bulk load for a while. Thanks, Todd
        Hide
        HBase Review Board added a comment -

        Message from: "Jonathan Gray" <jgray@facebook.com>

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        http://review.hbase.org/r/111/#review117
        -----------------------------------------------------------

        Ship it!

        This is some fancy stuff.

        • Jonathan
        Show
        HBase Review Board added a comment - Message from: "Jonathan Gray" <jgray@facebook.com> ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: http://review.hbase.org/r/111/#review117 ----------------------------------------------------------- Ship it! This is some fancy stuff. Jonathan
        Hide
        Todd Lipcon added a comment -

        attaching patch from the review - will commit this one.

        Show
        Todd Lipcon added a comment - attaching patch from the review - will commit this one.
        Hide
        Todd Lipcon added a comment -

        Committed, thanks for review, Lars and Jonathan.

        Show
        Todd Lipcon added a comment - Committed, thanks for review, Lars and Jonathan.
        Hide
        Todd Lipcon added a comment -

        Oops, I think this new test is going to break the build momentarily, since guava isn't on the classpath until HBASE-1923 is committed. Sorry!

        Show
        Todd Lipcon added a comment - Oops, I think this new test is going to break the build momentarily, since guava isn't on the classpath until HBASE-1923 is committed. Sorry!
        Hide
        HBase Review Board added a comment -

        Message from: "Lars George" <larsgeorge@apache.org>

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        http://review.hbase.org/r/111/#review120
        -----------------------------------------------------------

        Ship it!

        Looks good!

        • Lars
        Show
        HBase Review Board added a comment - Message from: "Lars George" <larsgeorge@apache.org> ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: http://review.hbase.org/r/111/#review120 ----------------------------------------------------------- Ship it! Looks good! Lars

          People

          • Assignee:
            Todd Lipcon
            Reporter:
            Todd Lipcon
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development