Hive
  1. Hive
  2. HIVE-2244

Add a Plugin Developer Kit to Hive

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.0
    • Component/s: UDF
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    1. HIVE-2244.1.patch
      32 kB
      John Sichi
    2. HIVE-2244.patch
      29 kB
      John Sichi

      Issue Links

        Activity

        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/977/
        -----------------------------------------------------------

        Review request for hive.

        Summary
        -------

        Review request from JVS.

        This addresses bug HIVE-2244.
        https://issues.apache.org/jira/browse/HIVE-2244

        Diffs


        http://svn.apache.org/repos/asf/hive/trunk/build.xml 1139984
        http://svn.apache.org/repos/asf/hive/trunk/pdk/build.xml PRE-CREATION
        http://svn.apache.org/repos/asf/hive/trunk/pdk/scripts/README PRE-CREATION
        http://svn.apache.org/repos/asf/hive/trunk/pdk/scripts/build-plugin.xml PRE-CREATION
        http://svn.apache.org/repos/asf/hive/trunk/pdk/scripts/class-registration.xsl PRE-CREATION
        http://svn.apache.org/repos/asf/hive/trunk/pdk/src/java/org/apache/hive/pdk/FunctionExtractor.java PRE-CREATION
        http://svn.apache.org/repos/asf/hive/trunk/pdk/src/java/org/apache/hive/pdk/HivePdkUnitTest.java PRE-CREATION
        http://svn.apache.org/repos/asf/hive/trunk/pdk/src/java/org/apache/hive/pdk/HivePdkUnitTests.java PRE-CREATION
        http://svn.apache.org/repos/asf/hive/trunk/pdk/src/java/org/apache/hive/pdk/PluginTest.java PRE-CREATION
        http://svn.apache.org/repos/asf/hive/trunk/pdk/test-plugin/build.xml PRE-CREATION
        http://svn.apache.org/repos/asf/hive/trunk/pdk/test-plugin/src/org/apache/hive/pdktest/Rot13.java PRE-CREATION
        http://svn.apache.org/repos/asf/hive/trunk/pdk/test-plugin/test/cleanup.sql PRE-CREATION
        http://svn.apache.org/repos/asf/hive/trunk/pdk/test-plugin/test/onerow.txt PRE-CREATION
        http://svn.apache.org/repos/asf/hive/trunk/pdk/test-plugin/test/setup.sql PRE-CREATION

        Diff: https://reviews.apache.org/r/977/diff

        Testing
        -------

        Thanks,

        John

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/977/ ----------------------------------------------------------- Review request for hive. Summary ------- Review request from JVS. This addresses bug HIVE-2244 . https://issues.apache.org/jira/browse/HIVE-2244 Diffs http://svn.apache.org/repos/asf/hive/trunk/build.xml 1139984 http://svn.apache.org/repos/asf/hive/trunk/pdk/build.xml PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/pdk/scripts/README PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/pdk/scripts/build-plugin.xml PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/pdk/scripts/class-registration.xsl PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/pdk/src/java/org/apache/hive/pdk/FunctionExtractor.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/pdk/src/java/org/apache/hive/pdk/HivePdkUnitTest.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/pdk/src/java/org/apache/hive/pdk/HivePdkUnitTests.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/pdk/src/java/org/apache/hive/pdk/PluginTest.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/pdk/test-plugin/build.xml PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/pdk/test-plugin/src/org/apache/hive/pdktest/Rot13.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/pdk/test-plugin/test/cleanup.sql PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/pdk/test-plugin/test/onerow.txt PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/pdk/test-plugin/test/setup.sql PRE-CREATION Diff: https://reviews.apache.org/r/977/diff Testing ------- Thanks, John
        Hide
        John Sichi added a comment -
        Show
        John Sichi added a comment - Review board at https://reviews.apache.org/r/977/
        Hide
        John Sichi added a comment -

        I'm tweaking a few things based on experience at the mini-hackathon, so this is just a first draft.

        Show
        John Sichi added a comment - I'm tweaking a few things based on experience at the mini-hackathon, so this is just a first draft.
        Hide
        JunHo Cho added a comment -

        I tested my udf using pdk. but it was slow because invoking cli repeatedly.
        So, I just changed PluginTest to use Driver directly not using invoking cli.
        then, I got a following result

        test case : Rot13 , Rot12 (just changed class-name of Rot13)
        total test count : 4

        pdk using cli : 2min 44s
        pdk using driver : 1min 49s

        I think that we need to improve it's performance in this release to apply many udf.

        Show
        JunHo Cho added a comment - I tested my udf using pdk. but it was slow because invoking cli repeatedly. So, I just changed PluginTest to use Driver directly not using invoking cli. then, I got a following result test case : Rot13 , Rot12 (just changed class-name of Rot13) total test count : 4 pdk using cli : 2min 44s pdk using driver : 1min 49s I think that we need to improve it's performance in this release to apply many udf.
        Hide
        John Sichi added a comment -

        I agree; I'm working on a similar optimization.

        Show
        John Sichi added a comment - I agree; I'm working on a similar optimization.
        Hide
        John Sichi added a comment -

        HIVE-2244.1.patch has the optimization.

        Show
        John Sichi added a comment - HIVE-2244 .1.patch has the optimization.
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/2030/
        -----------------------------------------------------------

        Review request for hive.

        Summary
        -------

        HIVE-2244

        This addresses bug HIVE-2244.
        https://issues.apache.org/jira/browse/HIVE-2244

        Diffs


        http://svn.apache.org/repos/asf/hive/trunk/pdk/test-plugin/build.xml PRE-CREATION
        http://svn.apache.org/repos/asf/hive/trunk/pdk/test-plugin/src/org/apache/hive/pdktest/Rot13.java PRE-CREATION
        http://svn.apache.org/repos/asf/hive/trunk/pdk/test-plugin/test/cleanup.sql PRE-CREATION
        http://svn.apache.org/repos/asf/hive/trunk/pdk/test-plugin/test/onerow.txt PRE-CREATION
        http://svn.apache.org/repos/asf/hive/trunk/pdk/test-plugin/test/setup.sql PRE-CREATION
        http://svn.apache.org/repos/asf/hive/trunk/pdk/src/java/org/apache/hive/pdk/PluginTest.java PRE-CREATION
        http://svn.apache.org/repos/asf/hive/trunk/pdk/src/java/org/apache/hive/pdk/HivePdkUnitTest.java PRE-CREATION
        http://svn.apache.org/repos/asf/hive/trunk/pdk/src/java/org/apache/hive/pdk/HivePdkUnitTests.java PRE-CREATION
        http://svn.apache.org/repos/asf/hive/trunk/pdk/src/java/org/apache/hive/pdk/FunctionExtractor.java PRE-CREATION
        http://svn.apache.org/repos/asf/hive/trunk/pdk/scripts/build-plugin.xml PRE-CREATION
        http://svn.apache.org/repos/asf/hive/trunk/pdk/scripts/class-registration.xsl PRE-CREATION
        http://svn.apache.org/repos/asf/hive/trunk/pdk/scripts/README PRE-CREATION
        http://svn.apache.org/repos/asf/hive/trunk/pdk/build.xml PRE-CREATION
        http://svn.apache.org/repos/asf/hive/trunk/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java 1174460
        http://svn.apache.org/repos/asf/hive/trunk/build.xml 1174460

        Diff: https://reviews.apache.org/r/2030/diff

        Testing
        -------

        Thanks,

        John

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2030/ ----------------------------------------------------------- Review request for hive. Summary ------- HIVE-2244 This addresses bug HIVE-2244 . https://issues.apache.org/jira/browse/HIVE-2244 Diffs http://svn.apache.org/repos/asf/hive/trunk/pdk/test-plugin/build.xml PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/pdk/test-plugin/src/org/apache/hive/pdktest/Rot13.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/pdk/test-plugin/test/cleanup.sql PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/pdk/test-plugin/test/onerow.txt PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/pdk/test-plugin/test/setup.sql PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/pdk/src/java/org/apache/hive/pdk/PluginTest.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/pdk/src/java/org/apache/hive/pdk/HivePdkUnitTest.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/pdk/src/java/org/apache/hive/pdk/HivePdkUnitTests.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/pdk/src/java/org/apache/hive/pdk/FunctionExtractor.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/pdk/scripts/build-plugin.xml PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/pdk/scripts/class-registration.xsl PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/pdk/scripts/README PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/pdk/build.xml PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java 1174460 http://svn.apache.org/repos/asf/hive/trunk/build.xml 1174460 Diff: https://reviews.apache.org/r/2030/diff Testing ------- Thanks, John
        Hide
        John Sichi added a comment -
        Show
        John Sichi added a comment - Review Board at https://reviews.apache.org/r/2030
        Hide
        Carl Steinbach added a comment -

        +1. Will commit if tests pass.

        Show
        Carl Steinbach added a comment - +1. Will commit if tests pass.
        Hide
        Edward Capriolo added a comment -

        I have a similar approach on github.

        https://github.com/edwardcapriolo/hive_test

        It pulls down all hive deps from maven and only requires a local copy of hadoop because hive always seems to fork a call to hadoop jar.

        Show
        Edward Capriolo added a comment - I have a similar approach on github. https://github.com/edwardcapriolo/hive_test It pulls down all hive deps from maven and only requires a local copy of hadoop because hive always seems to fork a call to hadoop jar.
        Hide
        Carl Steinbach added a comment -

        Committed to trunk and branch-0.8. Thanks John!

        Show
        Carl Steinbach added a comment - Committed to trunk and branch-0.8. Thanks John!
        Hide
        Hudson added a comment -

        Integrated in Hive-0.8.0-SNAPSHOT-h0.21 #38 (See https://builds.apache.org/job/Hive-0.8.0-SNAPSHOT-h0.21/38/)
        HIVE-2244. Add a Plugin Developer Kit to Hive (John Sichi via cws)

        cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1177386
        Files :

        • /hive/branches/branch-0.8/build.xml
        • /hive/branches/branch-0.8/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java
        • /hive/branches/branch-0.8/pdk
        • /hive/branches/branch-0.8/pdk/build.xml
        • /hive/branches/branch-0.8/pdk/scripts
        • /hive/branches/branch-0.8/pdk/scripts/README
        • /hive/branches/branch-0.8/pdk/scripts/build-plugin.xml
        • /hive/branches/branch-0.8/pdk/scripts/class-registration.xsl
        • /hive/branches/branch-0.8/pdk/src
        • /hive/branches/branch-0.8/pdk/src/java
        • /hive/branches/branch-0.8/pdk/src/java/org
        • /hive/branches/branch-0.8/pdk/src/java/org/apache
        • /hive/branches/branch-0.8/pdk/src/java/org/apache/hive
        • /hive/branches/branch-0.8/pdk/src/java/org/apache/hive/pdk
        • /hive/branches/branch-0.8/pdk/src/java/org/apache/hive/pdk/FunctionExtractor.java
        • /hive/branches/branch-0.8/pdk/src/java/org/apache/hive/pdk/HivePdkUnitTest.java
        • /hive/branches/branch-0.8/pdk/src/java/org/apache/hive/pdk/HivePdkUnitTests.java
        • /hive/branches/branch-0.8/pdk/src/java/org/apache/hive/pdk/PluginTest.java
        • /hive/branches/branch-0.8/pdk/test-plugin
        • /hive/branches/branch-0.8/pdk/test-plugin/build.xml
        • /hive/branches/branch-0.8/pdk/test-plugin/src
        • /hive/branches/branch-0.8/pdk/test-plugin/src/org
        • /hive/branches/branch-0.8/pdk/test-plugin/src/org/apache
        • /hive/branches/branch-0.8/pdk/test-plugin/src/org/apache/hive
        • /hive/branches/branch-0.8/pdk/test-plugin/src/org/apache/hive/pdktest
        • /hive/branches/branch-0.8/pdk/test-plugin/src/org/apache/hive/pdktest/Rot13.java
        • /hive/branches/branch-0.8/pdk/test-plugin/test
        • /hive/branches/branch-0.8/pdk/test-plugin/test/cleanup.sql
        • /hive/branches/branch-0.8/pdk/test-plugin/test/onerow.txt
        • /hive/branches/branch-0.8/pdk/test-plugin/test/setup.sql
        Show
        Hudson added a comment - Integrated in Hive-0.8.0-SNAPSHOT-h0.21 #38 (See https://builds.apache.org/job/Hive-0.8.0-SNAPSHOT-h0.21/38/ ) HIVE-2244 . Add a Plugin Developer Kit to Hive (John Sichi via cws) cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1177386 Files : /hive/branches/branch-0.8/build.xml /hive/branches/branch-0.8/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java /hive/branches/branch-0.8/pdk /hive/branches/branch-0.8/pdk/build.xml /hive/branches/branch-0.8/pdk/scripts /hive/branches/branch-0.8/pdk/scripts/README /hive/branches/branch-0.8/pdk/scripts/build-plugin.xml /hive/branches/branch-0.8/pdk/scripts/class-registration.xsl /hive/branches/branch-0.8/pdk/src /hive/branches/branch-0.8/pdk/src/java /hive/branches/branch-0.8/pdk/src/java/org /hive/branches/branch-0.8/pdk/src/java/org/apache /hive/branches/branch-0.8/pdk/src/java/org/apache/hive /hive/branches/branch-0.8/pdk/src/java/org/apache/hive/pdk /hive/branches/branch-0.8/pdk/src/java/org/apache/hive/pdk/FunctionExtractor.java /hive/branches/branch-0.8/pdk/src/java/org/apache/hive/pdk/HivePdkUnitTest.java /hive/branches/branch-0.8/pdk/src/java/org/apache/hive/pdk/HivePdkUnitTests.java /hive/branches/branch-0.8/pdk/src/java/org/apache/hive/pdk/PluginTest.java /hive/branches/branch-0.8/pdk/test-plugin /hive/branches/branch-0.8/pdk/test-plugin/build.xml /hive/branches/branch-0.8/pdk/test-plugin/src /hive/branches/branch-0.8/pdk/test-plugin/src/org /hive/branches/branch-0.8/pdk/test-plugin/src/org/apache /hive/branches/branch-0.8/pdk/test-plugin/src/org/apache/hive /hive/branches/branch-0.8/pdk/test-plugin/src/org/apache/hive/pdktest /hive/branches/branch-0.8/pdk/test-plugin/src/org/apache/hive/pdktest/Rot13.java /hive/branches/branch-0.8/pdk/test-plugin/test /hive/branches/branch-0.8/pdk/test-plugin/test/cleanup.sql /hive/branches/branch-0.8/pdk/test-plugin/test/onerow.txt /hive/branches/branch-0.8/pdk/test-plugin/test/setup.sql
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-h0.21 #983 (See https://builds.apache.org/job/Hive-trunk-h0.21/983/)
        HIVE-2244. Add a Plugin Developer Kit to Hive (John Sichi via cws)

        cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1177382
        Files :

        • /hive/trunk/build.xml
        • /hive/trunk/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java
        • /hive/trunk/pdk
        • /hive/trunk/pdk/build.xml
        • /hive/trunk/pdk/scripts
        • /hive/trunk/pdk/scripts/README
        • /hive/trunk/pdk/scripts/build-plugin.xml
        • /hive/trunk/pdk/scripts/class-registration.xsl
        • /hive/trunk/pdk/src
        • /hive/trunk/pdk/src/java
        • /hive/trunk/pdk/src/java/org
        • /hive/trunk/pdk/src/java/org/apache
        • /hive/trunk/pdk/src/java/org/apache/hive
        • /hive/trunk/pdk/src/java/org/apache/hive/pdk
        • /hive/trunk/pdk/src/java/org/apache/hive/pdk/FunctionExtractor.java
        • /hive/trunk/pdk/src/java/org/apache/hive/pdk/HivePdkUnitTest.java
        • /hive/trunk/pdk/src/java/org/apache/hive/pdk/HivePdkUnitTests.java
        • /hive/trunk/pdk/src/java/org/apache/hive/pdk/PluginTest.java
        • /hive/trunk/pdk/test-plugin
        • /hive/trunk/pdk/test-plugin/build.xml
        • /hive/trunk/pdk/test-plugin/src
        • /hive/trunk/pdk/test-plugin/src/org
        • /hive/trunk/pdk/test-plugin/src/org/apache
        • /hive/trunk/pdk/test-plugin/src/org/apache/hive
        • /hive/trunk/pdk/test-plugin/src/org/apache/hive/pdktest
        • /hive/trunk/pdk/test-plugin/src/org/apache/hive/pdktest/Rot13.java
        • /hive/trunk/pdk/test-plugin/test
        • /hive/trunk/pdk/test-plugin/test/cleanup.sql
        • /hive/trunk/pdk/test-plugin/test/onerow.txt
        • /hive/trunk/pdk/test-plugin/test/setup.sql
        Show
        Hudson added a comment - Integrated in Hive-trunk-h0.21 #983 (See https://builds.apache.org/job/Hive-trunk-h0.21/983/ ) HIVE-2244 . Add a Plugin Developer Kit to Hive (John Sichi via cws) cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1177382 Files : /hive/trunk/build.xml /hive/trunk/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java /hive/trunk/pdk /hive/trunk/pdk/build.xml /hive/trunk/pdk/scripts /hive/trunk/pdk/scripts/README /hive/trunk/pdk/scripts/build-plugin.xml /hive/trunk/pdk/scripts/class-registration.xsl /hive/trunk/pdk/src /hive/trunk/pdk/src/java /hive/trunk/pdk/src/java/org /hive/trunk/pdk/src/java/org/apache /hive/trunk/pdk/src/java/org/apache/hive /hive/trunk/pdk/src/java/org/apache/hive/pdk /hive/trunk/pdk/src/java/org/apache/hive/pdk/FunctionExtractor.java /hive/trunk/pdk/src/java/org/apache/hive/pdk/HivePdkUnitTest.java /hive/trunk/pdk/src/java/org/apache/hive/pdk/HivePdkUnitTests.java /hive/trunk/pdk/src/java/org/apache/hive/pdk/PluginTest.java /hive/trunk/pdk/test-plugin /hive/trunk/pdk/test-plugin/build.xml /hive/trunk/pdk/test-plugin/src /hive/trunk/pdk/test-plugin/src/org /hive/trunk/pdk/test-plugin/src/org/apache /hive/trunk/pdk/test-plugin/src/org/apache/hive /hive/trunk/pdk/test-plugin/src/org/apache/hive/pdktest /hive/trunk/pdk/test-plugin/src/org/apache/hive/pdktest/Rot13.java /hive/trunk/pdk/test-plugin/test /hive/trunk/pdk/test-plugin/test/cleanup.sql /hive/trunk/pdk/test-plugin/test/onerow.txt /hive/trunk/pdk/test-plugin/test/setup.sql

          People

          • Assignee:
            John Sichi
            Reporter:
            John Sichi
          • Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development