Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.0
    • Component/s: UDF
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Now that we have a PDK, we can make it easier to add builtin functions to Hive by putting them in a plugin which automatically gets loaded by Hive. This issue will add the necessary framework and one example function; then new functions can be added here, and over time we could migrate old ones here if desired.

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-h0.21 #1125 (See https://builds.apache.org/job/Hive-trunk-h0.21/1125/)
          Fix eclipse classpath template broken in HIVE-2523.
          (Carl Steinbach via jvs)

          jvs : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1210599
          Files :

          • /hive/trunk/eclipse-templates/.classpath
          Show
          Hudson added a comment - Integrated in Hive-trunk-h0.21 #1125 (See https://builds.apache.org/job/Hive-trunk-h0.21/1125/ ) Fix eclipse classpath template broken in HIVE-2523 . (Carl Steinbach via jvs) jvs : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1210599 Files : /hive/trunk/eclipse-templates/.classpath
          Hide
          Phabricator added a comment -

          jsichi has abandoned the revision "HIVE-2523 [jira] add a new builtins subproject".

          REVISION DETAIL
          https://reviews.facebook.net/D267

          Show
          Phabricator added a comment - jsichi has abandoned the revision " HIVE-2523 [jira] add a new builtins subproject". REVISION DETAIL https://reviews.facebook.net/D267
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-h0.21 #1119 (See https://builds.apache.org/job/Hive-trunk-h0.21/1119/)
          HIVE-2523. Add a new builtins subproject (John Sichi via cws)

          cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1209384
          Files :

          • /hive/trunk/bin/hive
          • /hive/trunk/build.xml
          • /hive/trunk/builtins
          • /hive/trunk/builtins/build-plugin.xml
          • /hive/trunk/builtins/build.xml
          • /hive/trunk/builtins/ivy.xml
          • /hive/trunk/builtins/src
          • /hive/trunk/builtins/src/org
          • /hive/trunk/builtins/src/org/apache
          • /hive/trunk/builtins/src/org/apache/hive
          • /hive/trunk/builtins/src/org/apache/hive/builtins
          • /hive/trunk/builtins/src/org/apache/hive/builtins/BuiltinUtils.java
          • /hive/trunk/builtins/src/org/apache/hive/builtins/UDAFUnionMap.java
          • /hive/trunk/builtins/test
          • /hive/trunk/builtins/test/cleanup.sql
          • /hive/trunk/builtins/test/iris.txt
          • /hive/trunk/builtins/test/onerow.txt
          • /hive/trunk/builtins/test/setup.sql
          • /hive/trunk/eclipse-templates/.classpath
          • /hive/trunk/pdk/scripts/build-plugin.xml
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java
          • /hive/trunk/ql/src/test/results/clientpositive/show_functions.q.out
          Show
          Hudson added a comment - Integrated in Hive-trunk-h0.21 #1119 (See https://builds.apache.org/job/Hive-trunk-h0.21/1119/ ) HIVE-2523 . Add a new builtins subproject (John Sichi via cws) cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1209384 Files : /hive/trunk/bin/hive /hive/trunk/build.xml /hive/trunk/builtins /hive/trunk/builtins/build-plugin.xml /hive/trunk/builtins/build.xml /hive/trunk/builtins/ivy.xml /hive/trunk/builtins/src /hive/trunk/builtins/src/org /hive/trunk/builtins/src/org/apache /hive/trunk/builtins/src/org/apache/hive /hive/trunk/builtins/src/org/apache/hive/builtins /hive/trunk/builtins/src/org/apache/hive/builtins/BuiltinUtils.java /hive/trunk/builtins/src/org/apache/hive/builtins/UDAFUnionMap.java /hive/trunk/builtins/test /hive/trunk/builtins/test/cleanup.sql /hive/trunk/builtins/test/iris.txt /hive/trunk/builtins/test/onerow.txt /hive/trunk/builtins/test/setup.sql /hive/trunk/eclipse-templates/.classpath /hive/trunk/pdk/scripts/build-plugin.xml /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java /hive/trunk/ql/src/test/results/clientpositive/show_functions.q.out
          Hide
          Carl Steinbach added a comment -

          Committed to trunk. Thanks John!

          Show
          Carl Steinbach added a comment - Committed to trunk. Thanks John!
          Hide
          Phabricator added a comment -

          cwsteinbach has accepted the revision "HIVE-2523 [jira] add a new builtins subproject".

          +1. Will commit if tests pass.

          REVISION DETAIL
          https://reviews.facebook.net/D267

          Show
          Phabricator added a comment - cwsteinbach has accepted the revision " HIVE-2523 [jira] add a new builtins subproject". +1. Will commit if tests pass. REVISION DETAIL https://reviews.facebook.net/D267
          Hide
          Phabricator added a comment -

          jsichi updated the revision "HIVE-2523 [jira] add a new builtins subproject".
          Reviewers: JIRA

          Rebase and fix a problem which was preventing this from passing parallel tests.

          REVISION DETAIL
          https://reviews.facebook.net/D267

          AFFECTED FILES
          eclipse-templates/.classpath
          builtins
          builtins/test
          builtins/test/iris.txt
          builtins/test/cleanup.sql
          builtins/test/onerow.txt
          builtins/test/setup.sql
          builtins/ivy.xml
          builtins/src
          builtins/src/org
          builtins/src/org/apache
          builtins/src/org/apache/hive
          builtins/src/org/apache/hive/builtins
          builtins/src/org/apache/hive/builtins/UDAFUnionMap.java
          builtins/src/org/apache/hive/builtins/BuiltinUtils.java
          builtins/build-plugin.xml
          builtins/build.xml
          build.xml
          bin/hive
          ql/src/test/results/clientpositive/show_functions.q.out
          ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
          pdk/scripts/build-plugin.xml

          Show
          Phabricator added a comment - jsichi updated the revision " HIVE-2523 [jira] add a new builtins subproject". Reviewers: JIRA Rebase and fix a problem which was preventing this from passing parallel tests. REVISION DETAIL https://reviews.facebook.net/D267 AFFECTED FILES eclipse-templates/.classpath builtins builtins/test builtins/test/iris.txt builtins/test/cleanup.sql builtins/test/onerow.txt builtins/test/setup.sql builtins/ivy.xml builtins/src builtins/src/org builtins/src/org/apache builtins/src/org/apache/hive builtins/src/org/apache/hive/builtins builtins/src/org/apache/hive/builtins/UDAFUnionMap.java builtins/src/org/apache/hive/builtins/BuiltinUtils.java builtins/build-plugin.xml builtins/build.xml build.xml bin/hive ql/src/test/results/clientpositive/show_functions.q.out ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java pdk/scripts/build-plugin.xml
          Hide
          John Sichi added a comment -

          Tests passed for me, yay!

          Show
          John Sichi added a comment - Tests passed for me, yay!
          Hide
          Phabricator added a comment -

          jsichi updated the revision "HIVE-2523 [jira] add a new builtins subproject".
          Reviewers: JIRA

          Attempt at fixing leak...gonna run through tests again.

          REVISION DETAIL
          https://reviews.facebook.net/D267

          AFFECTED FILES
          eclipse-templates/.classpath
          builtins
          builtins/test
          builtins/test/iris.txt
          builtins/test/cleanup.sql
          builtins/test/onerow.txt
          builtins/test/setup.sql
          builtins/ivy.xml
          builtins/src
          builtins/src/org
          builtins/src/org/apache
          builtins/src/org/apache/hive
          builtins/src/org/apache/hive/builtins
          builtins/src/org/apache/hive/builtins/UDAFUnionMap.java
          builtins/src/org/apache/hive/builtins/BuiltinUtils.java
          builtins/build-plugin.xml
          builtins/build.xml
          build.xml
          bin/hive
          ql/src/test/results/clientpositive/show_functions.q.out
          ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
          pdk/scripts/build-plugin.xml

          Show
          Phabricator added a comment - jsichi updated the revision " HIVE-2523 [jira] add a new builtins subproject". Reviewers: JIRA Attempt at fixing leak...gonna run through tests again. REVISION DETAIL https://reviews.facebook.net/D267 AFFECTED FILES eclipse-templates/.classpath builtins builtins/test builtins/test/iris.txt builtins/test/cleanup.sql builtins/test/onerow.txt builtins/test/setup.sql builtins/ivy.xml builtins/src builtins/src/org builtins/src/org/apache builtins/src/org/apache/hive builtins/src/org/apache/hive/builtins builtins/src/org/apache/hive/builtins/UDAFUnionMap.java builtins/src/org/apache/hive/builtins/BuiltinUtils.java builtins/build-plugin.xml builtins/build.xml build.xml bin/hive ql/src/test/results/clientpositive/show_functions.q.out ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java pdk/scripts/build-plugin.xml
          Hide
          John Sichi added a comment -

          Tests are failing due to a timeout after 5 hours, so I think I must have a leak of some kind...debugging.

          Show
          John Sichi added a comment - Tests are failing due to a timeout after 5 hours, so I think I must have a leak of some kind...debugging.
          Hide
          Phabricator added a comment -

          jsichi updated the revision "HIVE-2523 [jira] add a new builtins subproject".
          Reviewers: JIRA

          Fix some tests, still need to run through the whole suite again.

          REVISION DETAIL
          https://reviews.facebook.net/D267

          AFFECTED FILES
          eclipse-templates/.classpath
          builtins
          builtins/test
          builtins/test/iris.txt
          builtins/test/cleanup.sql
          builtins/test/onerow.txt
          builtins/test/setup.sql
          builtins/ivy.xml
          builtins/src
          builtins/src/org
          builtins/src/org/apache
          builtins/src/org/apache/hive
          builtins/src/org/apache/hive/builtins
          builtins/src/org/apache/hive/builtins/UDAFUnionMap.java
          builtins/src/org/apache/hive/builtins/BuiltinUtils.java
          builtins/build-plugin.xml
          builtins/build.xml
          build.xml
          bin/hive
          ql/src/test/results/clientpositive/show_functions.q.out
          ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
          pdk/scripts/build-plugin.xml

          Show
          Phabricator added a comment - jsichi updated the revision " HIVE-2523 [jira] add a new builtins subproject". Reviewers: JIRA Fix some tests, still need to run through the whole suite again. REVISION DETAIL https://reviews.facebook.net/D267 AFFECTED FILES eclipse-templates/.classpath builtins builtins/test builtins/test/iris.txt builtins/test/cleanup.sql builtins/test/onerow.txt builtins/test/setup.sql builtins/ivy.xml builtins/src builtins/src/org builtins/src/org/apache builtins/src/org/apache/hive builtins/src/org/apache/hive/builtins builtins/src/org/apache/hive/builtins/UDAFUnionMap.java builtins/src/org/apache/hive/builtins/BuiltinUtils.java builtins/build-plugin.xml builtins/build.xml build.xml bin/hive ql/src/test/results/clientpositive/show_functions.q.out ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java pdk/scripts/build-plugin.xml
          Hide
          Phabricator added a comment -

          jsichi requested code review of "HIVE-2523 [jira] add a new builtins subproject".
          Reviewers: JIRA

          I still need to run this through ant test.

          Now that we have a PDK, we can make it easier to add builtin functions to Hive by putting them in a plugin which automatically gets loaded by Hive. This issue will add the necessary framework and one example function; then new functions can be added here, and over time we could migrate old ones here if desired.

          TEST PLAN
          EMPTY

          REVISION DETAIL
          https://reviews.facebook.net/D267

          AFFECTED FILES
          eclipse-templates/.classpath
          builtins
          builtins/test
          builtins/test/iris.txt
          builtins/test/cleanup.sql
          builtins/test/onerow.txt
          builtins/test/setup.sql
          builtins/ivy.xml
          builtins/src
          builtins/src/org
          builtins/src/org/apache
          builtins/src/org/apache/hive
          builtins/src/org/apache/hive/builtins
          builtins/src/org/apache/hive/builtins/UDAFUnionMap.java
          builtins/src/org/apache/hive/builtins/BuiltinUtils.java
          builtins/build-plugin.xml
          builtins/build.xml
          build.xml
          bin/hive
          ql/src/test/results/clientpositive/show_functions.q.out
          ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
          pdk/scripts/build-plugin.xml

          MANAGE HERALD DIFFERENTIAL RULES
          https://reviews.facebook.net/herald/view/differential/

          WHY DID I GET THIS EMAIL?
          https://reviews.facebook.net/herald/transcript/501/

          Tip: use the X-Herald-Rules header to filter Herald messages in your client.

          Show
          Phabricator added a comment - jsichi requested code review of " HIVE-2523 [jira] add a new builtins subproject". Reviewers: JIRA I still need to run this through ant test. Now that we have a PDK, we can make it easier to add builtin functions to Hive by putting them in a plugin which automatically gets loaded by Hive. This issue will add the necessary framework and one example function; then new functions can be added here, and over time we could migrate old ones here if desired. TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D267 AFFECTED FILES eclipse-templates/.classpath builtins builtins/test builtins/test/iris.txt builtins/test/cleanup.sql builtins/test/onerow.txt builtins/test/setup.sql builtins/ivy.xml builtins/src builtins/src/org builtins/src/org/apache builtins/src/org/apache/hive builtins/src/org/apache/hive/builtins builtins/src/org/apache/hive/builtins/UDAFUnionMap.java builtins/src/org/apache/hive/builtins/BuiltinUtils.java builtins/build-plugin.xml builtins/build.xml build.xml bin/hive ql/src/test/results/clientpositive/show_functions.q.out ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java pdk/scripts/build-plugin.xml MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/501/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.
          Hide
          John Sichi added a comment -

          In particular, some of the new functions from HIVE-1545 want to live here.

          Show
          John Sichi added a comment - In particular, some of the new functions from HIVE-1545 want to live here.

            People

            • Assignee:
              John Sichi
              Reporter:
              John Sichi
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development