Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.0
    • Component/s: UDF
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Now that we have a PDK, we can make it easier to add builtin functions to Hive by putting them in a plugin which automatically gets loaded by Hive. This issue will add the necessary framework and one example function; then new functions can be added here, and over time we could migrate old ones here if desired.

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          23d 23h 7m 1 John Sichi 17/Nov/11 22:24
          Patch Available Patch Available Resolved Resolved
          14d 9h 50m 1 Carl Steinbach 02/Dec/11 08:14
          Resolved Resolved Closed Closed
          14d 15h 41m 1 Carl Steinbach 16/Dec/11 23:55
          Carl Steinbach made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Carl Steinbach made changes -
          Fix Version/s 0.9.0 [ 12317742 ]
          Carl Steinbach made changes -
          Fix Version/s 0.8.0 [ 12316178 ]
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-h0.21 #1125 (See https://builds.apache.org/job/Hive-trunk-h0.21/1125/)
          Fix eclipse classpath template broken in HIVE-2523.
          (Carl Steinbach via jvs)

          jvs : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1210599
          Files :

          • /hive/trunk/eclipse-templates/.classpath
          Show
          Hudson added a comment - Integrated in Hive-trunk-h0.21 #1125 (See https://builds.apache.org/job/Hive-trunk-h0.21/1125/ ) Fix eclipse classpath template broken in HIVE-2523 . (Carl Steinbach via jvs) jvs : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1210599 Files : /hive/trunk/eclipse-templates/.classpath
          Hide
          Phabricator added a comment -

          jsichi has abandoned the revision "HIVE-2523 [jira] add a new builtins subproject".

          REVISION DETAIL
          https://reviews.facebook.net/D267

          Show
          Phabricator added a comment - jsichi has abandoned the revision " HIVE-2523 [jira] add a new builtins subproject". REVISION DETAIL https://reviews.facebook.net/D267
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-h0.21 #1119 (See https://builds.apache.org/job/Hive-trunk-h0.21/1119/)
          HIVE-2523. Add a new builtins subproject (John Sichi via cws)

          cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1209384
          Files :

          • /hive/trunk/bin/hive
          • /hive/trunk/build.xml
          • /hive/trunk/builtins
          • /hive/trunk/builtins/build-plugin.xml
          • /hive/trunk/builtins/build.xml
          • /hive/trunk/builtins/ivy.xml
          • /hive/trunk/builtins/src
          • /hive/trunk/builtins/src/org
          • /hive/trunk/builtins/src/org/apache
          • /hive/trunk/builtins/src/org/apache/hive
          • /hive/trunk/builtins/src/org/apache/hive/builtins
          • /hive/trunk/builtins/src/org/apache/hive/builtins/BuiltinUtils.java
          • /hive/trunk/builtins/src/org/apache/hive/builtins/UDAFUnionMap.java
          • /hive/trunk/builtins/test
          • /hive/trunk/builtins/test/cleanup.sql
          • /hive/trunk/builtins/test/iris.txt
          • /hive/trunk/builtins/test/onerow.txt
          • /hive/trunk/builtins/test/setup.sql
          • /hive/trunk/eclipse-templates/.classpath
          • /hive/trunk/pdk/scripts/build-plugin.xml
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java
          • /hive/trunk/ql/src/test/results/clientpositive/show_functions.q.out
          Show
          Hudson added a comment - Integrated in Hive-trunk-h0.21 #1119 (See https://builds.apache.org/job/Hive-trunk-h0.21/1119/ ) HIVE-2523 . Add a new builtins subproject (John Sichi via cws) cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1209384 Files : /hive/trunk/bin/hive /hive/trunk/build.xml /hive/trunk/builtins /hive/trunk/builtins/build-plugin.xml /hive/trunk/builtins/build.xml /hive/trunk/builtins/ivy.xml /hive/trunk/builtins/src /hive/trunk/builtins/src/org /hive/trunk/builtins/src/org/apache /hive/trunk/builtins/src/org/apache/hive /hive/trunk/builtins/src/org/apache/hive/builtins /hive/trunk/builtins/src/org/apache/hive/builtins/BuiltinUtils.java /hive/trunk/builtins/src/org/apache/hive/builtins/UDAFUnionMap.java /hive/trunk/builtins/test /hive/trunk/builtins/test/cleanup.sql /hive/trunk/builtins/test/iris.txt /hive/trunk/builtins/test/onerow.txt /hive/trunk/builtins/test/setup.sql /hive/trunk/eclipse-templates/.classpath /hive/trunk/pdk/scripts/build-plugin.xml /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java /hive/trunk/ql/src/test/results/clientpositive/show_functions.q.out
          Carl Steinbach made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Resolution Fixed [ 1 ]
          Hide
          Carl Steinbach added a comment -

          Committed to trunk. Thanks John!

          Show
          Carl Steinbach added a comment - Committed to trunk. Thanks John!
          Hide
          Phabricator added a comment -

          cwsteinbach has accepted the revision "HIVE-2523 [jira] add a new builtins subproject".

          +1. Will commit if tests pass.

          REVISION DETAIL
          https://reviews.facebook.net/D267

          Show
          Phabricator added a comment - cwsteinbach has accepted the revision " HIVE-2523 [jira] add a new builtins subproject". +1. Will commit if tests pass. REVISION DETAIL https://reviews.facebook.net/D267
          Phabricator made changes -
          Attachment HIVE-2523.D267.4.patch [ 12505433 ]
          Hide
          Phabricator added a comment -

          jsichi updated the revision "HIVE-2523 [jira] add a new builtins subproject".
          Reviewers: JIRA

          Rebase and fix a problem which was preventing this from passing parallel tests.

          REVISION DETAIL
          https://reviews.facebook.net/D267

          AFFECTED FILES
          eclipse-templates/.classpath
          builtins
          builtins/test
          builtins/test/iris.txt
          builtins/test/cleanup.sql
          builtins/test/onerow.txt
          builtins/test/setup.sql
          builtins/ivy.xml
          builtins/src
          builtins/src/org
          builtins/src/org/apache
          builtins/src/org/apache/hive
          builtins/src/org/apache/hive/builtins
          builtins/src/org/apache/hive/builtins/UDAFUnionMap.java
          builtins/src/org/apache/hive/builtins/BuiltinUtils.java
          builtins/build-plugin.xml
          builtins/build.xml
          build.xml
          bin/hive
          ql/src/test/results/clientpositive/show_functions.q.out
          ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
          pdk/scripts/build-plugin.xml

          Show
          Phabricator added a comment - jsichi updated the revision " HIVE-2523 [jira] add a new builtins subproject". Reviewers: JIRA Rebase and fix a problem which was preventing this from passing parallel tests. REVISION DETAIL https://reviews.facebook.net/D267 AFFECTED FILES eclipse-templates/.classpath builtins builtins/test builtins/test/iris.txt builtins/test/cleanup.sql builtins/test/onerow.txt builtins/test/setup.sql builtins/ivy.xml builtins/src builtins/src/org builtins/src/org/apache builtins/src/org/apache/hive builtins/src/org/apache/hive/builtins builtins/src/org/apache/hive/builtins/UDAFUnionMap.java builtins/src/org/apache/hive/builtins/BuiltinUtils.java builtins/build-plugin.xml builtins/build.xml build.xml bin/hive ql/src/test/results/clientpositive/show_functions.q.out ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java pdk/scripts/build-plugin.xml
          John Sichi made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          John Sichi added a comment -

          Tests passed for me, yay!

          Show
          John Sichi added a comment - Tests passed for me, yay!
          Phabricator made changes -
          Attachment HIVE-2523.D267.3.patch [ 12504005 ]
          Hide
          Phabricator added a comment -

          jsichi updated the revision "HIVE-2523 [jira] add a new builtins subproject".
          Reviewers: JIRA

          Attempt at fixing leak...gonna run through tests again.

          REVISION DETAIL
          https://reviews.facebook.net/D267

          AFFECTED FILES
          eclipse-templates/.classpath
          builtins
          builtins/test
          builtins/test/iris.txt
          builtins/test/cleanup.sql
          builtins/test/onerow.txt
          builtins/test/setup.sql
          builtins/ivy.xml
          builtins/src
          builtins/src/org
          builtins/src/org/apache
          builtins/src/org/apache/hive
          builtins/src/org/apache/hive/builtins
          builtins/src/org/apache/hive/builtins/UDAFUnionMap.java
          builtins/src/org/apache/hive/builtins/BuiltinUtils.java
          builtins/build-plugin.xml
          builtins/build.xml
          build.xml
          bin/hive
          ql/src/test/results/clientpositive/show_functions.q.out
          ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
          pdk/scripts/build-plugin.xml

          Show
          Phabricator added a comment - jsichi updated the revision " HIVE-2523 [jira] add a new builtins subproject". Reviewers: JIRA Attempt at fixing leak...gonna run through tests again. REVISION DETAIL https://reviews.facebook.net/D267 AFFECTED FILES eclipse-templates/.classpath builtins builtins/test builtins/test/iris.txt builtins/test/cleanup.sql builtins/test/onerow.txt builtins/test/setup.sql builtins/ivy.xml builtins/src builtins/src/org builtins/src/org/apache builtins/src/org/apache/hive builtins/src/org/apache/hive/builtins builtins/src/org/apache/hive/builtins/UDAFUnionMap.java builtins/src/org/apache/hive/builtins/BuiltinUtils.java builtins/build-plugin.xml builtins/build.xml build.xml bin/hive ql/src/test/results/clientpositive/show_functions.q.out ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java pdk/scripts/build-plugin.xml
          Hide
          John Sichi added a comment -

          Tests are failing due to a timeout after 5 hours, so I think I must have a leak of some kind...debugging.

          Show
          John Sichi added a comment - Tests are failing due to a timeout after 5 hours, so I think I must have a leak of some kind...debugging.
          Phabricator made changes -
          Attachment D267.2.patch [ 12502978 ]
          Hide
          Phabricator added a comment -

          jsichi updated the revision "HIVE-2523 [jira] add a new builtins subproject".
          Reviewers: JIRA

          Fix some tests, still need to run through the whole suite again.

          REVISION DETAIL
          https://reviews.facebook.net/D267

          AFFECTED FILES
          eclipse-templates/.classpath
          builtins
          builtins/test
          builtins/test/iris.txt
          builtins/test/cleanup.sql
          builtins/test/onerow.txt
          builtins/test/setup.sql
          builtins/ivy.xml
          builtins/src
          builtins/src/org
          builtins/src/org/apache
          builtins/src/org/apache/hive
          builtins/src/org/apache/hive/builtins
          builtins/src/org/apache/hive/builtins/UDAFUnionMap.java
          builtins/src/org/apache/hive/builtins/BuiltinUtils.java
          builtins/build-plugin.xml
          builtins/build.xml
          build.xml
          bin/hive
          ql/src/test/results/clientpositive/show_functions.q.out
          ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
          pdk/scripts/build-plugin.xml

          Show
          Phabricator added a comment - jsichi updated the revision " HIVE-2523 [jira] add a new builtins subproject". Reviewers: JIRA Fix some tests, still need to run through the whole suite again. REVISION DETAIL https://reviews.facebook.net/D267 AFFECTED FILES eclipse-templates/.classpath builtins builtins/test builtins/test/iris.txt builtins/test/cleanup.sql builtins/test/onerow.txt builtins/test/setup.sql builtins/ivy.xml builtins/src builtins/src/org builtins/src/org/apache builtins/src/org/apache/hive builtins/src/org/apache/hive/builtins builtins/src/org/apache/hive/builtins/UDAFUnionMap.java builtins/src/org/apache/hive/builtins/BuiltinUtils.java builtins/build-plugin.xml builtins/build.xml build.xml bin/hive ql/src/test/results/clientpositive/show_functions.q.out ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java pdk/scripts/build-plugin.xml
          Phabricator made changes -
          Attachment D267.1.patch [ 12502551 ]
          Hide
          Phabricator added a comment -

          jsichi requested code review of "HIVE-2523 [jira] add a new builtins subproject".
          Reviewers: JIRA

          I still need to run this through ant test.

          Now that we have a PDK, we can make it easier to add builtin functions to Hive by putting them in a plugin which automatically gets loaded by Hive. This issue will add the necessary framework and one example function; then new functions can be added here, and over time we could migrate old ones here if desired.

          TEST PLAN
          EMPTY

          REVISION DETAIL
          https://reviews.facebook.net/D267

          AFFECTED FILES
          eclipse-templates/.classpath
          builtins
          builtins/test
          builtins/test/iris.txt
          builtins/test/cleanup.sql
          builtins/test/onerow.txt
          builtins/test/setup.sql
          builtins/ivy.xml
          builtins/src
          builtins/src/org
          builtins/src/org/apache
          builtins/src/org/apache/hive
          builtins/src/org/apache/hive/builtins
          builtins/src/org/apache/hive/builtins/UDAFUnionMap.java
          builtins/src/org/apache/hive/builtins/BuiltinUtils.java
          builtins/build-plugin.xml
          builtins/build.xml
          build.xml
          bin/hive
          ql/src/test/results/clientpositive/show_functions.q.out
          ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
          pdk/scripts/build-plugin.xml

          MANAGE HERALD DIFFERENTIAL RULES
          https://reviews.facebook.net/herald/view/differential/

          WHY DID I GET THIS EMAIL?
          https://reviews.facebook.net/herald/transcript/501/

          Tip: use the X-Herald-Rules header to filter Herald messages in your client.

          Show
          Phabricator added a comment - jsichi requested code review of " HIVE-2523 [jira] add a new builtins subproject". Reviewers: JIRA I still need to run this through ant test. Now that we have a PDK, we can make it easier to add builtin functions to Hive by putting them in a plugin which automatically gets loaded by Hive. This issue will add the necessary framework and one example function; then new functions can be added here, and over time we could migrate old ones here if desired. TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D267 AFFECTED FILES eclipse-templates/.classpath builtins builtins/test builtins/test/iris.txt builtins/test/cleanup.sql builtins/test/onerow.txt builtins/test/setup.sql builtins/ivy.xml builtins/src builtins/src/org builtins/src/org/apache builtins/src/org/apache/hive builtins/src/org/apache/hive/builtins builtins/src/org/apache/hive/builtins/UDAFUnionMap.java builtins/src/org/apache/hive/builtins/BuiltinUtils.java builtins/build-plugin.xml builtins/build.xml build.xml bin/hive ql/src/test/results/clientpositive/show_functions.q.out ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java pdk/scripts/build-plugin.xml MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/501/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.
          John Sichi made changes -
          Assignee John Sichi [ jvs ]
          John Sichi made changes -
          Link This issue relates to HIVE-1545 [ HIVE-1545 ]
          Hide
          John Sichi added a comment -

          In particular, some of the new functions from HIVE-1545 want to live here.

          Show
          John Sichi added a comment - In particular, some of the new functions from HIVE-1545 want to live here.
          John Sichi made changes -
          Field Original Value New Value
          Link This issue relates to HIVE-2244 [ HIVE-2244 ]
          John Sichi created issue -

            People

            • Assignee:
              John Sichi
              Reporter:
              John Sichi
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development