Hive
  1. Hive
  2. HIVE-5278

Move some string UDFs to GenericUDFs, for better varchar support

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.12.0
    • Component/s: Types, UDF
    • Labels:
      None

      Description

      To better support varchar/char types in string UDFs, select UDFs should be converted to GenericUDFs. This allows the UDF to return the resulting char/varchar length in the type metadata.

      This work is being split off as a separate task from HIVE-4844. The initial UDFs as part of this work are concat/lower/upper.

      1. D12909.1.patch
        73 kB
        Phabricator
      2. HIVE-5278.1.patch
        75 kB
        Jason Dere
      3. HIVE-5278.2.patch
        28 kB
        Jason Dere
      4. HIVE-5278.v12.1.patch
        28 kB
        Jason Dere

        Issue Links

          Activity

          Hide
          Jason Dere added a comment -

          Attaching patch v1
          Posted review at https://reviews.facebook.net/D12909

          Show
          Jason Dere added a comment - Attaching patch v1 Posted review at https://reviews.facebook.net/D12909
          Hide
          Phabricator added a comment -

          jdere requested code review of "HIVE-5278 [jira] Move some string UDFs to GenericUDFs, for better varchar support".

          Reviewers: JIRA

          HIVE-5278: Move some string UDFs to GenericUDFs, for better varchar support

          To better support varchar/char types in string UDFs, select UDFs should be converted to GenericUDFs. This allows the UDF to return the resulting char/varchar length in the type metadata.

          This work is being split off as a separate task from HIVE-4844. The initial UDFs as part of this work are concat/lower/upper.

          TEST PLAN
          EMPTY

          REVISION DETAIL
          https://reviews.facebook.net/D12909

          AFFECTED FILES
          ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
          ql/src/java/org/apache/hadoop/hive/ql/udf/UDFConcat.java
          ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLower.java
          ql/src/java/org/apache/hadoop/hive/ql/udf/UDFUpper.java
          ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFConcat.java
          ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFLower.java
          ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFUpper.java
          ql/src/test/results/compiler/plan/groupby2.q.xml
          ql/src/test/results/compiler/plan/udf6.q.xml

          MANAGE HERALD RULES
          https://reviews.facebook.net/herald/view/differential/

          WHY DID I GET THIS EMAIL?
          https://reviews.facebook.net/herald/transcript/30879/

          To: JIRA, jdere

          Show
          Phabricator added a comment - jdere requested code review of " HIVE-5278 [jira] Move some string UDFs to GenericUDFs, for better varchar support". Reviewers: JIRA HIVE-5278 : Move some string UDFs to GenericUDFs, for better varchar support To better support varchar/char types in string UDFs, select UDFs should be converted to GenericUDFs. This allows the UDF to return the resulting char/varchar length in the type metadata. This work is being split off as a separate task from HIVE-4844 . The initial UDFs as part of this work are concat/lower/upper. TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D12909 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java ql/src/java/org/apache/hadoop/hive/ql/udf/UDFConcat.java ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLower.java ql/src/java/org/apache/hadoop/hive/ql/udf/UDFUpper.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFConcat.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFLower.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFUpper.java ql/src/test/results/compiler/plan/groupby2.q.xml ql/src/test/results/compiler/plan/udf6.q.xml MANAGE HERALD RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/30879/ To: JIRA, jdere
          Hide
          Jason Dere added a comment -

          Looks like Eclipse may have made a whole bunch of formatting changes to FunctionRegistry. Attaching patch v2 which sticks to just the necessary changes to FunctionRegistry.

          Show
          Jason Dere added a comment - Looks like Eclipse may have made a whole bunch of formatting changes to FunctionRegistry. Attaching patch v2 which sticks to just the necessary changes to FunctionRegistry.
          Hide
          Ashutosh Chauhan added a comment -

          +1

          Show
          Ashutosh Chauhan added a comment - +1
          Hide
          Ashutosh Chauhan added a comment -

          Committed to trunk. Thanks, Jason!

          Show
          Ashutosh Chauhan added a comment - Committed to trunk. Thanks, Jason!
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hive-trunk-hadoop2-ptest #99 (See https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/99/)
          HIVE-5278 : Move some string UDFs to GenericUDFs, for better varchar support (Jason Dere via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1523518)

          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFConcat.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLower.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFUpper.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFConcat.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFLower.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFUpper.java
          • /hive/trunk/ql/src/test/results/compiler/plan/groupby2.q.xml
          • /hive/trunk/ql/src/test/results/compiler/plan/udf6.q.xml
          Show
          Hudson added a comment - FAILURE: Integrated in Hive-trunk-hadoop2-ptest #99 (See https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/99/ ) HIVE-5278 : Move some string UDFs to GenericUDFs, for better varchar support (Jason Dere via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1523518 ) /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFConcat.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLower.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFUpper.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFConcat.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFLower.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFUpper.java /hive/trunk/ql/src/test/results/compiler/plan/groupby2.q.xml /hive/trunk/ql/src/test/results/compiler/plan/udf6.q.xml
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hive-trunk-hadoop1-ptest #166 (See https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/166/)
          HIVE-5278 : Move some string UDFs to GenericUDFs, for better varchar support (Jason Dere via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1523518)

          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFConcat.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLower.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFUpper.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFConcat.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFLower.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFUpper.java
          • /hive/trunk/ql/src/test/results/compiler/plan/groupby2.q.xml
          • /hive/trunk/ql/src/test/results/compiler/plan/udf6.q.xml
          Show
          Hudson added a comment - FAILURE: Integrated in Hive-trunk-hadoop1-ptest #166 (See https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/166/ ) HIVE-5278 : Move some string UDFs to GenericUDFs, for better varchar support (Jason Dere via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1523518 ) /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFConcat.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLower.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFUpper.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFConcat.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFLower.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFUpper.java /hive/trunk/ql/src/test/results/compiler/plan/groupby2.q.xml /hive/trunk/ql/src/test/results/compiler/plan/udf6.q.xml
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hive-trunk-hadoop2 #432 (See https://builds.apache.org/job/Hive-trunk-hadoop2/432/)
          HIVE-5278 : Move some string UDFs to GenericUDFs, for better varchar support (Jason Dere via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1523518)

          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFConcat.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLower.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFUpper.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFConcat.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFLower.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFUpper.java
          • /hive/trunk/ql/src/test/results/compiler/plan/groupby2.q.xml
          • /hive/trunk/ql/src/test/results/compiler/plan/udf6.q.xml
          Show
          Hudson added a comment - FAILURE: Integrated in Hive-trunk-hadoop2 #432 (See https://builds.apache.org/job/Hive-trunk-hadoop2/432/ ) HIVE-5278 : Move some string UDFs to GenericUDFs, for better varchar support (Jason Dere via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1523518 ) /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFConcat.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLower.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFUpper.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFConcat.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFLower.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFUpper.java /hive/trunk/ql/src/test/results/compiler/plan/groupby2.q.xml /hive/trunk/ql/src/test/results/compiler/plan/udf6.q.xml
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hive-trunk-h0.21 #2335 (See https://builds.apache.org/job/Hive-trunk-h0.21/2335/)
          HIVE-5278 : Move some string UDFs to GenericUDFs, for better varchar support (Jason Dere via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1523518)

          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFConcat.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLower.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFUpper.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFConcat.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFLower.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFUpper.java
          • /hive/trunk/ql/src/test/results/compiler/plan/groupby2.q.xml
          • /hive/trunk/ql/src/test/results/compiler/plan/udf6.q.xml
          Show
          Hudson added a comment - FAILURE: Integrated in Hive-trunk-h0.21 #2335 (See https://builds.apache.org/job/Hive-trunk-h0.21/2335/ ) HIVE-5278 : Move some string UDFs to GenericUDFs, for better varchar support (Jason Dere via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1523518 ) /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFConcat.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLower.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFUpper.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFConcat.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFLower.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFUpper.java /hive/trunk/ql/src/test/results/compiler/plan/groupby2.q.xml /hive/trunk/ql/src/test/results/compiler/plan/udf6.q.xml
          Hide
          Jason Dere added a comment -

          attaching HIVE-5278.v12.1.patch, for use in 0.12 branch

          Show
          Jason Dere added a comment - attaching HIVE-5278 .v12.1.patch, for use in 0.12 branch
          Hide
          Thejas M Nair added a comment -

          Patch committed to 0.12 branch.

          Show
          Thejas M Nair added a comment - Patch committed to 0.12 branch.
          Hide
          Ashutosh Chauhan added a comment -

          This issue has been fixed and released as part of 0.12 release. If you find further issues, please create a new jira and link it to this one.

          Show
          Ashutosh Chauhan added a comment - This issue has been fixed and released as part of 0.12 release. If you find further issues, please create a new jira and link it to this one.

            People

            • Assignee:
              Jason Dere
              Reporter:
              Jason Dere
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development