Hive
  1. Hive
  2. HIVE-4025

Add reflect UDF for member method invocation of column

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Trivial Trivial
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.11.0
    • Component/s: UDF
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      There are many useful non-static methods on type of primitive types. But current reflect UDF cannot invoke those. For example,

      select reflect2(value, "replace", "val", "VALUE") from src;

      which replaces 'val' part of value column with 'VALUE'

      1. HIVE-4025.D8601.2.patch
        41 kB
        Phabricator
      2. HIVE-4025.D8601.1.patch
        41 kB
        Phabricator

        Activity

        Hide
        Phabricator added a comment -

        navis requested code review of "HIVE-4025 [jira] Add reflect UDF for member method invocation of column".

        Reviewers: JIRA

        HIVE-4025 Add reflect UDF for member method invocation of column

        There are many useful non-static methods on type of primitive types. But current reflect UDF cannot invoke those. For example,

        select reflect2(value, "replace", "val", "VALUE") from src;

        which replaces 'val' part of value column with 'VALUE'

        TEST PLAN
        EMPTY

        REVISION DETAIL
        https://reviews.facebook.net/D8601

        AFFECTED FILES
        ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
        ql/src/java/org/apache/hadoop/hive/ql/udf/generic/AbstractGenericUDFReflect.java
        ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFReflect.java
        ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFReflect2.java
        ql/src/test/queries/clientpositive/udf_reflect2.q
        ql/src/test/results/clientpositive/udf_reflect2.q.out

        MANAGE HERALD RULES
        https://reviews.facebook.net/herald/view/differential/

        WHY DID I GET THIS EMAIL?
        https://reviews.facebook.net/herald/transcript/20937/

        To: JIRA, navis

        Show
        Phabricator added a comment - navis requested code review of " HIVE-4025 [jira] Add reflect UDF for member method invocation of column". Reviewers: JIRA HIVE-4025 Add reflect UDF for member method invocation of column There are many useful non-static methods on type of primitive types. But current reflect UDF cannot invoke those. For example, select reflect2(value, "replace", "val", "VALUE") from src; which replaces 'val' part of value column with 'VALUE' TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D8601 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/AbstractGenericUDFReflect.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFReflect.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFReflect2.java ql/src/test/queries/clientpositive/udf_reflect2.q ql/src/test/results/clientpositive/udf_reflect2.q.out MANAGE HERALD RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/20937/ To: JIRA, navis
        Hide
        Navis added a comment -

        Currently supports methods only with primitive types for argument types and return type

        Show
        Navis added a comment - Currently supports methods only with primitive types for argument types and return type
        Hide
        Namit Jain added a comment -

        comments

        Show
        Namit Jain added a comment - comments
        Hide
        Phabricator added a comment -

        njain has commented on the revision "HIVE-4025 [jira] Add reflect UDF for member method invocation of column".

        INLINE COMMENTS
        ql/src/test/results/clientpositive/udf_reflect2.q.out:312 Have you run all tests ?

        I think, show_functions need to be updated.

        The last columns seem to be wrong:

        look at the results for

        getYear
        getMonth
        getDay etc.

        REVISION DETAIL
        https://reviews.facebook.net/D8601

        To: JIRA, navis
        Cc: njain

        Show
        Phabricator added a comment - njain has commented on the revision " HIVE-4025 [jira] Add reflect UDF for member method invocation of column". INLINE COMMENTS ql/src/test/results/clientpositive/udf_reflect2.q.out:312 Have you run all tests ? I think, show_functions need to be updated. The last columns seem to be wrong: look at the results for getYear getMonth getDay etc. REVISION DETAIL https://reviews.facebook.net/D8601 To: JIRA, navis Cc: njain
        Hide
        Phabricator added a comment -

        brock has commented on the revision "HIVE-4025 [jira] Add reflect UDF for member method invocation of column".

        Hi,

        Is there are a better name than reflect2? Perhaps reflect_obj, reflect_inst, or something else?

        Brock

        REVISION DETAIL
        https://reviews.facebook.net/D8601

        To: JIRA, navis
        Cc: njain, brock

        Show
        Phabricator added a comment - brock has commented on the revision " HIVE-4025 [jira] Add reflect UDF for member method invocation of column". Hi, Is there are a better name than reflect2? Perhaps reflect_obj, reflect_inst, or something else? Brock REVISION DETAIL https://reviews.facebook.net/D8601 To: JIRA, navis Cc: njain, brock
        Hide
        Phabricator added a comment -

        navis has commented on the revision "HIVE-4025 [jira] Add reflect UDF for member method invocation of column".

        INLINE COMMENTS
        ql/src/test/results/clientpositive/udf_reflect2.q.out:312 I'll update that.

        The last columns seem to be wrong:

        It's right result for TimeStamp class.

        getYear()

        • Returns a value that is the result of subtracting 1900 from the
        • year that contains or begins with the instant in time represented
        • by this <code>Date</code> object, as interpreted in the local
        • time zone.

        getMonth()

        • Returns a number representing the month that contains or begins
        • with the instant in time represented by this <tt>Date</tt> object.
        • The value returned is between <code>0</code> and <code>11</code>,
        • with the value <code>0</code> representing January.

        getDay()

        • Returns the day of the week represented by this date. The
        • returned value (<tt>0</tt> = Sunday, <tt>1</tt> = Monday,
        • <tt>2</tt> = Tuesday, <tt>3</tt> = Wednesday, <tt>4</tt> =
        • Thursday, <tt>5</tt> = Friday, <tt>6</tt> = Saturday)
        • represents the day of the week that contains or begins with
        • the instant in time represented by this <tt>Date</tt> object,
        • as interpreted in the local time zone.

        REVISION DETAIL
        https://reviews.facebook.net/D8601

        To: JIRA, navis
        Cc: njain, brock

        Show
        Phabricator added a comment - navis has commented on the revision " HIVE-4025 [jira] Add reflect UDF for member method invocation of column". INLINE COMMENTS ql/src/test/results/clientpositive/udf_reflect2.q.out:312 I'll update that. The last columns seem to be wrong: It's right result for TimeStamp class. getYear() Returns a value that is the result of subtracting 1900 from the year that contains or begins with the instant in time represented by this <code>Date</code> object, as interpreted in the local time zone. getMonth() Returns a number representing the month that contains or begins with the instant in time represented by this <tt>Date</tt> object. The value returned is between <code>0</code> and <code>11</code>, with the value <code>0</code> representing January. getDay() Returns the day of the week represented by this date. The returned value (<tt>0</tt> = Sunday, <tt>1</tt> = Monday, <tt>2</tt> = Tuesday, <tt>3</tt> = Wednesday, <tt>4</tt> = Thursday, <tt>5</tt> = Friday, <tt>6</tt> = Saturday) represents the day of the week that contains or begins with the instant in time represented by this <tt>Date</tt> object, as interpreted in the local time zone. REVISION DETAIL https://reviews.facebook.net/D8601 To: JIRA, navis Cc: njain, brock
        Hide
        Phabricator added a comment -

        navis has commented on the revision "HIVE-4025 [jira] Add reflect UDF for member method invocation of column".

        I've thought the name like "reflect_instance" or something, but found current "reflect" can invoke method of instance also(by calling default constructor of target class).

        Naming is hard. How about "type_reflect"?

        REVISION DETAIL
        https://reviews.facebook.net/D8601

        To: JIRA, navis
        Cc: njain, brock

        Show
        Phabricator added a comment - navis has commented on the revision " HIVE-4025 [jira] Add reflect UDF for member method invocation of column". I've thought the name like "reflect_instance" or something, but found current "reflect" can invoke method of instance also(by calling default constructor of target class). Naming is hard. How about "type_reflect"? REVISION DETAIL https://reviews.facebook.net/D8601 To: JIRA, navis Cc: njain, brock
        Hide
        Phabricator added a comment -

        brock has commented on the revision "HIVE-4025 [jira] Add reflect UDF for member method invocation of column".

        Agreed, naming is hard. I like the fact that both methods start with reflect. Perhaps reflect2 is as good as we can do. It's not likely we'll end up with reflect5_1().

        REVISION DETAIL
        https://reviews.facebook.net/D8601

        To: JIRA, navis
        Cc: njain, brock

        Show
        Phabricator added a comment - brock has commented on the revision " HIVE-4025 [jira] Add reflect UDF for member method invocation of column". Agreed, naming is hard. I like the fact that both methods start with reflect. Perhaps reflect2 is as good as we can do. It's not likely we'll end up with reflect5_1(). REVISION DETAIL https://reviews.facebook.net/D8601 To: JIRA, navis Cc: njain, brock
        Hide
        Phabricator added a comment -

        navis updated the revision "HIVE-4025 [jira] Add reflect UDF for member method invocation of column".

        Keeping the name 'reflect2' for not having better one
        Addressed comment (added result of show functions)

        Reviewers: JIRA

        REVISION DETAIL
        https://reviews.facebook.net/D8601

        CHANGE SINCE LAST DIFF
        https://reviews.facebook.net/D8601?vs=27819&id=28365#toc

        AFFECTED FILES
        ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
        ql/src/java/org/apache/hadoop/hive/ql/udf/generic/AbstractGenericUDFReflect.java
        ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFReflect.java
        ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFReflect2.java
        ql/src/test/queries/clientpositive/udf_reflect2.q
        ql/src/test/results/clientpositive/show_functions.q.out
        ql/src/test/results/clientpositive/udf_reflect2.q.out

        To: JIRA, navis
        Cc: njain, brock

        Show
        Phabricator added a comment - navis updated the revision " HIVE-4025 [jira] Add reflect UDF for member method invocation of column". Keeping the name 'reflect2' for not having better one Addressed comment (added result of show functions) Reviewers: JIRA REVISION DETAIL https://reviews.facebook.net/D8601 CHANGE SINCE LAST DIFF https://reviews.facebook.net/D8601?vs=27819&id=28365#toc AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/AbstractGenericUDFReflect.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFReflect.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFReflect2.java ql/src/test/queries/clientpositive/udf_reflect2.q ql/src/test/results/clientpositive/show_functions.q.out ql/src/test/results/clientpositive/udf_reflect2.q.out To: JIRA, navis Cc: njain, brock
        Hide
        Namit Jain added a comment -

        +1

        Show
        Namit Jain added a comment - +1
        Hide
        Namit Jain added a comment -

        Committed. Thanks Navis

        Show
        Namit Jain added a comment - Committed. Thanks Navis
        Hide
        Hudson added a comment -

        Integrated in hive-trunk-hadoop1 #97 (See https://builds.apache.org/job/hive-trunk-hadoop1/97/)
        HIVE-4025 Add reflect UDF for member method invocation of column
        (Navis via namit) (Revision 1448900)

        Result = ABORTED
        namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1448900
        Files :

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/AbstractGenericUDFReflect.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFReflect.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFReflect2.java
        • /hive/trunk/ql/src/test/queries/clientpositive/udf_reflect2.q
        • /hive/trunk/ql/src/test/results/clientpositive/show_functions.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/udf_reflect2.q.out
        Show
        Hudson added a comment - Integrated in hive-trunk-hadoop1 #97 (See https://builds.apache.org/job/hive-trunk-hadoop1/97/ ) HIVE-4025 Add reflect UDF for member method invocation of column (Navis via namit) (Revision 1448900) Result = ABORTED namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1448900 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/AbstractGenericUDFReflect.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFReflect.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFReflect2.java /hive/trunk/ql/src/test/queries/clientpositive/udf_reflect2.q /hive/trunk/ql/src/test/results/clientpositive/show_functions.q.out /hive/trunk/ql/src/test/results/clientpositive/udf_reflect2.q.out
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-h0.21 #1982 (See https://builds.apache.org/job/Hive-trunk-h0.21/1982/)
        HIVE-4025 Add reflect UDF for member method invocation of column
        (Navis via namit) (Revision 1448900)

        Result = SUCCESS
        namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1448900
        Files :

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/AbstractGenericUDFReflect.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFReflect.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFReflect2.java
        • /hive/trunk/ql/src/test/queries/clientpositive/udf_reflect2.q
        • /hive/trunk/ql/src/test/results/clientpositive/show_functions.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/udf_reflect2.q.out
        Show
        Hudson added a comment - Integrated in Hive-trunk-h0.21 #1982 (See https://builds.apache.org/job/Hive-trunk-h0.21/1982/ ) HIVE-4025 Add reflect UDF for member method invocation of column (Navis via namit) (Revision 1448900) Result = SUCCESS namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1448900 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/AbstractGenericUDFReflect.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFReflect.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFReflect2.java /hive/trunk/ql/src/test/queries/clientpositive/udf_reflect2.q /hive/trunk/ql/src/test/results/clientpositive/show_functions.q.out /hive/trunk/ql/src/test/results/clientpositive/udf_reflect2.q.out
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-hadoop2 #134 (See https://builds.apache.org/job/Hive-trunk-hadoop2/134/)
        HIVE-4025 Add reflect UDF for member method invocation of column
        (Navis via namit) (Revision 1448900)

        Result = FAILURE
        namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1448900
        Files :

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/AbstractGenericUDFReflect.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFReflect.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFReflect2.java
        • /hive/trunk/ql/src/test/queries/clientpositive/udf_reflect2.q
        • /hive/trunk/ql/src/test/results/clientpositive/show_functions.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/udf_reflect2.q.out
        Show
        Hudson added a comment - Integrated in Hive-trunk-hadoop2 #134 (See https://builds.apache.org/job/Hive-trunk-hadoop2/134/ ) HIVE-4025 Add reflect UDF for member method invocation of column (Navis via namit) (Revision 1448900) Result = FAILURE namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1448900 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/AbstractGenericUDFReflect.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFReflect.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFReflect2.java /hive/trunk/ql/src/test/queries/clientpositive/udf_reflect2.q /hive/trunk/ql/src/test/results/clientpositive/show_functions.q.out /hive/trunk/ql/src/test/results/clientpositive/udf_reflect2.q.out

          People

          • Assignee:
            Navis
            Reporter:
            Navis
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development