Hive
  1. Hive
  2. HIVE-1360

Allow UDFs to access constant parameter values at compile time

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.5.0
    • Fix Version/s: 0.8.0
    • Component/s: Query Processor, UDF
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      UDFs should be able to access constant parameter values at compile time.

      1. HIVE-1360.patch
        37 kB
        Jonathan Chang
      2. HIVE-1360.patch
        37 kB
        Jonathan Chang
      3. HIVE-1360.part2.patch
        28 kB
        Jonathan Chang
      4. HIVE-1360.3.patch
        36 kB
        Jonathan Chang
      5. HIVE-1360.2.test-failures.txt
        6 kB
        Carl Steinbach
      6. HIVE-1360.2.patch.txt
        36 kB
        Carl Steinbach

        Issue Links

          Activity

          Hide
          Esteban Gutierrez added a comment -

          Working on a patch on this since its a better alternative to solve my problem in HIVE-2124

          Show
          Esteban Gutierrez added a comment - Working on a patch on this since its a better alternative to solve my problem in HIVE-2124
          Hide
          Adam Kramer added a comment -

          Glad this is getting some traction! Ideally, the "constancy" here should be detectable in the init/initialize state, and, if it IS, the constant should be accessible on initialization.

          Show
          Adam Kramer added a comment - Glad this is getting some traction! Ideally, the "constancy" here should be detectable in the init/initialize state, and, if it IS, the constant should be accessible on initialization.
          Hide
          John Sichi added a comment -

          Some of the generic base class stuff from HIVE-1016 may be relevant here.

          Show
          John Sichi added a comment - Some of the generic base class stuff from HIVE-1016 may be relevant here.
          Hide
          Jonathan Chang added a comment -

          HIVE-1360. It has been a long-standing request for UDFs to be able to
          Edit
          access parameter values. This not only enables significant performance
          improvement possibilities, it also allows for fundamentally richer
          behavior, such as allowing the output type of a UDF to depend on its
          inputs.

          The strategy in this diff is to introduce the notion of a
          ConstantObjectInspector, like a regular ObjectInspector except that it
          encapsulates a constant value and knows what this constant value is.
          These COIs are created through a factory method by ExprNodeConstantDesc
          during plan generation hence UDFs will be able to capture these constant
          values during the initialize phase. Furthermore, because these
          ConstantObjectInspectors are simply subinterfaces of ObjectInspector,
          UDFs which are not "constant-aware" receive ObjectInspectors which
          also implement the same interfaces they are used to, so no special
          handling needs to be done for existing UDFs.

          An example UDF which uses this new functionality is also included in
          this diff. NAMED_STRUCT is like STRUCT except that it also allows users
          to specify the names of the fields of the struct, something previously
          not possible because the names of the fields must be known at compile
          time.

          Also see this pull request: https://github.com/apache/hive/pull/2

          Show
          Jonathan Chang added a comment - HIVE-1360 . It has been a long-standing request for UDFs to be able to Edit access parameter values. This not only enables significant performance improvement possibilities, it also allows for fundamentally richer behavior, such as allowing the output type of a UDF to depend on its inputs. The strategy in this diff is to introduce the notion of a ConstantObjectInspector, like a regular ObjectInspector except that it encapsulates a constant value and knows what this constant value is. These COIs are created through a factory method by ExprNodeConstantDesc during plan generation hence UDFs will be able to capture these constant values during the initialize phase. Furthermore, because these ConstantObjectInspectors are simply subinterfaces of ObjectInspector, UDFs which are not "constant-aware" receive ObjectInspectors which also implement the same interfaces they are used to, so no special handling needs to be done for existing UDFs. An example UDF which uses this new functionality is also included in this diff. NAMED_STRUCT is like STRUCT except that it also allows users to specify the names of the fields of the struct, something previously not possible because the names of the fields must be known at compile time. Also see this pull request: https://github.com/apache/hive/pull/2
          Hide
          Carl Steinbach added a comment -

          Reassigning to Jonathan.

          @Jonathan: Can you please upload the patch again and this time click the box that gives license rights to the Apache Foundation? Thanks!

          Show
          Carl Steinbach added a comment - Reassigning to Jonathan. @Jonathan: Can you please upload the patch again and this time click the box that gives license rights to the Apache Foundation? Thanks!
          Hide
          Jonathan Chang added a comment -

          Like previous except license is now granted and also fixes show_functions.q.

          Show
          Jonathan Chang added a comment - Like previous except license is now granted and also fixes show_functions.q.
          Hide
          John Sichi added a comment -

          Checked with Carl; he said he'll review this one.

          Show
          John Sichi added a comment - Checked with Carl; he said he'll review this one.
          Hide
          Jonathan Chang added a comment -

          Sweet, thx!

          Show
          Jonathan Chang added a comment - Sweet, thx!
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/1486/
          -----------------------------------------------------------

          Review request for hive.

          Summary
          -------

          Review request for HIVE-1360.

          This addresses bug HIVE-1360.
          https://issues.apache.org/jira/browse/HIVE-1360

          Diffs


          ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeConstantEvaluator.java af62934
          ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java a7b1b77
          ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeConstantDesc.java bf092d4
          ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDesc.java 6b1ba1b
          ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java e264709
          ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFNamedStruct.java PRE-CREATION
          ql/src/test/queries/clientpositive/udf_named_struct.q PRE-CREATION
          ql/src/test/results/clientpositive/show_functions.q.out d66e844
          ql/src/test/results/clientpositive/udf_named_struct.q.out PRE-CREATION
          serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ConstantObjectInspector.java PRE-CREATION
          serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorFactory.java 40fd4e3
          serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantBooleanObjectInspector.java PRE-CREATION
          serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantByteObjectInspector.java PRE-CREATION
          serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantDoubleObjectInspector.java PRE-CREATION
          serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantFloatObjectInspector.java PRE-CREATION
          serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantIntObjectInspector.java PRE-CREATION
          serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantLongObjectInspector.java PRE-CREATION
          serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantShortObjectInspector.java PRE-CREATION
          serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantStringObjectInspector.java PRE-CREATION
          serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantVoidObjectInspector.java PRE-CREATION

          Diff: https://reviews.apache.org/r/1486/diff

          Testing
          -------

          Thanks,

          Carl

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1486/ ----------------------------------------------------------- Review request for hive. Summary ------- Review request for HIVE-1360 . This addresses bug HIVE-1360 . https://issues.apache.org/jira/browse/HIVE-1360 Diffs ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeConstantEvaluator.java af62934 ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java a7b1b77 ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeConstantDesc.java bf092d4 ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDesc.java 6b1ba1b ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java e264709 ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFNamedStruct.java PRE-CREATION ql/src/test/queries/clientpositive/udf_named_struct.q PRE-CREATION ql/src/test/results/clientpositive/show_functions.q.out d66e844 ql/src/test/results/clientpositive/udf_named_struct.q.out PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ConstantObjectInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorFactory.java 40fd4e3 serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantBooleanObjectInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantByteObjectInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantDoubleObjectInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantFloatObjectInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantIntObjectInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantLongObjectInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantShortObjectInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantStringObjectInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantVoidObjectInspector.java PRE-CREATION Diff: https://reviews.apache.org/r/1486/diff Testing ------- Thanks, Carl
          Hide
          Carl Steinbach added a comment -

          Attaching HIVE-1360.2.patch.txt which is just HIVE-1360.patch modified to apply with patch -p0.

          Show
          Carl Steinbach added a comment - Attaching HIVE-1360 .2.patch.txt which is just HIVE-1360 .patch modified to apply with patch -p0 .
          Hide
          Carl Steinbach added a comment -

          +1. Will commit if tests pass.

          Show
          Carl Steinbach added a comment - +1. Will commit if tests pass.
          Hide
          Carl Steinbach added a comment -

          This patch causes a bunch of test failures. Looks like this is caused by mixing o.a.h.io.*Writable classes with o.a.h.hive.serde2.io.*Writable classes:

          FAILED: Hive Internal Error: java.lang.ClassCastException(org.apache.hadoop.hive.serde2.io.DoubleWritable cannot be cast to org.apache.hadoop.io.DoubleWritable)
          java.lang.ClassCastException: org.apache.hadoop.hive.serde2.io.DoubleWritable cannot be cast to org.apache.hadoop.io.DoubleWritable
          at org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory.getPrimitiveWritableConstantObjectInspector(PrimitiveObjectInspectorFactory.java:179)
          at org.apache.hadoop.hive.ql.plan.ExprNodeConstantDesc.getWritableObjectInspector(ExprNodeConstantDesc.java:67)

          Show
          Carl Steinbach added a comment - This patch causes a bunch of test failures. Looks like this is caused by mixing o.a.h.io.*Writable classes with o.a.h.hive.serde2.io.*Writable classes: FAILED: Hive Internal Error: java.lang.ClassCastException(org.apache.hadoop.hive.serde2.io.DoubleWritable cannot be cast to org.apache.hadoop.io.DoubleWritable) java.lang.ClassCastException: org.apache.hadoop.hive.serde2.io.DoubleWritable cannot be cast to org.apache.hadoop.io.DoubleWritable at org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory.getPrimitiveWritableConstantObjectInspector(PrimitiveObjectInspectorFactory.java:179) at org.apache.hadoop.hive.ql.plan.ExprNodeConstantDesc.getWritableObjectInspector(ExprNodeConstantDesc.java:67)
          Hide
          Jonathan Chang added a comment -

          Import DoubleWritable from serde2 instead of serde. This fixes a bunch of unittest failures; sorry for overlooking those!

          The TestParser tests still fail; the .q files seem ok but the xmls seem to differ in a way whose significance I fail to understand, e.g.,

          [junit] 571c571
          [junit] < <string>CNTR_NAME_TS_0_FATAL_ERROR</string>
          [junit]
          [junit] > <string>CNTR_NAME_TS_637_FATAL_ERROR</string>
          [junit] 579c579
          [junit] < <string>TS_0</string>
          [junit]
          [junit] > <string>TS_637</string>

          Do you have any insight on what these mean and what causes them?

          Show
          Jonathan Chang added a comment - Import DoubleWritable from serde2 instead of serde. This fixes a bunch of unittest failures; sorry for overlooking those! The TestParser tests still fail; the .q files seem ok but the xmls seem to differ in a way whose significance I fail to understand, e.g., [junit] 571c571 [junit] < <string>CNTR_NAME_TS_0_FATAL_ERROR</string> [junit] — [junit] > <string>CNTR_NAME_TS_637_FATAL_ERROR</string> [junit] 579c579 [junit] < <string>TS_0</string> [junit] — [junit] > <string>TS_637</string> Do you have any insight on what these mean and what causes them?
          Hide
          Carl Steinbach added a comment -

          TestParse and TestParseNegative both work for me. Running full tests now.

          Show
          Carl Steinbach added a comment - TestParse and TestParseNegative both work for me. Running full tests now.
          Hide
          Carl Steinbach added a comment -

          Tests passed. Committed to trunk. Thanks Jonathan!

          Show
          Carl Steinbach added a comment - Tests passed. Committed to trunk. Thanks Jonathan!
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-h0.21 #900 (See https://builds.apache.org/job/Hive-trunk-h0.21/900/)
          HIVE-1360. Allow UDFs to access constant parameter values at compile time (Jonathan Chang via cws)

          cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1158446
          Files :

          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeConstantDesc.java
          • /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantBooleanObjectInspector.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDesc.java
          • /hive/trunk/ql/src/test/results/clientpositive/show_functions.q.out
          • /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantIntObjectInspector.java
          • /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantStringObjectInspector.java
          • /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantDoubleObjectInspector.java
          • /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantVoidObjectInspector.java
          • /hive/trunk/ql/src/test/results/clientpositive/udf_named_struct.q.out
          • /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ConstantObjectInspector.java
          • /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorFactory.java
          • /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantLongObjectInspector.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFNamedStruct.java
          • /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantShortObjectInspector.java
          • /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantFloatObjectInspector.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeConstantEvaluator.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java
          • /hive/trunk/ql/src/test/queries/clientpositive/udf_named_struct.q
          • /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantByteObjectInspector.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
          Show
          Hudson added a comment - Integrated in Hive-trunk-h0.21 #900 (See https://builds.apache.org/job/Hive-trunk-h0.21/900/ ) HIVE-1360 . Allow UDFs to access constant parameter values at compile time (Jonathan Chang via cws) cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1158446 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeConstantDesc.java /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantBooleanObjectInspector.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDesc.java /hive/trunk/ql/src/test/results/clientpositive/show_functions.q.out /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantIntObjectInspector.java /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantStringObjectInspector.java /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantDoubleObjectInspector.java /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantVoidObjectInspector.java /hive/trunk/ql/src/test/results/clientpositive/udf_named_struct.q.out /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ConstantObjectInspector.java /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorFactory.java /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantLongObjectInspector.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFNamedStruct.java /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantShortObjectInspector.java /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantFloatObjectInspector.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeConstantEvaluator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java /hive/trunk/ql/src/test/queries/clientpositive/udf_named_struct.q /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantByteObjectInspector.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-h0.21 #905 (See https://builds.apache.org/job/Hive-trunk-h0.21/905/)
          HIVE-2393. Fix whitespace test diff accidentally introduced in HIVE-1360. (Carl Steinbach via amareshwari)

          amareshwari : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1159496
          Files :

          • /hive/trunk/ql/src/test/results/clientpositive/udf_named_struct.q.out
          Show
          Hudson added a comment - Integrated in Hive-trunk-h0.21 #905 (See https://builds.apache.org/job/Hive-trunk-h0.21/905/ ) HIVE-2393 . Fix whitespace test diff accidentally introduced in HIVE-1360 . (Carl Steinbach via amareshwari) amareshwari : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1159496 Files : /hive/trunk/ql/src/test/results/clientpositive/udf_named_struct.q.out
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2078/
          -----------------------------------------------------------

          Review request for hive, Carl Steinbach, John Sichi, and Igor Kabiljo.

          Summary
          -------

          ConstantObjectInspectors were introduced in a previous patch so that UDF's could have access to constant values at initialize time. This diff adds the following improvements:

          1.) Extends this functionality beyond primitive types to List/Map types.
          2.) Fixes a bug in ExprNodeGenericFuncDesc where the constant value would not propagate beyond one level of evaluation.
          3.) Adds a helper class to make functions constant-aware. By inheriting from this class they return constant OIs if all their arguments are constant, and they cache the result so that computation is only done during initialize(). Some functions such as ARRAY/IF/MAP have been migrated.

          This addresses bug HIVE-1360.
          https://issues.apache.org/jira/browse/HIVE-1360

          Diffs


          ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java 7da5d6a
          ql/src/java/org/apache/hadoop/hive/ql/udf/generic/ConstantFoldableGenericUDF.java PRE-CREATION
          ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFArray.java 1cf97c8
          ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFArrayContains.java d60dc86
          ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFIf.java 9a6a7ff
          ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFMap.java 3b71c48
          ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFMapKeys.java b384717
          ql/src/test/queries/clientpositive/constant_prop.q PRE-CREATION
          ql/src/test/results/clientpositive/constant_prop.q.out PRE-CREATION
          serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java a4689fb
          serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java 2c1a7e0
          serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/StandardConstantListObjectInspector.java PRE-CREATION
          serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/StandardConstantMapObjectInspector.java PRE-CREATION

          Diff: https://reviews.apache.org/r/2078/diff

          Testing
          -------

          ant test

          Thanks,

          Jonathan

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2078/ ----------------------------------------------------------- Review request for hive, Carl Steinbach, John Sichi, and Igor Kabiljo. Summary ------- ConstantObjectInspectors were introduced in a previous patch so that UDF's could have access to constant values at initialize time. This diff adds the following improvements: 1.) Extends this functionality beyond primitive types to List/Map types. 2.) Fixes a bug in ExprNodeGenericFuncDesc where the constant value would not propagate beyond one level of evaluation. 3.) Adds a helper class to make functions constant-aware. By inheriting from this class they return constant OIs if all their arguments are constant, and they cache the result so that computation is only done during initialize(). Some functions such as ARRAY/IF/MAP have been migrated. This addresses bug HIVE-1360 . https://issues.apache.org/jira/browse/HIVE-1360 Diffs ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java 7da5d6a ql/src/java/org/apache/hadoop/hive/ql/udf/generic/ConstantFoldableGenericUDF.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFArray.java 1cf97c8 ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFArrayContains.java d60dc86 ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFIf.java 9a6a7ff ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFMap.java 3b71c48 ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFMapKeys.java b384717 ql/src/test/queries/clientpositive/constant_prop.q PRE-CREATION ql/src/test/results/clientpositive/constant_prop.q.out PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java a4689fb serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java 2c1a7e0 serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/StandardConstantListObjectInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/StandardConstantMapObjectInspector.java PRE-CREATION Diff: https://reviews.apache.org/r/2078/diff Testing ------- ant test Thanks, Jonathan
          Hide
          Jonathan Chang added a comment -

          Needs some additional work to be useful. See latest patch.

          Show
          Jonathan Chang added a comment - Needs some additional work to be useful. See latest patch.
          Hide
          Jonathan Chang added a comment -

          Additional patch for more functionality.

          Show
          Jonathan Chang added a comment - Additional patch for more functionality.
          Hide
          John Sichi added a comment -

          Jonathan, once we commit, we mark the issue resolved and then open up a new followup JIRA issue if more work is needed (linked to the original). So this one should stay resolved since it already got committed.

          Show
          John Sichi added a comment - Jonathan, once we commit, we mark the issue resolved and then open up a new followup JIRA issue if more work is needed (linked to the original). So this one should stay resolved since it already got committed.
          Hide
          Jonathan Chang added a comment -

          Re-resolving per Jvs.

          Show
          Jonathan Chang added a comment - Re-resolving per Jvs.
          Hide
          Jonathan Chang added a comment -

          Please visit HIVE-2470 for followup.

          Show
          Jonathan Chang added a comment - Please visit HIVE-2470 for followup.

            People

            • Assignee:
              Jonathan Chang
              Reporter:
              Carl Steinbach
            • Votes:
              3 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development