Please visit HIVE-2470 for followup.

Re-resolving per Jvs.

Jonathan, once we commit, we mark the issue resolved and then open up a new followup JIRA issue if more work is needed (linked to the original). So this one should stay resolved since it already got committed.

Additional patch for more functionality.

Needs some additional work to be useful. See latest patch.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2078/
-----------------------------------------------------------

Review request for hive, Carl Steinbach, John Sichi, and Igor Kabiljo.

Summary
-------

ConstantObjectInspectors were introduced in a previous patch so that UDF's could have access to constant values at initialize time. This diff adds the following improvements:

1.) Extends this functionality beyond primitive types to List/Map types.
2.) Fixes a bug in ExprNodeGenericFuncDesc where the constant value would not propagate beyond one level of evaluation.
3.) Adds a helper class to make functions constant-aware. By inheriting from this class they return constant OIs if all their arguments are constant, and they cache the result so that computation is only done during initialize(). Some functions such as ARRAY/IF/MAP have been migrated.

This addresses bug HIVE-1360.
https://issues.apache.org/jira/browse/HIVE-1360

Diffs

ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java 7da5d6a
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/ConstantFoldableGenericUDF.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFArray.java 1cf97c8
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFArrayContains.java d60dc86
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFIf.java 9a6a7ff
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFMap.java 3b71c48
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFMapKeys.java b384717
ql/src/test/queries/clientpositive/constant_prop.q PRE-CREATION
ql/src/test/results/clientpositive/constant_prop.q.out PRE-CREATION
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java a4689fb
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java 2c1a7e0
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/StandardConstantListObjectInspector.java PRE-CREATION
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/StandardConstantMapObjectInspector.java PRE-CREATION

Diff: https://reviews.apache.org/r/2078/diff

Testing
-------

ant test

Thanks,

Jonathan

Integrated in Hive-trunk-h0.21 #905 (See https://builds.apache.org/job/Hive-trunk-h0.21/905/)
HIVE-2393. Fix whitespace test diff accidentally introduced in HIVE-1360. (Carl Steinbach via amareshwari)

amareshwari : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1159496
Files :

• /hive/trunk/ql/src/test/results/clientpositive/udf_named_struct.q.out

Integrated in Hive-trunk-h0.21 #900 (See https://builds.apache.org/job/Hive-trunk-h0.21/900/)
HIVE-1360. Allow UDFs to access constant parameter values at compile time (Jonathan Chang via cws)

cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1158446
Files :

• /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeConstantDesc.java
• /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantBooleanObjectInspector.java
• /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDesc.java
• /hive/trunk/ql/src/test/results/clientpositive/show_functions.q.out
• /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantIntObjectInspector.java
• /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantStringObjectInspector.java
• /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantDoubleObjectInspector.java
• /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantVoidObjectInspector.java
• /hive/trunk/ql/src/test/results/clientpositive/udf_named_struct.q.out
• /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ConstantObjectInspector.java
• /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorFactory.java
• /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantLongObjectInspector.java
• /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFNamedStruct.java
• /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantShortObjectInspector.java
• /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantFloatObjectInspector.java
• /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeConstantEvaluator.java
• /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java
• /hive/trunk/ql/src/test/queries/clientpositive/udf_named_struct.q
• /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantByteObjectInspector.java
• /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java

Tests passed. Committed to trunk. Thanks Jonathan!

TestParse and TestParseNegative both work for me. Running full tests now.

Import DoubleWritable from serde2 instead of serde. This fixes a bunch of unittest failures; sorry for overlooking those!

The TestParser tests still fail; the .q files seem ok but the xmls seem to differ in a way whose significance I fail to understand, e.g.,

[junit] 571c571
[junit] < <string>CNTR_NAME_TS_0_FATAL_ERROR</string>
[junit] —
[junit] > <string>CNTR_NAME_TS_637_FATAL_ERROR</string>
[junit] 579c579
[junit] < <string>TS_0</string>
[junit] —
[junit] > <string>TS_637</string>

Do you have any insight on what these mean and what causes them?

This patch causes a bunch of test failures. Looks like this is caused by mixing o.a.h.io.*Writable classes with o.a.h.hive.serde2.io.*Writable classes:

FAILED: Hive Internal Error: java.lang.ClassCastException(org.apache.hadoop.hive.serde2.io.DoubleWritable cannot be cast to org.apache.hadoop.io.DoubleWritable)
java.lang.ClassCastException: org.apache.hadoop.hive.serde2.io.DoubleWritable cannot be cast to org.apache.hadoop.io.DoubleWritable
at org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory.getPrimitiveWritableConstantObjectInspector(PrimitiveObjectInspectorFactory.java:179)
at org.apache.hadoop.hive.ql.plan.ExprNodeConstantDesc.getWritableObjectInspector(ExprNodeConstantDesc.java:67)

+1. Will commit if tests pass.

Attaching HIVE-1360.2.patch.txt which is just HIVE-1360.patch modified to apply with patch -p0.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1486/
-----------------------------------------------------------

Review request for hive.

Summary
-------

Review request for HIVE-1360.

This addresses bug HIVE-1360.
https://issues.apache.org/jira/browse/HIVE-1360

Diffs

ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeConstantEvaluator.java af62934
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java a7b1b77
ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeConstantDesc.java bf092d4
ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDesc.java 6b1ba1b
ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java e264709
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFNamedStruct.java PRE-CREATION
ql/src/test/queries/clientpositive/udf_named_struct.q PRE-CREATION
ql/src/test/results/clientpositive/show_functions.q.out d66e844
ql/src/test/results/clientpositive/udf_named_struct.q.out PRE-CREATION
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ConstantObjectInspector.java PRE-CREATION
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorFactory.java 40fd4e3
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantBooleanObjectInspector.java PRE-CREATION
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantByteObjectInspector.java PRE-CREATION
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantDoubleObjectInspector.java PRE-CREATION
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantFloatObjectInspector.java PRE-CREATION
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantIntObjectInspector.java PRE-CREATION
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantLongObjectInspector.java PRE-CREATION
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantShortObjectInspector.java PRE-CREATION
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantStringObjectInspector.java PRE-CREATION
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantVoidObjectInspector.java PRE-CREATION

Diff: https://reviews.apache.org/r/1486/diff

Testing
-------

Thanks,

Carl

Sweet, thx!

Checked with Carl; he said he'll review this one.

Like previous except license is now granted and also fixes show_functions.q.

Reassigning to Jonathan.

@Jonathan: Can you please upload the patch again and this time click the box that gives license rights to the Apache Foundation? Thanks!

HIVE-1360. It has been a long-standing request for UDFs to be able to
Edit
access parameter values. This not only enables significant performance
improvement possibilities, it also allows for fundamentally richer
behavior, such as allowing the output type of a UDF to depend on its
inputs.

The strategy in this diff is to introduce the notion of a
ConstantObjectInspector, like a regular ObjectInspector except that it
encapsulates a constant value and knows what this constant value is.
These COIs are created through a factory method by ExprNodeConstantDesc
during plan generation hence UDFs will be able to capture these constant
values during the initialize phase. Furthermore, because these
ConstantObjectInspectors are simply subinterfaces of ObjectInspector,
UDFs which are not "constant-aware" receive ObjectInspectors which
also implement the same interfaces they are used to, so no special
handling needs to be done for existing UDFs.

An example UDF which uses this new functionality is also included in
this diff. NAMED_STRUCT is like STRUCT except that it also allows users
to specify the names of the fields of the struct, something previously
not possible because the names of the fields must be known at compile
time.

Also see this pull request: https://github.com/apache/hive/pull/2

Some of the generic base class stuff from HIVE-1016 may be relevant here.

Glad this is getting some traction! Ideally, the "constancy" here should be detectable in the init/initialize state, and, if it IS, the constant should be accessible on initialization.

Working on a patch on this since its a better alternative to solve my problem in HIVE-2124