[HIVE-2124] Specified functions in the partitioning predicates should not generate a M/R job.

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 0.5.0, 0.6.0, 0.7.0
• Component/s: Query Processor
• Labels: features new
• Assignee: Unassigned
• Reporter: Esteban Gutierrez
• Created: 22/Apr/11 03:49
• Updated: 19/Aug/11 00:46

Description

For certain situations specifying which functions should be evaluated once would help to make syntax simpler to avoid launching M/R jobs.

Example:

# myhql.time=`date "+%s"` -> constant
# counting rows from the last 30 days generates a M/R job using all the partitions
$ hive -hiveconf myhql.time=`date "+%s"` -e "SELECT COUNT(*) FROM mybigtable WHERE mypartition >= from_unixtime(\${hiveconf:myhql.time}-2592000,'yyyy-MM-dd');

Suggested feature:

# will scan only the right partitions
$ hive -hiveconf hive.partition.evaluateonce=unix_timestamp -e "SELECT COUNT(*) FROM mybigtable WHERE mypartition >= from_unixtime(unix_timestamp()-2592000,'yyyy-MM-dd');

Issue Links

• relates to: HIVE-1986 partition pruner do not take effect for non-deterministic UDF
• relates to: HIVE-1360 Allow UDFs to access constant parameter values at compile time