[HIVE-1016] Ability to access DistributedCache from UDFs

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Component/s: Query Processor
• Tags: UDF DistributedCache
• Assignee: Carl Steinbach
• Reporter: Carl Steinbach
• Created: 28/Dec/09 20:34
• Updated: 06/May/13 18:17
• Resolved: 24/Apr/13 11:55

Description

There have been several requests on the mailing list for
information about how to access the DistributedCache from UDFs, e.g.:

http://www.mail-archive.com/[e-mail]/msg01650.html
http://www.mail-archive.com/[e-mail]/msg01926.html

While responses to these emails suggested several workarounds, the only correct
way of accessing the distributed cache is via the static methods of Hadoop's
DistributedCache class, and all of these methods require that the JobConf be passed
in as a parameter. Hence, giving UDFs access to the distributed cache
reduces to giving UDFs access to the JobConf.

I propose the following changes to GenericUDF/UDAF/UDTF:

• Add an exec_init(Configuration conf) method that is called during Operator initialization at runtime.
• Change the name of the "initialize" method to "compile_init" to make it clear that this method is called at compile-time.

Issue Links

• blocks: HIVE-2312 Make CLI variables available to UDFs
• duplicates: HIVE-3628 Provide a way to use counters in Hive through UDF
• is related to: HIVE-2261 Add cleanup stages for UDFs
• relates to: HIVE-1360 Allow UDFs to access constant parameter values at compile time