[HIVE-1397] histogram() UDAF for a numerical column

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.6.0
• Fix Version/s: 0.6.0
• Component/s: Query Processor
• Hadoop Flags: Reviewed
• Assignee: Mayank Lahiri
• Reporter: Mayank Lahiri
• Created: 08/Jun/10 21:22
• Updated: 17/Dec/11 00:03
• Resolved: 16/Jun/10 02:39

Description

A histogram() UDAF to generate an approximate histogram of a numerical (byte, short, double, long, etc.) column. The result is returned as a map of (x,y) histogram pairs, and can be plotted in Gnuplot using impulses (for example). The algorithm is currently adapted from "A streaming parallel decision tree algorithm" by Ben-Haim and Tom-Tov, JMLR 11 (2010), and uses space proportional to the number of histogram bins specified. It has no approximation guarantees, but seems to work well when there is a lot of data and a large number (e.g. 50-100) of histogram bins specified.

A typical call might be:

SELECT histogram(val, 10) FROM some_table;

where the result would be a histogram with 10 bins, returned as a Hive map object.