[HIVE-3552] HIVE-3552 performant manner for performing cubes/rollups/grouping sets for a high number of grouping set keys

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.11.0
• Component/s: Query Processor
• Assignee: Namit Jain
• Reporter: Namit Jain
• Created: 08/Oct/12 20:58
• Updated: 31/Jan/15 19:58
• Resolved: 09/Jan/13 17:59

Description

This is a follow up for HIVE-3433.

Had a offline discussion with Sambavi - she pointed out a scenario where the
implementation in HIVE-3433 will not scale. Assume that the user is performing
a cube on many columns, say '8' columns. So, each row would generate 256 rows
for the hash table, which may kill the current group by implementation.

A better implementation would be to add an additional mr job - in the first
mr job perform the group by assuming there was no cube. Add another mr job, where
you would perform the cube. The assumption is that the group by would have
decreased the output data significantly, and the rows would appear in the order of
grouping keys which has a higher probability of hitting the hash table.

Issue Links

• depends upon: HIVE-3433 Implement CUBE and ROLLUP operators in Hive