[HIVE-15580] Eliminate unbounded memory usage for orderBy and groupBy in Hive on Spark

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.3.0
• Component/s: Spark
• Assignee: Xuefu Zhang
• Reporter: Xuefu Zhang
• Created: 11/Jan/17 06:02
• Updated: 21/Jul/17 18:36
• Resolved: 20/Jan/17 20:57

Description

Currently, orderBy (sortBy) and groupBy in Hive on Spark uses unbounded memory. For orderBy, Hive accumulates key groups using ArrayList (described in HIVE-15527). For groupBy, Hive currently uses Spark's groupByKey operator, which has a shortcoming of not being able to spill to disk within a key group. Thus, for large key group, memory usage is also unbounded.

It's likely that this will impact performance. We will profile and optimize afterwards. We could also make this change configurable.

Issue Links

• incorporates: HIVE-15527 Memory usage is unbound in SortByShuffler for Spark
• relates to: HIVE-15682 Eliminate per-row based dummy iterator creation
• relates to: HIVE-15683 Make what's done in HIVE-15580 for group by configurable