[HIVE-1139] GroupByOperator sometimes throws OutOfMemory error when there are too many distinct keys

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.5.0
• Component/s: Query Processor
• Assignee: Arvind Prabhakar
• Reporter: Ning Zhang
• Created: 08/Feb/10 19:58
• Updated: 31/Jul/18 05:51

Description

When a partial aggregation performed on a mapper, a HashMap is created to keep all distinct keys in main memory. This could leads to OOM exception when there are too many distinct keys for a particular mapper. A workaround is to set the map split size smaller so that each mapper takes less number of rows. A better solution is to use the persistent HashMapWrapper (currently used in CommonJoinOperator) to spill overflow rows to disk.