Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
0.8.0, 0.8.1, 0.9.0, 0.10.0
-
None
-
None
Description
Today, Hive decides to perform a map side aggregation by default. If the number of unique keys in the aggregation is small, performing a map side aggregation is beneficial. However, if the number of keys is sufficiently large, it can lead to OOMEs. Upon encountering an OOME, hive.map.aggr has be set to false to turn it off. Instead, we can use the number of distinct values in the group by column along with the number of rows in the table to decide if map side aggregation should be used.