[HIVE-11502] Map side aggregation is extremely slow

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.2.0
• Fix Version/s: 1.3.0, 2.0.0
• Component/s: Logical Optimizer, Physical Optimizer
• Assignee: Yongzhi Chen
• Reporter: Yongzhi Chen
• Created: 08/Aug/15 21:34
• Updated: 16/Feb/16 23:53
• Resolved: 20/Aug/15 01:23

Description

For the query as following:

create table tbl2 as 
select col1, max(col2) as col2 
from tbl1 group by col1;

If the column for group by has many different values (for example 400000) and it is in type double, the map side aggregation is very slow. I ran the query which took more than 3 hours , after 3 hours, I have to kill the query.
The same query can finish in 7 seconds, if I turn off map side aggregation by:

set hive.map.aggr = false;

Issue Links

• is duplicated by: HIVE-9495 Map Side aggregation affecting map performance
• is duplicated by: HIVE-12093 launch local task to process map join cost long time
• is related to: HADOOP-12217 hashCode in DoubleWritable returns same value for many numbers
• relates to: HIVE-11761 DoubleWritable hashcode for GroupBy is not properly generated