[SPARK-12978] Skip unnecessary final group-by when input data already clustered with group-by keys

• Type: Improvement
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Component/s: SQL
• Assignee: Takeshi Yamamuro
• Reporter: Takeshi Yamamuro
• Created: 25/Jan/16 08:10
• Updated: 19/Nov/20 23:17

Description

This ticket targets the optimization to skip an unnecessary group-by operation below;

Without opt.:

== Physical Plan ==
TungstenAggregate(key=[col0#159], functions=[(sum(col1#160),mode=Final,isDistinct=false),(avg(col2#161),mode=Final,isDistinct=false)], output=[col0#159,sum(col1)#177,avg(col2)#178])
+- TungstenAggregate(key=[col0#159], functions=[(sum(col1#160),mode=Partial,isDistinct=false),(avg(col2#161),mode=Partial,isDistinct=false)], output=[col0#159,sum#200,sum#201,count#202L])
   +- TungstenExchange hashpartitioning(col0#159,200), None
      +- InMemoryColumnarTableScan [col0#159,col1#160,col2#161], InMemoryRelation [col0#159,col1#160,col2#161], true, 10000, StorageLevel(true, true, false, true, 1), ConvertToUnsafe, None

With opt.:

== Physical Plan ==
TungstenAggregate(key=[col0#159], functions=[(sum(col1#160),mode=Complete,isDistinct=false),(avg(col2#161),mode=Final,isDistinct=false)], output=[col0#159,sum(col1)#177,avg(col2)#178])
+- TungstenExchange hashpartitioning(col0#159,200), None
  +- InMemoryColumnarTableScan [col0#159,col1#160,col2#161], InMemoryRelation [col0#159,col1#160,col2#161], true, 10000, StorageLevel(true, true, false, true, 1), ConvertToUnsafe, None

Issue Links

• is duplicated by: SPARK-11580 Just do final aggregation when there is no Exchange
• is related to: SPARK-33486 Collapse Partial and Final Aggregation into Complete Aggregation mode
• links to: [Github] Pull Request #10896 (maropu)
• links to: [Github] Pull Request #15945 (maropu)