[SPARK-34253] Object hash aggregate should not fallback if no more input rows

• Type: Sub-task
• Status: Resolved
• Priority: Trivial
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.2.0
• Component/s: SQL
• Assignee: Cheng Su
• Reporter: Cheng Su
• Created: 26/Jan/21 23:39
• Updated: 29/Jan/21 07:05
• Resolved: 28/Jan/21 15:19

Description

Object hash aggregate will fallback to sort-based aggregation based on number of keys seen so far [0]. The default config threshold is 128 (spark.sql.objectHashAggregate.sortBased.fallbackThreshold in [1]). There's an edge case we can do better, where we do not fallback if there's no more input rows. Suppose the task only has 128 group-by keys in hash ma, we don't need to fallback in this case, and we can save the extra sort.

 

[0]: https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/ObjectAggregationIterator.scala#L161 

[1]: https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L1615 

Issue Links

• links to: [Github] Pull Request #31353 (c21)