[SPARK-34237] Add more metrics (fallback, spill) for object hash aggregate

• Type: Sub-task
• Status: Resolved
• Priority: Trivial
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.2.0
• Component/s: SQL
• Assignee: Cheng Su
• Reporter: Cheng Su
• Created: 26/Jan/21 06:38
• Updated: 29/Jan/21 07:04
• Resolved: 29/Jan/21 04:36

Description

As object hash aggregate fallback mechanism is special - it will fallback to sort-based aggregation based on number of keys seen so far [0]. This fallback logic sometimes is sub-optimal and leads to unnecessary sort, and performance degradation in run-time. The first step to help user/developer debug is to add more related metrics on UI, e.g. spill size, and number of fallback to sort-based aggregation. (spill size metrics was already added for hash aggregate [1])

 

[0]: https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/ObjectAggregationIterator.scala#L161 

[1]: https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala#L68 

Issue Links

• links to: [Github] Pull Request #31340 (c21)