[SPARK-40224] Make ObjectHashAggregateExec release memory eagerly when fallback to sort-based

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: SQL
• Assignee: XiDuo You
• Reporter: XiDuo You
• Created: 26/Aug/22 02:46
• Updated: 29/Aug/22 07:23
• Resolved: 29/Aug/22 07:22

Description

Avoid OOM issue as far as possible:

ObjectAggregationIterator INFO - Aggregation hash map size 128 reaches threshold capacity (128 entries), spilling and falling back to sort based aggregation. You may change the threshold by adjust option spark.sql.objectHashAggregate.sortBased.fallbackThreshold

#
# java.lang.OutOfMemoryError: Java heap space
# -XX:OnOutOfMemoryError="kill %p"
#   Executing /bin/sh -c "kill 46725"...

Issue Links

• links to: [Github] Pull Request #37668 (ulysses-you)