[SPARK-27207] There exists a bug with SortBasedAggregator where merge()/update() operations get invoked on the aggregate buffer without calling initialize

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Not A Problem
• Affects Version/s: 3.0.0
• Component/s: SQL
• Assignee: Parth Gandhi
• Reporter: Parth Gandhi
• Created: 19/Mar/19 21:13
• Updated: 18/May/19 03:19
• Resolved: 09/May/19 03:21

Description

Normally, the aggregate operations that are invoked for an aggregation buffer for User Defined Aggregate Functions(UDAF) follow the order like initialize(), update(), eval() OR initialize(), merge(), eval(). However, after a certain threshold configurable by spark.sql.objectHashAggregate.sortBased.fallbackThreshold is reached, ObjectHashAggregate falls back to SortBasedAggregator which invokes the merge or update operation without calling initialize() on the aggregate buffer.

Issue Links

• links to: GitHub Pull Request #24149