[SPARK-18487] Add task completion listener to HashAggregate to avoid memory leak

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Not A Problem
• Component/s: SQL
• Assignee: Unassigned
• Reporter: L. C. Hsieh
• Created: 17/Nov/16 08:44
• Updated: 25/Nov/16 18:37
• Resolved: 25/Nov/16 18:36

Description

The methods such as Dataset.show and take use Limit (CollectLimitExec) which leverages SparkPlan.executeTake to efficiently collect required number of elements back to the driver.

However, under wholestage codege, we usually release resources after all elements are consumed (e.g., HashAggregate). In this case, we will not release the resources and cause memory leak with Dataset.show, for example.

We can add task completion listener to HashAggregate to avoid the memory leak.

Issue Links

• relates to: SPARK-18557 Downgrade the memory leak warning message
• links to: [Github] Pull Request #15916 (viirya)