[SPARK-25266] Fix memory leak in Barrier Execution Mode

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 2.4.0
• Component/s: Scheduler, Spark Core
• Target Version/s: 2.4.0
• Epic Link: Support Barrier Execution Mode
• Assignee: Kousuke Saruta
• Reporter: Kousuke Saruta
• Created: 28/Aug/18 17:44
• Updated: 02/Sep/18 23:06
• Resolved: 29/Aug/18 14:13

Description

BarrierCoordinator uses Timer and TimerTask. `TimerTask#cancel()` is invoked in ContextBarrierState#cancelTimerTask but `Timer#purge()` is never invoked.

Once a TimerTask is scheduled, the reference to it is not released until `Timer#purge()` is invoked even though `TimerTask#cancel()` is invoked.

 

Issue Links

• duplicates: SPARK-25265 Fix memory leak in Barrier Execution Mode
• links to: [Github] Pull Request #22258 (sarutak)