[SPARK-25901] Barrier mode spawns a bunch of threads that get collected on gc

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 3.0.0
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: yogesh garg
• Created: 31/Oct/18 18:59
• Updated: 03/Nov/18 06:11
• Resolved: 03/Nov/18 06:11

Description

After a barrier job is terminated (successfully or interrupted), the accompanying thread created with `Timer` in `BarrierTaskContext` shows in a waiting state until gc is called. We should probably have just one thread to schedule all such tasks, since they just log every 60 seconds.

Here's a screen shot of the threads growing with more tasks:

Here's a screen shot of constant number of threads with more tasks:

Issue Links

• links to: [Github] Pull Request #22912 (yogeshg)