[SPARK-27228] Spark long delay on close, possible problem with killing executors

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.3.0
• Component/s: Block Manager, Spark Core
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Lukas Waldmann
• Created: 21/Mar/19 12:17
• Updated: 12/Dec/22 18:10
• Resolved: 25/May/21 01:44

Description

When using dynamic allocations after all jobs finishes spark delays for several minutes before finally finishes. Log suggest that executors are not cleared up properly.

See the attachment for log