[SPARK-47121] Avoid noisy RejectedExecutionExceptions during StandaloneSchedulerBackend shutdown

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.5.0
• Fix Version/s: 4.0.0
• Component/s: Scheduler
• Labels: pull-request-available
• Assignee: Josh Rosen
• Reporter: Josh Rosen
• Created: 21/Feb/24 19:55
• Updated: 21/Feb/24 21:35
• Resolved: 21/Feb/24 21:35

Description

While it is in the process of shutting down, the StandaloneSchedulerBackend might throw RejectedExecutionExceptions when RPC handler `onDisconnected` methods attempt to submit new tasks to a stopped executorDelayRemoveThread executor service. We can reduce log and uncaught exception noise by catching and ignoring these exceptions if they occur during shudown.

Issue Links

• links to: GitHub Pull Request #45203