[SPARK-48997] Maintenance thread pool error should not cause the entire executor to crash

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: Structured Streaming
• Labels: pull-request-available
• Assignee: Neil Ramaswamy
• Reporter: Neil Ramaswamy
• Created: 24/Jul/24 22:55
• Updated: 30/Jul/24 08:22
• Resolved: 30/Jul/24 08:22

Description

Today, it's possible for an exception within a thread in the maintenance pool to cause the entire executor to crash. Here's how:

1. An error occurs in a maintenance pool thread
2. It gets passed to the maintenance task thread, which `throw`s it
3. That gets caught by `onError`, which `.stop()`s the maintenance thread pool
4. If any of the maintenance pool threads are waiting on a lock, they will receive an `InterruptedException` (this happens if they are verifying if the their state store instance is active)
5. This `InterruptedException` is not caught, which is not `NonFatal`
6. This uncaught exception bubbles all the way to the `SparkUncaughtExceptionHandler`, causing the executor to exit

A fix that is better is to modify the maintenance thread pool to only `unload` providers that experience errors, not stop the entire thread pool.

Issue Links

• links to: GitHub Pull Request #47475