[SPARK-44438] Shutdown maintenance task thread on error

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.5.0
• Fix Version/s: 3.5.0
• Component/s: Structured Streaming
• Assignee: Anish Shrigondekar
• Reporter: Anish Shrigondekar
• Created: 14/Jul/23 19:55
• Updated: 15/Jul/23 13:33
• Resolved: 15/Jul/23 13:33

Description

Shutdown maintenance task thread on error

 

Currently if we get an error on maintenance, the future is set to false, but the thread is actually never reaped. If this happens often, we end up accumulating the state store threads that don't have any reference. Eventually this could lead to thread exhaustion.

 

Eg logs -

log4j-2023-07-13-18.log:6968:23/07/13 18:34:25 INFO StateStore: State Store maintenance task started
log4j-2023-07-13-18.log:282671:23/07/13 18:43:28 INFO StateStore: State Store maintenance task started
log4j-2023-07-13-18.log:322145:23/07/13 18:44:32 INFO StateStore: State Store maintenance task started
log4j-2023-07-13-18.log:361192:23/07/13 18:45:37 INFO StateStore: State Store maintenance task started
log4j-2023-07-13-18.log:398145:23/07/13 18:46:37 INFO StateStore: State Store maintenance task started
log4j-2023-07-13-18.log:436591:23/07/13 18:47:42 INFO StateStore: State Store maintenance task started
log4j-2023-07-13-18.log:478531:23/07/13 18:48:47 INFO StateStore: State Store maintenance task started