[SPARK-40492] Perform maintenance of StateStore instances when they become inactive

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.4.0
• Component/s: Structured Streaming
• Assignee: Chaoqin Li
• Reporter: Chaoqin Li
• Created: 20/Sep/22 04:39
• Updated: 26/Sep/22 02:45
• Resolved: 26/Sep/22 02:45

Description

Current the maintenance of StateStore is performed by a periodic task in the management thread. If a streaming query become inactive before the next maintenance task fire, its StateStore will be unloaded before cleanup.
There are 2 cases when a StateStore is unloaded.

1. StateStoreProvider is not longer active in the system, for example, when a query ends or the spark context terminates.
2. There is other active StateStoreProvider in the system, for example, when a partition is reassigned.

In case 1, we should do one last maintenance before unloading the instance.

Issue Links

• links to: [Github] Pull Request #37935 (chaoqin-li1123)
• links to: [Github] Pull Request #37935 (chaoqin-li1123)