Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
3.3.0
-
None
Description
Current the maintenance of StateStore is performed by a periodic task in the management thread. If a streaming query become inactive before the next maintenance task fire, its StateStore will be unloaded before cleanup.
There are 2 cases when a StateStore is unloaded.
- StateStoreProvider is not longer active in the system, for example, when a query ends or the spark context terminates.
- There is other active StateStoreProvider in the system, for example, when a partition is reassigned.
In case 1, we should do one last maintenance before unloading the instance.