Details
-
Sub-task
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
3.2.0
-
None
Description
We are proposing the new functionality in Structured Streaming’s state store, “prefix match scan”, to group the keys into logical groups, and scan the keys in the same group efficiently.
This change is to enable us to construct the “logically list-like” data structure on top of the current state store, which only provides the operations for key-value. There’s an iterator in the state store to scan through the key space, but only supports full scan. Prefix match scan would open the possibility to scan on a specific range of the keys efficiently.
This helps to simplify the state and also logic to the streaming session window, and also achieve similar performance (or even higher) than the approach we proposed before.
