Details
-
Sub-task
-
Status: Closed
-
Critical
-
Resolution: Won't Fix
-
None
-
None
-
None
-
None
Description
WindowedKeyedStorage is basically a Map<Window, Map<K, V>>, and we need the capability of getting all K's given a window, and getting V given a window and K.
Currently, the spillable implementation of WindowedKeyedStorage uses two spillable data structures – SpillableByteMapImpl<Pair<Window, K>, V> and SpillableArrayListMultimapImpl<Window, K>. This will not work because we need to be able to remove a key from a window, which SpillableArrayList does not support, and having two separate spillable data structures if it can be achieved by just one should be avoided in general.
We will implement the solution that supports prefix scanning (given a window, return all keys), which requires the keys to be stored in order in managed state.
Traversing all keys in the spillable map is needed by WindowStateMap. The WindowedOperator needs a way to traverse all windows in its state when firing a trigger. However, this is less urgent since the Window meta info is small and should fit in memory even with millions of windows. This is more for the checkpointing efficiency.