Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
-
None
Description
Partitioned states are currently stored per-key in statehandles. This is alright for in-memory storage but is very inefficient for HDFS.
The logic behind the current mechanism is that this approach provides a way to repartition a state without fetching the data from the external storage and only manipulating handles.
We should come up with a solution that can achieve both.