Details
-
Improvement
-
Status: Closed
-
Major
-
Resolution: Won't Fix
-
1.11.1
-
None
Description
To restore the states, union state needs to read state handles produced by all operators. And currently during the restore operation, Flink iterates the state handles one by one, which could last tens of minutes if the magnitude of state handles exceeds ten thousand.
To accelerate the process, I propose to parallelize the random reads on HDFS and deserialization. We can create a runnable for each state handle and let it return the metadata and deserialized data, which can be aggregated in main thread.