[FLINK-23342] State ownership: compute shared state on recovery

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Fix Version/s: 1.16.0
• Component/s: Runtime / State Backends
• Labels: pull-request-available stale-assigned
• Assignee: Unassigned
• Reporter: Roman Khachatryan
• Created: 09/Jul/21 21:36
• Updated: 11/Jul/24 11:40
• Resolved: 15/Aug/22 14:25

Description

1. Introduce / refactor State object ID (consider StringBasedID, SharedStateRegistryKey, StateHandleID; consider merging them)
2. Add methods to obtain all object ids from a state handle
3. Upate StateAssignmentOperation to compute shared state as a last step
4. Send to TM (inject into JMTaskRestore)

See https://docs.google.com/document/d/1NJJQ30P27BmUvD7oa4FChvkYxMEgjRPTVdO1dHLl_9I/edit#heading=h.uednun2snp4l for more details

 

Code changes might look like this: https://github.com/rkhachatryan/flink/commit/025813c68b71e9fc357d7417d4403141998ad5f9#diff-cc692216ebbbeda497a335d79a1e568bb5164d98a45bd8efba08b9c1e9742ebdR144 

Issue Links

• blocks: FLINK-23137 State ownership: prevent private state deletion on JM
• blocks: FLINK-23343 State ownership: prevent shared state deletion on TM
• relates to: FLINK-24611 Prevent JM from discarding state on checkpoint abortion
• links to: GitHub Pull Request #16510
• links to: GitHub Pull Request #17258
• links to: GitHub Pull Request #17282