[FLINK-3755] Introduce key groups for key-value state to support dynamic scaling

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.1.0
• Fix Version/s: 1.2.0
• Assignee: Till Rohrmann
• Reporter: Till Rohrmann
• Created: 14/Apr/16 09:05
• Updated: 18/Oct/23 03:16
• Resolved: 01/Sep/16 08:00

Description

In order to support dynamic scaling, it is necessary to sub-partition the key-value states of each operator. This sub-partitioning, which produces a set of key groups, allows to easily scale in and out Flink jobs by simply reassigning the different key groups to the new set of sub tasks. The idea of key groups is described in this design document [1].

[1] https://docs.google.com/document/d/1G1OS1z3xEBOrYD4wSu-LuBCyPUWyFd9l3T9WyssQ63w/edit?usp=sharing

Issue Links

• links to: GitHub Pull Request #2376
• links to: GitHub Pull Request #2440

Sub-Tasks

• Introduce state hierarchy in CheckpointCoordinator: Closed
• Refactor State Backends/Make Keyed State Key-Group Aware: Resolved
• Introduce KeyGroupAssigner and Max-Parallelism Parameter: Resolved
• Refactor State to Prepare For Key-Group State Backends: Resolved