[FLINK-14058] FLIP-53 Fine-grained Operator Resource Management

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.9.0
• Fix Version/s: 1.10.0
• Component/s: Runtime / Coordination
• Labels: Umbrella

Release Note

Config options `table.exec.resource.external-buffer-memory`,
`table.exec.resource.hash-agg.memory`, `table.exec.resource.hash-join.memory`,
and `table.exec.resource.sort.memory` have been deprecated. Beginning from Flink
1.10, these config options are interpreted as weight hints instead of absolute
memory requirements. Flink choses sensible default weight hints which should
not be adjustment by users.

• Assignee: Zhu Zhu
• Reporter: Xintong Song
• Created: 12/Sep/19 02:46
• Updated: 28/Jan/20 10:22
• Resolved: 08/Jan/20 09:37

Description

This is the umbrella issue of 'FLIP-53: Fine Grained Operator Resource Management'.

Issue Links

• is related to: FLINK-14883 Resource management on state backends
• supercedes: FLINK-5131 Fine-grained Resource Configuration
• links to: FLIP-53: Fine Grained Operator Resource Management

Sub-Tasks

• Introduce option allVerticesInSameSlotSharingGroupByDefault in ExecutionConfig: Closed
• Set slot sharing groups according to pipelined regions: Resolved
• Introduce managed memory fractions to StreamConfig: Resolved
• Set managed memory fractions according to slot sharing groups: Closed
• Operators use fractions to decide how many managed memory to allocate: Closed
• Support building logical pipelined regions from JobGraph: Resolved
• Allocate shared slot resources respecting the resources of all vertices in the group: Closed
• Make clear the way to aggregate specified cpuCores resources: Resolved
• Enable to get/set whether an operator uses managed memory: Resolved
• Fix matching logics of ResourceSpec/ResourceProfile/Resource considering double values: Closed
• Move allVerticesInSameSlotSharingGroupByDefault from ExecutionConfig to StreamGraph: Closed
• Add a ResourceSpec in SlotSharingGroup to describe its overall resources: Closed
• Managed memory fractions should be rounded properly to not summed up to be more than 1.0: Closed
• Introduce unknown memory setting to table in blink planner: Closed
• Remove Config Options table.exec.resource.*memory from Documentation: Closed