[FLINK-10848] Flink's Yarn ResourceManager can allocate too many excess containers

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.3.3, 1.4.2, 1.5.5, 1.6.2
• Fix Version/s: 1.6.4, 1.7.2, 1.8.0
• Component/s: Deployment / YARN
• Labels: pull-request-available
• Assignee: Till Rohrmann
• Reporter: Shuyi Chen
• Created: 11/Nov/18 07:35
• Updated: 29/Mar/19 11:42
• Resolved: 11/Jan/19 15:18

Description

Currently, both the YarnFlinkResourceManager and YarnResourceManager do not call removeContainerRequest() on container allocation success. Because the YARN AM-RM protocol is not a delta protocol (please see YARN-1902), AMRMClient will keep all ContainerRequests that are added and send them to RM.

In production, we observe the following that verifies the theory: 16 containers are allocated and used upon cluster startup; when a TM is killed, 17 containers are allocated, 1 container is used, and 16 excess containers are returned; when another TM is killed, 18 containers are allocated, 1 container is used, and 17 excess containers are returned.

Issue Links

• is duplicated by: FLINK-10735 flink on yarn close container exception
• is duplicated by: FLINK-11149 Flink will request too more containers than it actually needs
• links to: GitHub Pull Request #7078
• links to: GitHub Pull Request #7430