[FLINK-3927] TaskManager registration may fail if Yarn versions don't match

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.1.0
• Fix Version/s: 1.1.0
• Component/s: Runtime / Coordination
• Assignee: Maximilian Michels
• Reporter: Maximilian Michels
• Created: 18/May/16 15:52
• Updated: 28/Feb/19 14:00
• Resolved: 23/May/16 10:39

Description

Flink's ResourceManager uses the Yarn container ids to identify connecting task managers. Yarn's stringified container id may not be consistent across different Hadoop versions, e.g. Hadoop 2.3.0 and Hadoop 2.7.1. The ResourceManager gets it from the Yarn reports while the TaskManager infers it from the Yarn environment variables. The ResourceManager may use Hadoop 2.3.0 version while the cluster runs Hadoop 2.7.1.

The solution is to pass the ID through a custom environment variable which is set by the ResourceManager before launching the TaskManager in the container. That way we will always use the Hadoop client's id generation method.

Issue Links

• links to: GitHub Pull Request #2013