[FLINK-3300] Concurrency Bug in Yarn JobManager

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.0.0
• Component/s: Runtime / Coordination
• Assignee: Maximilian Michels
• Reporter: Stephan Ewen
• Created: 28/Jan/16 20:20
• Updated: 28/Feb/19 14:01
• Resolved: 01/Feb/16 09:12

Description

The change to use the async ResourceManager client introduced concurrency problems: The ResourceManager callback threads run and change data structures at the same time as the actor methods, voiding the actor concurrency model.

One example that can happen is that the callback tries to start containers while the ContainerLaunchContext is still not set (because the actor method is still in the StartYarnSession method).

Bug introducing commit: https://github.com/apache/flink/commit/4e52fe4304566e5239996b3d48290e0c1f0772e8

Quick fix could be to revert the commit. Better solution would be to let the callback methods send actor messages to the YobManager, rather than directly acting.