[FLINK-5254] Implement Yarn HA Services

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Component/s: Deployment / YARN
• Labels: flip-6
• Assignee: Stephan Ewen
• Reporter: Stephan Ewen
• Created: 05/Dec/16 02:07
• Updated: 29/Jun/17 12:08
• Resolved: 29/Jun/17 12:08

Description

The Yarn HighAvailability Services should be

*Default*

• This option takes the YARN Application's working directory as HA storage
• It automatically uses that working directory for the BlobStore
• It creates a HDFS based "RunningJobsRegistry" (see below)
• ResourceManager leader election has a pre-configured leader, via the configuration, pointing to the AppMaster address.

*ZooKeeper Based*

• The ZooKeeper based services use ZooKeeper for the ResourceManager and JobManager leader election. That way, they are safe against network partition scenarios that otherwise lead to "split brain" situations

A prototype for the simple "single job" RunningJobsRegistry based on HDFS is here: https://github.com/StephanEwen/incubator-flink/commit/aaa2d7758797b2d6c9b6da42be6a5c4989468e3b