[FLINK-3667] Generalize client<->cluster communication

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.1.0
• Component/s: Deployment / YARN
• Assignee: Maximilian Michels
• Reporter: Maximilian Michels
• Created: 24/Mar/16 14:34
• Updated: 19/Dec/16 12:47
• Resolved: 17/Jun/16 08:57

Description

Here are some notes I took when inspecting the client<->cluster classes with regard to future integration of other resource management frameworks in addition to Yarn (e.g. Mesos).

1 Cluster Client Abstraction
════════════════════════════

1.1 Status Quo
──────────────

1.1.1 FlinkYarnClient
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

  • Holds the cluster configuration (Flink-specific and Yarn-specific)
  • Contains the deploy() method to deploy the cluster
  • Creates the Hadoop Yarn client
  • Receives the initial job manager address
  • Bootstraps the FlinkYarnCluster


1.1.2 FlinkYarnCluster
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

  • Wrapper around the Hadoop Yarn client
  • Queries cluster for status updates
  • Life time methods to start and shutdown the cluster
  • Flink specific features like shutdown after job completion


1.1.3 ApplicationClient
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

  • Acts as a middle-man for asynchronous cluster communication
  • Designed to communicate with Yarn, not used in Standalone mode


1.1.4 CliFrontend
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

  • Deeply integrated with FlinkYarnClient and FlinkYarnCluster
  • Constantly distinguishes between Yarn and Standalone mode
  • Would be nice to have a general abstraction in place


1.1.5 Client
╌╌╌╌╌╌╌╌╌╌╌╌

  • Job submission and Job related actions, agnostic of resource framework


1.2 Proposal
────────────

1.2.1 ClusterConfig (before: AbstractFlinkYarnClient)
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

  • Extensible cluster-agnostic config
  • May be extended by specific cluster, e.g. YarnClusterConfig


1.2.2 ClusterClient (before: AbstractFlinkYarnClient)
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

  • Deals with cluster (RM) specific communication
  • Exposes framework agnostic information
  • YarnClusterClient, MesosClusterClient, StandaloneClusterClient


1.2.3 FlinkCluster (before: AbstractFlinkYarnCluster)
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

  • Basic interface to communicate with a running cluster
  • Receives the ClusterClient for cluster-specific communication
  • Should not have to care about the specific implementations of the
    client


1.2.4 ApplicationClient
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

  • Can be changed to work cluster-agnostic (first steps already in
    FLINK-3543)


1.2.5 CliFrontend
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

  • CliFrontend does never have to differentiate between different
    cluster types after it has determined which cluster class to load.
  • Base class handles framework agnostic command line arguments
  • Pluggables for Yarn, Mesos handle specific commands

I would like to create/refactor the affected classes to set us up for a more flexible client side resource management abstraction.

Issue Links

• is depended upon by: FLINK-3863 Yarn Cluster shutdown may fail if leader changed recently
• is related to: FLINK-3543 Introduce ResourceManager component
• links to: GitHub Pull Request #1978
• links to: GitHub Pull Request #1978
• links to: GitHub Pull Request #2189