[FLINK-16655] Introduce the EmbeddedExecutor for executing the main() on the Dispatcher

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Implemented
• Affects Version/s: 1.10.0
• Fix Version/s: 1.11.0
• Component/s: Client / Job Submission
• Labels: pull-request-available
• Assignee: Kostas Kloudas
• Reporter: Kostas Kloudas
• Created: 18/Mar/20 13:22
• Updated: 11/Jun/20 07:29
• Resolved: 06/Apr/20 18:41

Description

Running the user's main method in "Application Mode" implies:
1) launching a dedicated cluster for the application's jobs
2) running the user's main on the cluster, alongside the Dispatcher

The EmbeddedExecutor will conceptually be like the existing Executors for session clusters, with the difference that this time there is no need to go through REST as this will be running already on the same machine as the Dispatcher.

This executor, apart from the application mode can also be used in the case of the web submission and the StandaloneJobClusterEntryPoint which so far do not use executors.

Issue Links

• Dependency: FLINK-16657 Wire the EmbeddedExecutor to the Web Submission logic.
• Dependency: FLINK-16658 Wire the EmbeddedExecutor to the StandaloneJobClusterEntryPoint.
• Dependency: FLINK-16661 Introduce the YarnApplicationClusterEntrypoint
• links to: GitHub Pull Request #11460